Interesting Difference in Math Calculations in different programs

SteveH66 · 05 Nov 2012

I noticed a difference in the results of a math problem I did in 4 separate programs, and they split 2 to 2 on the result. I multiplied the following 2 numbers in Windows Calculator, a Firefox add-on calculator, Libre Office Calc and MS Excel with interesting resuts :

82.19777664*33.20088775

Results
Windows Calculator = 2,729.039155524212
Firefox Add-on Calculator = 2729.039155524212
Libre Office Calc = 2,729.039155524210
Microsoft Excel 2010 = 2,729.039155524210

As you can see, the calculators agree on the numbers and the spreadsheet programs agree on the numbers, but as you can also see the 2 different types of programs agreed on different numbers - it appears the spreadsheets rounded the last number down to zero.

I allowed for 12 decimal places in the spreadsheet programs, I even added decimal places to 14 or 15 and I got the same results - with additional trailing zero's in the additional decimal places in the spreadsheet programs.

Anyone have any idea how and why these programs are handling the math this way? Thanks for any insight anyone can shed on this riddle

logicearth · 05 Nov 2012

Floating point numbers (whole numbers with decimals) is not something that computers are good at. And there could be a lot of reasons why they display them as such.

SteveH66 · 05 Nov 2012

Thanks for that reply logicearth, interesting to know

mjf · 07 Nov 2012

Actually computers are just as fine with floating point numbers as they are with integers. You are observing roundoff errors (last digit) which is simply due to the number of bits assigned to the floating point number. Typically
32 bit = single precision
64 bit = double precision
128 bit = quadruple precision
etc.
In floating point bits are assigned to the mantissa and exponent. The precision and dynamic range is quite defined. Integer operations can also have roundoff errors.