Windows 7 Forums
Welcome to Windows 7 Forums. Our forum is dedicated to helping you find support and solutions for any problems regarding your Windows 7 PC be it Dell, HP, Acer, Asus or a custom build. We also provide an extensive Windows 7 tutorial section that covers a wide range of tips and tricks.


Windows 7: Interesting Difference in Math Calculations in different programs


05 Nov 2012   #1

Windows 7 Home Premium 64 bit
 
 
Interesting Difference in Math Calculations in different programs

I noticed a difference in the results of a math problem I did in 4 separate programs, and they split 2 to 2 on the result. I multiplied the following 2 numbers in Windows Calculator, a Firefox add-on calculator, Libre Office Calc and MS Excel with interesting resuts :

82.19777664*33.20088775

Results
Windows Calculator = 2,729.039155524212
Firefox Add-on Calculator = 2729.039155524212
Libre Office Calc = 2,729.039155524210
Microsoft Excel 2010 = 2,729.039155524210

As you can see, the calculators agree on the numbers and the spreadsheet programs agree on the numbers, but as you can also see the 2 different types of programs agreed on different numbers - it appears the spreadsheets rounded the last number down to zero.

I allowed for 12 decimal places in the spreadsheet programs, I even added decimal places to 14 or 15 and I got the same results - with additional trailing zero's in the additional decimal places in the spreadsheet programs.

Anyone have any idea how and why these programs are handling the math this way? Thanks for any insight anyone can shed on this riddle

My System SpecsSystem Spec
.

05 Nov 2012   #2

Windows 8.1 Pro (x64)
 
 

Floating point numbers (whole numbers with decimals) is not something that computers are good at. And there could be a lot of reasons why they display them as such.
My System SpecsSystem Spec
05 Nov 2012   #3

Windows 7 Home Premium 64 bit
 
 

Thanks for that reply logicearth, interesting to know
My System SpecsSystem Spec
.


07 Nov 2012   #4
mjf

Windows 7x64 Home Premium SP1
 
 

Actually computers are just as fine with floating point numbers as they are with integers. You are observing roundoff errors (last digit) which is simply due to the number of bits assigned to the floating point number. Typically
32 bit = single precision
64 bit = double precision
128 bit = quadruple precision
etc.
In floating point bits are assigned to the mantissa and exponent. The precision and dynamic range is quite defined. Integer operations can also have roundoff errors.
My System SpecsSystem Spec
Reply

 Interesting Difference in Math Calculations in different programs




Thread Tools



Similar help and support threads for2: Interesting Difference in Math Calculations in different programs
Thread Forum
math quiz Chillout Room
Change File Size Calculations General Discussion
Microsoft: 'Do the math' before you buy a Mac Hardware & Devices
need help with math/finance problem Chillout Room
running Long calculations General Discussion
Math Input Panel General Discussion
Is there a difference between defragging programs? Software

Our Sites

Site Links

About Us

Find Us

Windows 7 Forums is an independent web site and has not been authorized, sponsored, or otherwise approved by Microsoft Corporation. "Windows 7" and related materials are trademarks of Microsoft Corp.

Designer Media Ltd

All times are GMT -5. The time now is 12:16 AM.
Twitter Facebook Google+



Windows 7 Forums

Seven Forums Android App Seven Forums IOS App
  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33