Windows 7 Forums
Welcome to Windows 7 Forums. Our forum is dedicated to helping you find support and solutions for any problems regarding your Windows 7 PC be it Dell, HP, Acer, Asus or a custom build. We also provide an extensive Windows 7 tutorial section that covers a wide range of tips and tricks.

Windows 7: Interesting Difference in Math Calculations in different programs

05 Nov 2012   #1

Windows 7 Home Premium 64 bit
Interesting Difference in Math Calculations in different programs

I noticed a difference in the results of a math problem I did in 4 separate programs, and they split 2 to 2 on the result. I multiplied the following 2 numbers in Windows Calculator, a Firefox add-on calculator, Libre Office Calc and MS Excel with interesting resuts :


Windows Calculator = 2,729.039155524212
Firefox Add-on Calculator = 2729.039155524212
Libre Office Calc = 2,729.039155524210
Microsoft Excel 2010 = 2,729.039155524210

As you can see, the calculators agree on the numbers and the spreadsheet programs agree on the numbers, but as you can also see the 2 different types of programs agreed on different numbers - it appears the spreadsheets rounded the last number down to zero.

I allowed for 12 decimal places in the spreadsheet programs, I even added decimal places to 14 or 15 and I got the same results - with additional trailing zero's in the additional decimal places in the spreadsheet programs.

Anyone have any idea how and why these programs are handling the math this way? Thanks for any insight anyone can shed on this riddle

My System SpecsSystem Spec
05 Nov 2012   #2

Windows 10 Pro (x64)

Floating point numbers (whole numbers with decimals) is not something that computers are good at. And there could be a lot of reasons why they display them as such.
My System SpecsSystem Spec
05 Nov 2012   #3

Windows 7 Home Premium 64 bit

Thanks for that reply logicearth, interesting to know
My System SpecsSystem Spec

07 Nov 2012   #4

Windows 7x64 Home Premium SP1

Actually computers are just as fine with floating point numbers as they are with integers. You are observing roundoff errors (last digit) which is simply due to the number of bits assigned to the floating point number. Typically
32 bit = single precision
64 bit = double precision
128 bit = quadruple precision
In floating point bits are assigned to the mantissa and exponent. The precision and dynamic range is quite defined. Integer operations can also have roundoff errors.
My System SpecsSystem Spec

 Interesting Difference in Math Calculations in different programs

Thread Tools

Similar help and support threads
Thread Forum
A math quiz
You've seen a SSD for $97. Since you have no money you borrow $50 from your dad and $50 from your mom. $50 + $50 = $100 right? You buy the SSD and get $3 back. You give your dad and mom $1 each and $1 you keep. So you now owe your parents $49 each. $49 + $49 = $98 + your $1 = $99, right?...
Chillout Room
math quiz
a boy was very naughty his teacher gave him an assignment to keep him busy that is add all the numbers from 1 to 100 in couple of minutes he came back with the answer 5,050 that is he multiplied 50 * 101
Chillout Room
Change File Size Calculations
It appears windows does not cnform to the standard method of calculating file size. e.g. the standard value of 1GB is 1000MB, however windows uses 1024MB. As such it is not in fact a "Gigabyte" But a "Gibibyte". This is extremley annoying, as for egxample, I have a 2TB hard Drive, which in the...
General Discussion
running Long calculations
Just upgraded to W7. This problem arises, which I didn't have in XP: I run a program (one that I've developed) that does a huge number of numerical calculations. To reach its conclusion and give the answer takes hours and hours. Best to start it before bedtime, hoping to have the answer in the...
General Discussion
Is there a difference between defragging programs?
Hi, I was just wondering if there is a difference between disk defrag programs? Is there anyone that isnt as good as the other? I would like to know between Iobit smart defrag, Tuneup Utilities 2009 defrag, diskeeper and auslogic disk defrag. Which one is better or doesnt it make a difference?...

Our Sites

Site Links

About Us

Find Us

Windows 7 Forums is an independent web site and has not been authorized, sponsored, or otherwise approved by Microsoft Corporation. "Windows 7" and related materials are trademarks of Microsoft Corp.

Designer Media Ltd

All times are GMT -5. The time now is 15:56.
Twitter Facebook Google+