Re: Tough problem...

You should be permanently aware of the fact that the FPU does not use
a decimal representation, but a binary notation. Multiplication by
1000 is not really that simple in binary; 1024 for example is simpler.

Actually, it's just as easy to demonstrate with decimals. Suppose you
have an FP unit which calculates in decimals, with five digits of
accuracy. We want to calculate the remainder of dividing by 2*Pi.

For this FPU pi equals 3,1416 and therefore 2*Pi equals 6,2832. We will
calculate the remainder for 10 and for 1000:

Rem( 10, 6,2832) = 10 - 1 * 6,2832 = 3,7168
Rem(1000, 6,2832) = 1000 - 159 * 6,2832 = 0,9712

The real answers, without intermediate rounding to 5 positions are:

Rem( 10, pi) = 3,7168
Rem(1000, pi) = 0,9735

We see that the difference is caused by intermediate rounding to five
decimal digits. This is unavoidable, since the FPU cannot do arithmetic
with larger numbers. For example, it doesn't know a more accurate
version of pi, which it would need to reach the same accuracy for both.

--
The Fastcode Project: http://www.fastcodeproject.org/
.

Relevant Pages

  • RE: prePEP: Decimal data type
    ... > the rounding procedure to be used in computations involving Euros ... implementing a fixed number of decimal digits "after the decimal point" ... To convert from the euro to the national currency, ... currency should then be rounded to two decimals. ...
    (comp.lang.python)
  • Re: Round off
    ... rounding to two decimal places which necessarily gives a String. ... The OP asked about rounding a number to 2 decimals - not about converting it ... Number 3.5 is a multiple of ... FAQ 4.6 can be correspondingly changed. ...
    (comp.lang.javascript)
  • Re: Decimal
    ... It is even calculating the decimals ... You can see that by clicking on a cell you entered data in and looking at the formula bar... ... You can truncate off any decimal value as if they were never typed in by the user in the first place, or you can round off the typed in value, but then there are two ways to do rounding in a macro, so you have to make sure you specify the right one. ...
    (microsoft.public.excel.misc)
  • Re: Need help with rounding on a calculation
    ... a multiple 0f 0.01 and rounding to two decimals. ... Showing your ineptness in following good advice is one thing, ...
    (comp.lang.javascript)
  • Re: a print bug?
    ... the first version will favor the correct rounding, ... so even if you round it off, it is still represented the same way.... ... how "print" handles 1.2345 and treating it and printing it as ... If you use 4 decimals after the decimal sign, you get the same result as with a plain print statement. ...
    (comp.lang.python)