Re: Rounding error when doing float-to-integer

From: Richard Maine (nospam_at_see.signature)
Date: 10/16/03

  • Next message: john: "Re: problem with pointer and module" 
    Date: 16 Oct 2003 09:33:13 -0700

    "Glen Herrmannsfeldt" <gah@ugcs.caltech.edu> writes:

    > "José Rui Faustino de Sousa" <jrfsousaYETFOC@esoterica.pt> wrote in message
    > news:bmm3eu$o4425$1@ID-134488.news.uni-berlin.de...

    > > data->round (seven significant figures)....

    > If you don't do it in binary, I don't think you can get the right answer.

    Agree with Glenn here. In particular, there is no "round to seven
    significant figures" step. What there is instead is a convert to
    binary step. The rounding is to 24 binary digits. That amounts to
    approximately 7 decimal digits, give or take some, depending. But
    the actual rounding is in binary. Yes, it matters. In particular,
    as Glenn mentioned in this case, there is probably *NOT* any rounding
    error in the conversion of the integer to real, where you posit.
    Instead, the rounding error almost certainly comes from the
    conversion of the 0.002 to binary, which you seem to be ignoring.

    The "7 decimal digits" bit is only a rough approximation that allows
    you to gauge the approximate magnitude of the possible rounding
    errors. In this case, since errors in the 9th significant decimal
    digit would change the result, you can see that it is plausible for
    rounding to introduce such an error. But if you want to compute the
    actual numbers intead, you have to do it in binary. 

    -- 
Richard Maine                       |  Good judgment comes from experience;
email: my first.last at org.domain  |  experience comes from bad judgment.
org: nasa, domain: gov              |        -- Mark Twain
  • Next message: john: "Re: problem with pointer and module"

    Relevant Pages

    • Re: Deviation of double
      ... int iDigits); ... I believe you're confused about rounding. ... round to 90.62 if only two decimal digits are required. ... Why are you expecting 90.625? ...
      (comp.lang.cpp)
    • Re: This calculation is just wrong / computer cant count!
      ... BCD also has the advantage of providing more significant decimal ... arithemtic, AS IMPLEMENTED ON THE PENTIUM FPU, as demonstrated by my Floating Point ... If you can't understand that using binary to represent decimal digits ... properly rounding. ...
      (microsoft.public.vc.mfc)
    • Re: Arithmetic operation bug
      ... Manoj Chourasia wrote: ... decimal digits are shown, taking rounding into account. ... doesn't round it), ...
      (comp.lang.ruby)
    • Re: How to display numbers when VB6 wants to say "1E8" ?
      ... Thank you, that works (albeit with some rounding, but 6 significant figures is easily accurate enough for this data). ...
      (comp.lang.basic.visual.misc)
    • Re: Meaning of INT(x) function for negative numbers
      ... For those interested in the code for my original problem of "rounding ... real*8 function round_real(rr, sig) ... round a double to a certain number of significant figures ...
      (comp.lang.fortran)