Re: Rounding error when doing float-to-integer

From: José Rui Faustino de Sousa (jrfsousaYETFOC_at_esoterica.pt)
Date: 10/16/03 

Date: Thu, 16 Oct 2003 22:05:48 +0100

"Richard Maine" <nospam@see.signature> wrote in message
> In particular, there is no "round to seven
> significant figures" step. What there is instead is a convert to
> binary step.
>

Yes you are both right. I was wrong.

The fact that it perfectly explained the results and that a few extra
tests I run seemed to confirm this blinded me to more carefully check
the binary representation.

> The rounding is to 24 binary digits. That amounts to
> approximately 7 decimal digits, give or take some, depending. But
> the actual rounding is in binary. Yes, it matters.
>

I know my whole case was based on that.

> In particular,
> as Glenn mentioned in this case, there is probably *NOT* any rounding
> error in the conversion of the integer to real, where you posit.
>

No. I said that the rounding error was due to first rounding to seven
decimal digits and then converting to binary.

Well at least that was what I intended to say :-(

> Instead, the rounding error almost certainly comes from the
> conversion of the 0.002 to binary, which you seem to be ignoring.
>

No I am not ignoring it, the rounding error alone does not seem to
explain the result it had to be a larger error.

Even if it was 2.0000001E-3 it would not give 30641.

Now returning to the problem I will advance a new explanation what is
happening is:

int(real(dprod(real(c,sp),b1),sp))

That is the double precision product of the single precision arguments
is then rounded back to single precision and then converted to integer.

The fact that real() rounds to nearest would explain the unexpected
rounding up.

I hope this time I got it right ;-)

Best regards
José Rui 

-- 
========================================================================
"To surrender to ignorance and call it God has always been premature,
and it remains premature today." Isaac Asimov
========================================================================
mailto://jrfsousaYETFOC@esoterica.pt/ Crawler baith. UnCaps me to reply.
Contact information: http://homepage.esoterica.pt/~jrfsousa/contact.html
========================================================================

Relevant Pages

  • Re: converting points to centimeters...
    ... I suppose that Word's rounding to the nearest ... I'm not comfortable with the rounding because it's not controllable - it's ... solution has to be Word 97 compatible and I don't believe there's a Round ... > The problem is not the conversion itself, but the act that Word rounds the ...
    (microsoft.public.word.vba.general)
  • Re: rounding
    ... that by the BLR using ROUND(A*BLR,2) I get 8104.93 ... The difference is large because in the first case, any rounding error is small relative to the unrounded result, whereas in the second case, the rounding error is larger relative to the unround result. ...
    (microsoft.public.excel.misc)
  • Re: XQ and ->Qpi bug on large X
    ... you shouldn't be rounding to only two digits ... I am trying to make a similar point here about rounding; ... and round it, ... prior to using the input values in calculations. ...
    (comp.sys.hp48)
  • Re: My Completed RATECRYPT() function included below
    ... I was interested because most rounding functions do not round in an unbiased manner. ... I have seen various reports of clipper rounding problems but there are many different rounding methods and clipper chose a common one. ... The gaussian method (an unbiased estimator) gives the same answer you would get using a standard c compiler and printed with printf. ...
    (comp.lang.clipper)
  • Re: XQ and ->Qpi bug on large X
    ... plot of all specific values resembling a "normal distribution" ... we should round each value ... which fails if we start computing "mean square" ... the moment we begin rounding data, we make the data less reliable ...
    (comp.sys.hp48)
Loading