Precision on size int in C...



The output of 2 to the 30 for a signed int in C on my 32 bit machine
is:

1073741824

The question is how does my computer get 10 digits of precision?

I thought it might be something like

2^K = 10^N
N = K Log (2) //log is base 10

If I let 1 bit for the sign, then K = 31 bits. Plugging 31 into the
above formula yields N roughly equals 9.33.

Now if I let K = 32, N is about 9.63.

In either case, it seems I'm coming out to 9 digits of precision.

Chad






.



Relevant Pages

  • Re: float bug? perl 5.8, DBI and oracle 10.2.0
    ... precision numbers in oracle, you've got 38 decimal digits to play ... and with minimal coaxing perl will handle them as ... digits from a 32 bit floating point number - I'll go out on a limb ... and hazard that one can expect 12 or so digits from a 64 bit floating ...
    (perl.dbi.users)
  • Re: keeping precision when parsing number from string
    ... But in some cases all the digits past some point are zero, ... that no default bound on precision will be good in ... It seems the floating-point default is for optimization ... be as general-purpose as is possible. ...
    (comp.lang.lisp)
  • Re: float bug? perl 5.8, DBI and oracle 10.2.0
    ... I would not characterise 32-bit signed integers as giving 10 digits ... truncate and tell people you get 9 digits of precision. ... perl 5.8, DBI and oracle 10.2.0 ... Floating point values are typically stored in 64 bits or sometimes 96 ...
    (perl.dbi.users)
  • RE: float bug? perl 5.8, DBI and oracle 10.2.0
    ... I would not characterise 32-bit signed integers as giving 10 digits ... truncate and tell people you get 9 digits of precision. ... perl 5.8, DBI and oracle 10.2.0 ... Floating point values are typically stored in 64 bits or sometimes 96 ...
    (perl.dbi.users)
  • Re: Fun With Numbers
    ... If you want to do exact addition, subtraction, multiplication, ... has reports that generate nine digits worth in some reports. ... referred to the ECMAScript standard (ECMA-262 5.1 Edition of June ... that it is good for 15.95 digits of precision. ...
    (comp.lang.javascript)