Re: numeric issue: 0.95 - 0.05 == 0.8499996

Christian <fakemail@xxxxxx> wrote:

but around 1 the precision of floats should be much higher .. I could
understand this "mistake" if he subtracted 3 times 0.05f

This is about the inability to accurately represent decimal numbers
in binary format. 8/9 is close to 1 - not a large or small number.
But, you can not represent it in decimal format with a finite number
of digits. In the same way, you can not represent various finite decimal
numbers in binary with a finite number of 1's and 0's.

- Kurt
.

Relevant Pages

  • Re: numeric issue: 0.95 - 0.05 == 0.8499996
    ... understand this "mistake" if he subtracted 3 times 0.05f ... in binary format. ... the expected precision.. ...
    (comp.lang.java.programmer)
  • Re: Raw data
    ... a 3rd party application. ... converted into a binary format. ... more digits are required. ... Gene Wirchenko ...
    (comp.databases.ms-sqlserver)