Re: variable value rounding...




Am Mon, 11 Dec 2006 10:40:37 -0800 schrieb Mark Janssen:

Can anybody help me understand how this expression is executed such
that resulting value contains a 2 in the end instead of 1.

How can I avoid this?

% puts [expr 2.0 + 0.23385937500000001] 2.2338593750000002

Thanks,Short answer, you can't.
Longer answer, the reason that you can't avoid this is that floating
point numbers cannot always be represented exactly in binary leading to
roundoff error. [http://en.wikipedia.org/wiki/Floating_point] this
makes precise calculations using floating point on computers a tricky
business. If you really need this precision you can change to using
ints (Tcl 8.5 bigints for example)

Mark

Another nice writeup on the subject can be found at
[http://wiki.tcl.tk/11969]


Thanks Mark for your helpful reply. I am new to tcl and can't solve this
problem:

%set x 2.0
%set y 0.23385937500000001
%puts [expr $x +$y]

How would I typecast $x and $y or $z to bigfloat/double/bigints? My goal
is to have a result that gives me the precision required to sum up the
above mentioned $y (with 17 number after point) and $y without loss.

Is it feacible?

Thanks,

--
Luqman

.



Relevant Pages

  • Re: This calculation is just wrong / computer cant count!
    ... developed to HOW TO AVOID THE PROBLEM IN THE FIRST PLACE not WHY IS THIS ... The only way to avoid the problem is to not use the IEEE floating ... actually the precision of 25.0/30.0 but the change of sign of a "zero" ... The Microsoft article above cites a use of EPSILON and DBL_EPSILON ...
    (microsoft.public.vc.mfc)
  • Re: This calculation is just wrong / computer cant count!
    ... developed to HOW TO AVOID THE PROBLEM IN THE FIRST PLACE not WHY IS THIS ... The only way to avoid the problem is to not use the IEEE floating ... actually the precision of 25.0/30.0 but the change of sign of a "zero" ... The Microsoft article above cites a use of EPSILON and DBL_EPSILON ...
    (microsoft.public.vc.mfc)
  • Re: Precision
    ... Whether one does the subsequent calculations in internal ie integer format ... I have always used external format with Precision 4 since my ... floating point calcs which are done in binary have around 3% error albeit at ... positions it is IMPOSIBLE to have a result of 4 decimals with the 2 ...
    (comp.databases.pick)
  • Re: Linear Algebra Challenge
    ... Since I'm using floating point, so I'll never be able to calculate one ... floating point math set to 99 digits. ... As close as I'm willing to wait if I use arbitrary precision. ... This mode is fast; when you select arbitry ...
    (comp.sys.hp48)
  • Re: Precision
    ... It's the same if we use Precision, floating point, or ... positions it is IMPOSIBLE to have a result of 4 decimals with the 2 ... May be something wrong in the floating ... remember what the associated actuarial calculations were. ...
    (comp.databases.pick)