Re: Calculate the precision of a floating point number (ie: the number of decimal places)
Ed Prochak wrote:
On Oct 30, 10:21 am, Lori <steve.lori...@xxxxxxxxx> wrote:
Hi all
I'm trying to write a reliable and efficient library function that
will calculate the precision of an input number (floating point).
IOW number of significant digits?
First, can you tell me how many significant digits exist in these
numbers
0.01e-21
0.00
0.000000
How are the last two going to look any different from each other in
floating point?
- Logan
.
Relevant Pages
- Re: This calculation is just wrong / computer cant count!
... This is because of how 64-bit floating point arithmetic works. ... Any finite precision arithmetic, whether it is ... significant digits yourself. ... So we have units of inches, feet, ...
(microsoft.public.vc.mfc) - Re: Floating-point arithmetic in CL
... When pocket calculators were introduced in French college, ... has a decimal point, but it's not a floating point, it's a fixed ... or 4 significant digits. ... The point is that the 3 significant digits of a slide rule are enough ...
(comp.lang.lisp) - Re: Bug in Excel 2007
... number of significant digits (varies depending on the size of the data ... which fits with the double data type in C. ... nor with the IEEE standard for floating point. ... The bug is from the FPU, excel just converts its own algorithm to ...
(microsoft.public.excel) - Re: how do I get more numbers past the decimal?
... A floating point number is stored on a 32 bit machine using the 24 bit for the mantissa and its sign and 8 bits for the exponent and its sign. ... You are showing 12 significant digits. ... A float value can be exactly represented by up to 9 decimal digits. ... this is only accurate for binary32 format. ...
(comp.lang.php) - Re: Why do simple floating point calculations work ?
... > Skybuck Flying wrote: ... Suppose we define cell1 as: ... But because CellWidth is a floating point and position is a floating point. ...
(alt.comp.lang.borland-delphi) |
|
