Re: Newbie Question

"Oliver Wong" <owong@xxxxxxxxxxxxxx> wrote in message
news:S1kfg.223$I61.46@xxxxxxxxxxx

It takes 34 bits to store an unsigned 10 digit number, and 35 bits to
store a signed 10 digit number, so you're right that LONG INTEGER (32 bit
signed) won't be enough to store all values of the PIC 9(10). However, if
you move to floating point numbers (e.g. SINGLE or DOUBLE), you're opening
yourself up to problems with rounding, which may also yield big headaches.

What additional problems could there be ..considering this application was
already MOVEing a PIC 9(10) field to a PIC 9(1) field.... seems to me any
rounding problems introduced with floating point numbers are kind of
immaterial .....

(Besides, a DOUBLE has 16 decimal digits of precision).


MCM










.

Relevant Pages

  • Re: Precision of real numbers
    ... defined to have 15 digits of precision. ... that is the 16th digit. ... arithmetic expression involving real variables of that kind will be ... floating point" to look for a popular paper on floating point arithmetic ...
    (comp.lang.fortran)
  • Re: How to build a quantum computer and time machine.
    ... need a math system to store information say in a number no bigger or ... and get a five digit number. ... Our (time machine map) would look like this x, y, z, t, n ... Such a math system described here as N would have to do with an object ...
    (sci.physics)
  • Re: How to build a quantum computer and time machine.
    ... and get a five digit number. ... We can do this for as long as their is information to store forever ... Lloyd Dudley Burris ... Such a math system described here as N would have to do with an object ...
    (sci.physics.relativity)
  • Re: precision problems in base conversion of rational numbers
    ... > involves use of the decimal module, ... > convert them to any base between 2 and 36, to desired precision. ... The last digit should be "D". ... >>> # Convert rational to floating point with 300 bits precision. ...
    (comp.lang.python)
  • Re: This calculation is just wrong / computer cant count!
    ... extra digit on the end that it can't handle and more importantly - how ... "floating point" in decimals: guess you have 10 digits, ... The computer should surely only store as ... rounding at the last stage, so my final calculations are rounded ...
    (microsoft.public.vc.mfc)