Re: storing an integer in a double precision



On 2011-05-05 17:50:38 -0300, Lynn McGuire said:

On 5/5/2011 12:48 PM, dpb wrote:
On 5/5/2011 12:35 PM, Lynn McGuire wrote:
...

But, do double precision variables actually store 32 bit integers
that were converted but without roundoff ? So roundoff only comes
into play for whole numbers greater than 52 bits ?
...

Well, yes, that's what the mantissa bits are for... :)

Why wouldn't they--I'm thinking there must be something behind the question, here, Lynn.

--

Just a lack of understanding of conversion between integers and
double precision on my part. Back when we were primarily single
precision, we had to be very careful, using "ii = xx + 0.1" to
ensure that we did not loose the whole number value of a
incomplete conversion of 12,000 which was stored in the float
as 11.9999.

I have noticed lately that I have gotten lazy since we converted
the entire program to double precision and don't add the 0.1
conversion factor anymore as we don't seem to need it anymore.

Thanks,
Lynn

Use NINT. NINT beats INT(x+0.5) for being clear as to intent.

Finite representation issues can go either way over time.
The 10 * 0.1 is only approximately 1 issue. You might see
a single precision 0.1 slightly small and a double precison
one slightly large or whatever fate deals out today.

.



Relevant Pages

  • Re: IBM2435I on ROUND(x,-3)
    ... fraction. ... If an intermediate result is necessary, as in evaluation of an operational expression, the attributes of the intermediate result are the same as if a decimal fixed-point value of precision had appeared in place of the string. ... In other words, FIXED DECIMALwould be converted to FLOAT DECIMAL, and in the actual conversion process, any fractional part of the constant expressed in character form would influence the exponent of the converted result and the fractional digits would not actually be lost. ... ROUNDreturns the value of the real expression x rounded at a digit specified by n. ...
    (comp.lang.pl1)
  • Re: IBM2435I on ROUND(x,-3)
    ... fraction. ... If an intermediate result is necessary, as in evaluation of an operational expression, the attributes of the intermediate result are the same as if a decimal fixed-point value of precision had appeared in place of the string. ... In other words, FIXED DECIMALwould be converted to FLOAT DECIMAL, and in the actual conversion process, any fractional part of the constant expressed in character form would influence the exponent of the converted result and the fractional digits would not actually be lost. ... ROUNDreturns the value of the real expression x rounded at a digit specified by n. ...
    (comp.lang.pl1)
  • Re: double to int conversion yields strange results
    ... giving me two different values for the int. ... the question raised is not about the precision of the ... is done in extended precision. ... It is the conversion to 'double' before the conversion to 'int' that ...
    (comp.lang.c)
  • Re: printf and style g
    ... a G conversion specifier), ... Let P equal the precision if nonzero, ... if a conversion with style E would have an exponent ... formatting and a precision of 2. ...
    (comp.std.c)
  • Re: f77 vs f90
    ... On 11/5/12 4:06 PM, glen herrmannsfeldt wrote: ... processor to magically add precision to what is now always a single ... the standard doesn't exactly specify the conversion process from single to double. ... it does specifically define words that have a technical meaning that is either ambiguous or different from the normal every-day meaning. ...
    (comp.lang.fortran)