Question about precision

From: Paul Van Delst (paul.vandelst_at_noaa.gov)
Date: 08/25/04 

Date: Wed, 25 Aug 2004 10:41:40 -0400

Hello,

Apologies if the subject line is a bit vague.

I have code where I need to interpolate several instrument response functions to the same
frequency grid and begin and end frequencies. The data I read in from file is at a set
spacing, 0.1cm^-1, but after computing that value from the begin and end frequency and the
number of points, the value is always something like 0.1000000000000002cm^-1. This has
caused problems since the extra 0.0000000000000002 makes a test that I use to ensure the
interpolation is alway within the actual data (extrapolation is a no-no here), fail. So,
what I've done is the following:

   REAL( fp_kind ), DIMENSION( : ), ALLOCATABLE :: Begin_Frequency
   REAL( fp_kind ), DIMENSION( : ), ALLOCATABLE :: End_Frequency
   INTEGER, DIMENSION( : ), ALLOCATABLE :: n_Points
   REAL( fp_kind ), DIMENSION( : ), ALLOCATABLE :: Max_Delta_Frequency
   INTEGER :: dF_Exponent
   INTEGER :: dF_Scaled
   REAL( fp_kind ) :: dF_Multiplier

where fp_kind == Selected_Real_Kind(15) and all the allocatable arrays have been allocated
to the same size and filled with data and Max_Delta_Frequency is computed using,

       Max_Delta_Frequency = ( End_Frequency - Begin_Frequency ) / &
       ! -----------------------------------
                                REAL( n_Points - 1, fp_kind )

At this point I get the 0.1000000000000002 value. To ensure I get the
0.1000000000000000cm^-1 I want (printing out to 16dp), I do the following, looping over
every element of Max_Delta_Frequency using the loop index, l,

     dF_Exponent = INT( ABS( LOG10( Max_Delta_Frequency(l) ) ) ) + 2
     dF_Multiplier = TEN**dF_Exponent
     dF_Scaled = INT( Max_Delta_Frequency(l) * dF_Multiplier )
     Max_Delta_Frequency(l) = REAL( dF_Scaled, fp_Kind ) / dF_Multiplier

Is there a simpler way of "truncating" the 0.0000000000000002 from the number? When I
compute the actual frequency grid (Frequency array), the last frequency is
695.1000000000001cm-1 rather than the expected 695.1000000000000 (the "End_Frequency").
This is causing the test in the interpolation function

     IF ( Frequency(1) < SRF%Begin_Frequency .OR. &
          Frequency(n_Frequencies) > SRF%End_Frequency ) THEN

to fail. Printing out the values

print *, Frequency(1), Frequency(n_Frequencies)
print *, SRF%Begin_Frequency, SRF%End_Frequency

gives

  665.1000000000000 695.1000000000001
  663.7000000000000 695.1000000000000

Is there a

Relevant Pages

  • Re: Question about precision
    ... >>I have code where I need to interpolate several instrument response ... >>functions to the same frequency grid and begin and end ... I could get data at a spacing of 0.135cm-1 or 0.0025cm-1 spacing. ... And, besides, the calculations themselves (interpolation and integration) are performed ...
    (comp.lang.fortran)
  • Re: Why pixel peeping (usually) makes no sense
    ... John Navas wrote: ... To produce an 11x14 print without interpolation from your FZ8, you can only print at 200ppi. ... Printing at 216dpi is only going to ...
    (rec.photo.digital)
  • Re: Why pixel peeping (usually) makes no sense
    ... To produce an 11x14 print without interpolation from your FZ8, ... Printing at 216dpi is only going to ... Never let reality get in the way of a meaningless hairsplitting rant! ...
    (rec.photo.digital)
  • Re: Why pixel peeping (usually) makes no sense
    ... cmyk wrote: ... To produce an 11x14 print without interpolation from your FZ8, you can only print at 200ppi. ... Printing at 216dpi is only going to ...
    (rec.photo.digital)