Re: Question about precision

From: Keith Refson - real email address in signature (nobody_at_nowhere.uk)
Date: 08/25/04 

Date: Wed, 25 Aug 2004 16:46:45 +0100

Paul Van Delst <paul.vandelst@noaa.gov> writes:

> I have code where I need to interpolate several instrument response
> functions to the same frequency grid and begin and end
> frequencies. The data I read in from file is at a set spacing,
> 0.1cm^-1, but after computing that value from the begin and end
> frequency and the number of points, the value is always something like
> 0.1000000000000002cm^-1. This has caused problems since the extra
> 0.0000000000000002 makes a test that I use to ensure the interpolation
> is alway within the actual data (extrapolation is a no-no here),
> fail.

Might I suggest a different approach. Your problem arises because
your grid spacing of 0.1 cm^-1 has no exact representation in binary
floating point. But the numerical value depends on the units
and can be changed to suit. If your program used internal units of
0.1cm^-1 then the grid spacing would be 1.0 which *is* exactly representable.
In fact you could use integers which would eliminate representation
and rounding errors entirely.

Keith Refson 

--
Dr Keith Refson, 
Building R3
Rutherford Appleton Laboratory
Chilton
Didcot                       kr AT                        
Oxfordshire OX11 0QX         isise D@T rl D.T ac D?T uk

Relevant Pages

  • Re: Free / Opensource Grid XML editor
    ... What XML markup language are you trying to put this data into? ... There is no single universal representation for "a grid" in XML; it may be anything from HTML-like rows of data fields, to columns, to various sparse representations, to things that aren't obviously grids at all but that can be represented that way. ...
    (comp.text.xml)
  • Re: Change default design grid spacing?
    ... "Jeanette Cunningham" wrote: ... Create a new form and set its grid spacing to your specs. ... properties every time I want to design a new form. ...
    (microsoft.public.access.tablesdbdesign)
  • Re: Grid Lines Do Not Appear Even When Grid is Selected
    ... Grid Spacing is set to Fine and the Minimum spacing and Grid origin is set to ... I do not see a density setting. ... "Paul Herber" wrote: ...
    (microsoft.public.visio)
  • Re: Change default design grid spacing?
    ... Create a new form and set its grid spacing to your specs. ... in the box for Form Template type in the name of that form with the grid set ... properties every time I want to design a new form. ...
    (microsoft.public.access.tablesdbdesign)
  • Re: Access 2007 grid problem ????
    ... Does anyone know if the grid spacing is changeable? ... The Snap to grid possibly has some issues? ... I bet it's look great when it's finished Bob, I loved those Rubiks Cubes! ... The text boxes are supposed to be adjacent to each other ... ...
    (microsoft.public.access.reports)