Precision rounding

From: Olivier Sauterel (sauterel_at_ioware.ch)
Date: 02/24/04 

Date: Tue, 24 Feb 2004 15:25:05 +0100

Hi,
I' ve some problem with my round routine, because in test "tmp >= precision
/ 2" does not work in sample:
RoundMultiple(0.725, 0.01)
It will return 0.72 because tmp after Modulo represent 0.00499999999996 and
should
be 0.005 ! And i think the problem is the same with expression "precision /
2".

I've already try to set the PrecisionMode with pmDouble, and it will work in
some case, but not everytime ...
Anyone can help me and maybe explain me why i've this problem ?

Thanks

Olivier
---------------------------------------------------------------
  function RoundMultiple(value, precision: Double): Double;
  var
    tmp: Double;
  begin
    tmp := Modulo(value, precision);
    if tmp <> 0 then
    begin
      result := value - tmp;
      if tmp >= precision / 2 then
        result := result + precision;
    end
    else
      result := value;
  end;
---------------------------------------------------------------
  function Modulo(value, divisor: Double): Double;
  begin
    result := value - Trunc(value / divisor) * divisor;
  end;
---------------------------------------------------------------

Relevant Pages

  • Re: Reading unformatted files
    ... I am no working on a code that uses single precision ... The saved file has been saved using the following ...  WRITE tmp ...
    (comp.lang.fortran)
  • Reading unformatted files
    ... I am no working on a code that uses single precision ... The saved file has been saved using the following ... WRITE tmp ...
    (comp.lang.fortran)
  • Re: Precision and Scale innacuracies when dividing
    ... Books Online, under the topic "Precision, Scale, and Length". ... arithmetic with decimals causes overflow when it shouldn't. ... > dividend DECIMAL, ... > divisor DECIMAL ...
    (microsoft.public.sqlserver.programming)
  • Precision and Scale innacuracies when dividing
    ... When numbers of high precision are being divided they can come up completely ... INSERT INTO #bad_precision (dividend, divisor) ... -- The quotient should be: ...
    (microsoft.public.sqlserver.programming)
  • Re: Precision and Scale innacuracies when dividing
    ... It shows that reducing the precision ... INSERT INTO #values (dividend, divisor) ...
    (microsoft.public.sqlserver.programming)