Re: Square root of a negative rral value
- From: Glen Herrmannsfeldt <gah@xxxxxxxxxxxxxxxx>
- Date: Wed, 05 Nov 2008 16:59:38 -0700
Terence wrote:
I found that running the converted program with the original check
data set that went with this, crashes on trying to take the square
root of a negative variable (to get the standard deviation as a
measure for selecting later optimum adjustments).
I was utterly lost to discover why all the many computational results
checked exactly to some 6 places, EXCEPT for one standard error and
its significance (both unused later) and the program crashing on one
square root operation before terminating correctly with a final table.
I have seen this many times. Note that six places isn't enough,
if the result is negative in the seventh place.
When the standard deviation is zero, or very close to zero
it is possible for rounding in the variance to give a
negative result. Probably better to set it to zero
instead of the sqrt() of the absolute value, but it will be
very small in any case.
See http://en.wikipedia.org/wiki/Standard_deviation#Definition_and_calculation
The popular standard deviation algorithm uses an identity that is
true mathematically, but not always numerically. That is, the
mean of the squares minus the square of the mean should be positive
but with some types of rounding, it isn't.
-- glen
.
- References:
- Square root of a negative rral value
- From: Terence
- Square root of a negative rral value
- Prev by Date: Square root of a negative rral value
- Next by Date: Re: Square root of a negative rral value
- Previous by thread: Square root of a negative rral value
- Next by thread: Re: Square root of a negative rral value
- Index(es):
Relevant Pages
|
