Re: Precision Errors

["Kent Rollins" <kentr@xxxxxxxxxxxxxx>]

As much as I hate to perpetuate this discussion, I thought I would
contribute this: when combining 2 floats, you have to consider relative
magnitude.  Although a float (single prec) can range from 1e+38 to 1e-38, if
you try to add a number like 0.0000000000001 to 1,000,000,000,000,000, it
will have either no effect or an unexpected effect because a float cannot
represent both numbers at the same time.  It is too many places of
precision.  When you try to add/subtract floats that have closer magnitudes,
the result will only be reflected to the extent that the places of precision
overlap.  For example, if you add 0.000111111111111 to 1,000,000,000 you
might end up with something like 1,000,000,000.1.

These are extreme examples but this is how precision is lost across repeated
math operations.

There is also another factor that must be taken into account and that is the
method that TradeStation uses to convert a float to ASCII.  For these
experiments, it doesn't matter how accurately the number is stored
internally, if the conversion to human-readable form screws something up.

Kent


----- Original Message -----
From: "Bob Fulks" <bfulks@xxxxxxxxxx>
To: <omega-list@xxxxxxxxxx>
Sent: Saturday, July 21, 2001 4:47 PM
Subject: RE: Precision Errors


Sent privately:

>Whoops Bob, I believe you are wrong on that last one....
>the single prec floating spec has a limit of
>+/-2,147,000,000 or something like that.....
>effectively 9 digits of precision....give or take.
>Pierre suggested 7 9's.....when in fact 9 9's would have been the correct
>test of the limit.


I was going from a post by Gary Fritz to the Code List a while back.
It seems correct to me... See attached.

I did some checking and Gary is correct. The IEEE floating point
format is described in detail at:

   <http://www.psc.edu/general/software/packages/ieee/ieee.html>

But in any case Pierre's test is for an absolute value when, in fact,
the exponent can be as high as 2^128 which about 3.40 * 10^38 (a very
large number).

Bob


At 10:02 AM -0600 7/2/01, Gary Fritz wrote (Code List):

>As background:  single-precision floats have a 24-bit mantissa.  That
>is, they use 24 bits to represent the base of the number, and 8 bits
>for the exponent.  24 bits can represent a signed value in the range
>+/-8388608 or so, or not quite 7 digits of precision.  You can
>calculate the decimal (base 10) digits available with the formula
>
>  (MatissaBits - 1) * log10(2)
>
>MantissaBits is 24, and we must subtract one bit for the sign of the
>mantissa.  That formula says there are 6.92 decimal digits available.