[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: Precision Errors - Part 3



PureBytes Links

Trading Reference Links

I ran the same test on TS2000i with very similar results.
 
The largest number the PowerEditor would allow was slightly different:

    TS2000i  100,000,000,000,000,008.0063
    TS4.0    100,000,000,000,000,008.0090

The results of the arithmetic are the same up until very large numbers.

It handles the very large numbers better:

TS4.0:   12345678407663620.00     -539223040.00    12884901888.000000000000000
TS2000:  12345678407663616.00    12345678848.00              0.000000000000000
Correct: 12345678901234567.89    12345678901.00              0.234567800000000

TS4.0:   99999998430674940.00     1215750144.00    98784247808.000000000000000
TS2000:  99999998430674944.00    99999997952.00              0.000000000000000
Correct: 100000000000000008.00  100000000000.00              0.000008006300000

So the conclusions are about the same as for TS4.0 - both use single
precision floats.

The test results and code are appended below.

Bob Fulks

----------

Calc:          1234567936.00           1234.00              0.567993000000000
Good:          1234567890.12           1234.00              0.567890123456780

Calc:         12345678848.00          12345.00              0.678711000000000
Good:         12345678901.23          12345.00              0.678901234567800

Calc:        123456790528.00         123456.00              0.789063000000000
Good:        123456789012.34         123456.00              0.789012345678000

Calc:       1234567954432.00        1234568.00              0.000000000000000
Good:       1234567890123.45        1234567.00              0.890123456780000

Calc:      12345679020032.00       12345679.00              0.000000000000000
Good:      12345678901234.56       12345678.00              0.901234567800000

Calc:     123456788103168.00      123456792.00              0.000000000000000
Good:     123456789012345.67      123456789.00              0.012345678000000

Calc:    1234567948140544.00     1234567936.00              0.000000000000000
Good:    1234567890123456.78     1234567890.00              0.123456780000000

Calc:   12345678407663616.00    12345678848.00              0.000000000000000
Good:   12345678901234567.89    12345678901.00              0.234567800000000

Calc:   99999998430674944.00    99999997952.00              0.000000000000000
Good:  100000000000000008.00   100000000000.00              0.000008006300000

Calc:          1000000128.00           1000.00              0.000122070000000
Good:          1000000100.00           1000.00              0.000100000111111