|
PureBytes Links
Trading Reference Links
|
Ken:
Agreed, the basic word size is defined by the computer, but software can
generate and do calculations in longer word sizes, like double precision and
long precision. There is software that can generate very long multi
precision numbers and perform math operations on them.
Lionel Issen
lissen@xxxxxxxxx
----- Original Message -----
From: PD Manager <pdmanager@xxxxxxxxxxxxxxxxx>
To: <metastock@xxxxxxxxxxxxx>
Sent: Wednesday, September 06, 2000 10:13 AM
Subject: RE: First calculation problem
> Lionel:
>
> This is what I have been saying all along. The errors are there because
of
> hardware. The interface of a program determines the accuracy of the
> DISPLAYED NUMBERS. In another posting, I have given an example of how
even
> Excel shows the basic precision errors. We provided a way for our users
to
> change the accuracy of formulas by using the precision function.
>
> Also, for what it is worth, a word size is defined by the computer and not
> the software written for it. In programming, a word defines the natural
> data size of a computer, or the smallest addressable unit of memory. The
> size of a word varies from one computer to another, depending on the CPU.
> For computers with a 16-bit CPU, a word is 16-bits. On some large
> mainframes a word can be as long as 64-bits.
>
> Ken Hunt
> Programming Manager
> Equis International
>
>
> -----Original Message-----
> From: Lionel Issen [mailto:lissen@xxxxxxxxx]
> Sent: Tuesday, September 05, 2000 9:21 PM
> To: metastock@xxxxxxxxxxxxx
> Subject: Re: First calculation problem
>
>
> Guy:
>
> I spoke with a professional programmer who programs mostly in c/c++. He
said
> that the problem is NOT one of limitations in the programming language,
but
> the way that the interface is designed. Metastock could be designed with
an
> interface that would allow the user to specify whatever precision was
> desired. (I also suspect that the program is using a 16 bit word length.)
> Have you determined the word length in Metastock? Of course a more
flexible
> interface would require more memory, but memory is now relatively cheap.
>
> Regards
> Lionel Issen
> lissen@xxxxxxxxx
> ----- Original Message -----
> From: Guy Tann <grt@xxxxxxxxxxxx>
> To: <metastock@xxxxxxxxxxxxx>
> Sent: Tuesday, September 05, 2000 1:43 PM
> Subject: RE: First calculation problem
>
>
> > Thanks Lionel,
> >
> > Since charting is not our primary concern and basic arithmetic is, we
> might
> > have to bail on MS as much as I'd hate to do it. I'm going to spend
some
> > time in the next week or two to make a decision.
> >
> > Guy
> >
> > " When I die, I want to go peacefully like my grandfather did, in his
> sleep.
> > Not yelling and screaming like the passengers in his car."
> >
> > -----Original Message-----
> > From: owner-metastock@xxxxxxxxxxxxx
> [mailto:owner-metastock@xxxxxxxxxxxxx]On
> > Behalf Of Lionel Issen
> > Sent: Tuesday, September 05, 2000 7:18 AM
> > To: metastock@xxxxxxxxxxxxx
> > Subject: Re: First calculation problem
> >
> > Guy:
> > You have done us a great service by showing why metastock can't/doesn't
> work
> > as we expect it. The problem is very subtle. When similar questions
were
> > raised in the past, Equis denied that anything was wrong with the
program.
> >
> > Now if only TAS were upgraded to a windows program....!
> >
> > This reminds me of one of Murphy's rules: in any calculation, the number
> > that you are most confident that it is correct, is in error.
> >
> > Please keep us posted on your work.
> >
> > Warm regards.
> > Lionel Issen
> > lissen@xxxxxxxxx
> > ----- Original Message -----
> > From: Guy Tann <grt@xxxxxxxxxxxx>
> > To: <metastock@xxxxxxxxxxxxx>
> > Sent: Tuesday, September 05, 2000 12:47 AM
> > Subject: RE: First calculation problem
> >
> >
> > > Lionel,
> > >
> > > Detroit was called Murder City for years. Maybe Chicago inherited the
> > title
> > > posthumously. :)
> > >
> > > No one was more surprised than I was with this "little" error that
crept
> > > into my calculations. With our indicator, this "error" occurred in my
> > test
> > > set near the beginning and added .2 to every subsequent calculation
for
> > that
> > > indicator and impacted all other calculations using that number (C -
L).
> > I
> > > probably would have found it earlier if I had known what I was looking
> > for.
> > > This comes from originally developing our systems using higher level
> > > business languages with good computational capabilities, and then
> > expecting
> > > future language to be able to duplicate these calculations.
> > >
> > > Foolish me, I assumed that MS could subtract a 2 decimal number from
> > another
> > > 2 decimal number and provide me with an answer represented as a two
> > decimal
> > > number. Every other piece of software I own has that capability,
> > regardless
> > > of the precision used. While I understand there are methods to
confuse
> > > programs that use single precision arithmatic, and if I was doing
> anything
> > > sophisticated I could understand how you might possibly have
significant
> > > digit errors occurring in the computation. But we're talking
> subtraction
> > > here! We're talking about taking one number with 2 decimals [PICTURE
> > > 9(4)v99 to use an old COBOL representation] and subtracting another
> number
> > > with 2 decimals from it. I don't really give a flying whatever about
> > > internal representation, I would just like the right answer to
magically
> > > appear. What this means to me is that I have to, in order to play
safe,
> > go
> > > back and check every one of my indicators against one or two other
> > languages
> > > to insure that MS hasn't magically influenced my system. Since we're
> > > talking about trading some serious money here, it means we'll be
running
> > our
> > > parallel systems for a longer than planned.
> > >
> > > My first conversion to MS took almost 6 months several years ago. I
can
> > > just see taking the next four months off doing this one. More and
more
> > I'm
> > > starting to think Walter's right and that I really need to get my act
> > > together, bite the bullet, and move everything into VBA and Excel (and
> > yes,
> > > I did test Excel and it can perform the correct calculation) or just
> plain
> > > VB. I'm going to sit down next week and take a look at my various
> options
> > > and at the different languages I own. I had hoped to avoid this but
> even
> > if
> > > I get this running in MS, I think I'm going to have to pursue other
> > options.
> > > At least I have a test data set for the S&P futures available in MS,
> xBase
> > > and .TXT formats. This will help with the debugging if nothing else.
> > >
> > > Guy
> > >
> > > " When I die, I want to go peacefully like my grandfather did, in his
> > sleep.
> > > Not yelling and screaming like the passengers in his car."
> > >
> > > -----Original Message-----
> > > From: owner-metastock@xxxxxxxxxxxxx
> > [mailto:owner-metastock@xxxxxxxxxxxxx]On
> > > Behalf Of Lionel Issen
> > > Sent: Monday, September 04, 2000 6:47 PM
> > > To: metastock@xxxxxxxxxxxxx
> > > Subject: Re: First calculation problem
> > >
> > > Guy:
> > >
> > > I am also puzzled by this. Some time ago, someone from Equis mentioned
> on
> > > this listserver that Metastock was written in C++ or C. While I do
not
> > > program in C++ or C, any manual on these languages shows that numbers
> can
> > be
> > > declared as double precision and long precision. It follows that
> Metastock
> > > can also have this capability. Perhaps the default precision in
> Metastock
> > > should be double precision instead of single precision.
> > >
> > > Guy, no one should have to go thru these tedious exercises to get
> correct
> > > answers. You have clarified for me why Metastock sometimes produces
> weird
> > > answers that don't seem to conform to reality.
> > >
> > > I thought that Chicago was called Murder City. Has the title passed
to
> > > Detroit?
> > >
> > > Regards
> > > Lionel Issen
> > > lissen@xxxxxxxxx
> > > ----- Original Message -----
> > > From: Guy Tann <grt@xxxxxxxxxxxx>
> > > To: <metastock@xxxxxxxxxxxxx>
> > > Sent: Monday, September 04, 2000 7:28 PM
> > > Subject: RE: First calculation problem
> > >
> > >
> > > > Ken,
> > > >
> > > > Sorry for the delay, but my DSL modem crapped out and then I was
> > visiting
> > > > Murder City until today.
> > > >
> > > > I wasn't typing anything into the computer. I used Equis'
Downloader
> > and
> > > > Reuter's data service to download my data. I then use MS to massage
> it.
> > > > Interestingly, TAS uses the Downloader data as well without any of
> these
> > > > problems. Using OLE to access the data in Excel works properly as
> well.
> > > It
> > > > appears that these other software packages manage to keep track of
the
> > > > correct number of decimal places, whereas MS just sort of goes with
> the
> > > flow
> > > > and takes whatever it might find stored. I really don't care as
long
> as
> > I
> > > > know that accuracy isn't one of MS' strong suits. I can understand
> your
> > > > points, but in working with many programming languages, they all
> manage
> > to
> > > > do something to keep track of significant digits without barfing.
> > Again,
> > > if
> > > > I was doing anything complex, I could understand your point
> completely,
> > > but
> > > > the inability to subtract two numbers and consistently come up with
> the
> > > > correct answer is still hard for me to fathom.
> > > >
> > > > Anyway, here is a sample of some of the code I've had to use to get
> > around
> > > > this in one instance. By the way, none of my other programming
> language
> > s
> > > or
> > > > Excel require these mechanizations. It appears that MS was designed
> as
> > a
> > > > charting or graphics program with computational capabilities added
as
> an
> > > > afterthought. What we need to remember is that if we want to use it
> as
> > > > such, we'll need to work around its deficiencies ourselves.
> > > >
> > > > COMN0:= C - COMMODLOW; {simple subtraction of C - L}
> > > > COMN:= PREC( If( COMN0 > 0 , (COMN0+.005), If( COMN0 < 0 , (COMN0 -
> > .005 )
> > > ,
> > > > COMN0 )) , 2 ); {needed to fix MS inability to maintain significant
> > > digits}
> > > > COMN1:= COMN / COMAR; {simple divide}
> > > > COMN2:= COMN1 - .5; {Simple subtraction}
> > > > COMN3:= COMN2 * 2; {simple multiplication}
> > > > COMN3R:= If( COMN3 > 0 , (COMN3 + .00005 ) ,
> > > > If( COMN3 < 0 , (COMN3 - .00005 ) ,
> > > > 0 ) ); {rounding to 4 decimal places performed manually}
> > > > COMN4:= ( PREC( COMN3R, 4 ) * 10); {take answer to 4 decimals and
> > > multiply
> > > > by 10}
> > > > COMN5:= If( COMN4 > 0 , ( COMN4 ) + .5 , If( COMN4 < 0 , ( COMN4 ) -
> .5
> > ,
> > > > ( COMN4 ) )); {round to nearest whole number}
> > > > COMY:= Int( COMN5 ); {store integer}
> > > >
> > > > This does work, albeit in a slightly confused manner and requires 8
> more
> > > > variables than it should. Using this fixed the one erroneous
> > calculation
> > > I
> > > > had and let me move on for further testing.
> > > >
> > > > OTOH, I am now working on trying to get the Ref() function to work
per
> > the
> > > > manual and will try to get some time to spend on it tonight if I can
> > stay
> > > > awake. I was up at 4AM Detroit time (1AM local time) and have over
> 400
> > > > e-mails to wade through since 8/29 (most of which I deleted).
> > > >
> > > > Guy
> > > >
> > > > " When I die, I want to go peacefully like my grandfather did, in
his
> > > sleep.
> > > > Not yelling and screaming like the passengers in his car."
> > > >
> > > > -----Original Message-----
> > > > From: owner-metastock@xxxxxxxxxxxxx
> > > [mailto:owner-metastock@xxxxxxxxxxxxx]On
> > > > Behalf Of PD Manager
> > > > Sent: Tuesday, August 29, 2000 10:05 AM
> > > > To: 'metastock@xxxxxxxxxxxxx'
> > > > Subject: RE: First calculation problem
> > > >
> > > > Guy:
> > > >
> > > > The precision error occurs as soon as the number is stored in your
> > > computer.
> > > > If you are typing a number into a computer (such as 12.1), as soon
as
> > that
> > > > number is stored in a single precision floating point number in your
> > > > computer, the precision error is there. The number is already
stored
> as
> > > an
> > > > approximation. This is a CPU / hardware issue and is not related to
> the
> > > > software itself.
> > > >
> > > > Most software will mask this error by doing rounding of a floating
> point
> > > > number before it is displayed. In the case of the 1469.3999999, if
> you
> > > ask
> > > > a computer to display that number with two digits to the right of
the
> > > > decimal point, rounding occurs and you will see 1469.40, but the
> number
> > is
> > > > actually stored in the computer as 1469.3999999. This is true even
if
> > you
> > > > entered a number at the keyboard (or downloaded the number or read
it
> > from
> > > > some other source) that was 1469.40.
> > > >
> > > > The difference between MetaStock and your other packages is that the
> > other
> > > > software is performing the rounding before the results are displayed
> on
> > > the
> > > > screen. MetaStock is not doing this and is displaying the numbers
out
> > to
> > > > the maximum possible length. I assure you that if other software
> > packages
> > > > are displaying 1469.40, the internal representation is actually
> > > > 1469.3999999. The approximation and actual storage of the numbers
is
> a
> > > > function of the CPU hardware and not the software package itself.
> > > >
> > > > Ken Hunt
> > > > Programming Manager
> > > > Equis International
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Guy Tann [mailto:grt@xxxxxxxxxxxx]
> > > > Sent: Monday, August 28, 2000 12:34 PM
> > > > To: metastock@xxxxxxxxxxxxx
> > > > Subject: RE: First calculation problem
> > > >
> > > >
> > > > Ken,
> > > >
> > > > I guess I still don't understand what's happening here.
> > > >
> > > > These numbers were downloaded from Reuters and were stored in the O,
> H,
> > L,
> > > C
> > > > data arrays, all handled internally by Equis. My assumption is that
> > these
> > > > numbers were downloaded properly and that they contain only the two
> > > decimal
> > > > places shown in the data and in the data window. Going out and
looking
> > at
> > > > the numbers in Downloader and in the related Chart supports that
> > > assumption.
> > > > Is that an invalid assumption? These numbers only have two decimal
> > places
> > > > to begin with. Did MS somehow managed to "modify" the original
input
> > and
> > > > store them as something like 1469.3999999?
> > > >
> > > > If that's the case, then using Precision in order to insure that
> simple
> > > > arithmetic calculations to maintain the two decimals places appears
to
> > be
> > > a
> > > > requirement.
> > > >
> > > > Generally, there are very few problems occurring, but they are
> > sufficient
> > > to
> > > > throw off some of our results.
> > > >
> > > > We have these calculations running in Clipper, Excel, COBOL, and TAS
> > > without
> > > > problems.
> > > >
> > > > We only have three different levels of precision in our system and
> they
> > > are
> > > > 0, 2, and 4. We either use the Rnd() function or in cases like the
> one
> > > I'm
> > > > working on right now, manually code our own rounding.
> > > >
> > > > Guy
> > > >
> > > > " When I die, I want to go peacefully like my grandfather did, in
his
> > > sleep.
> > > > Not yelling and screaming like the passengers in his car."
> > > >
> > > > -----Original Message-----
> > > > From: owner-metastock@xxxxxxxxxxxxx
> > > [mailto:owner-metastock@xxxxxxxxxxxxx]On
> > > > Behalf Of PD Manager
> > > > Sent: Monday, August 28, 2000 8:15 AM
> > > > To: 'metastock@xxxxxxxxxxxxx'
> > > > Subject: RE: First calculation problem
> > > >
> > > > Isn't programming with floating point numbers fun? <g>
> > > >
> > > > Floating point errors tend to compound as more calculations are
> > performed.
> > > > I could write an entire book on the subject (I'm sure there are
books
> > > > written on just this subject). Typically, floating point numbers
are
> > > > guaranteed to have 6-7 digits of precision. When you start doing
> > > > mathematical operations on these numbers, there are times when some
of
> > > these
> > > > strange issues will creep in.
> > > >
> > > > What I have found is that you should use the precision function only
> if
> > > you
> > > > are trying to compare floating point numbers. Otherwise just stick
> with
> > > the
> > > > standard calculations.
> > > >
> > > > What is amazing is that although your numbers look good to start
with
> > > > (1486.20 and 1469.40) the error is probably already there. The
actual
> > > > numbers stored in the computer may already be something like
> > > (1486.20000001
> > > > or 1469.3999999). Usually when these numbers are prepared for
output
> > > > (display or printed) the software will perform rounding to get them
to
> > > look
> > > > like the numbers you entered. When you perform a mathematical
> operation
> > > on
> > > > these numbers, the error becomes more apparent so that the rounding
> > before
> > > > output didn't make your result look correct.
> > > >
> > > > Virtually any computer that stored floating point numbers has this
> > > problem.
> > > > I worked on flight simulations for several years and the only way we
> > could
> > > > get around the problem was to do EVERYTHING in integer math and keep
> > track
> > > > of assumed decimal points.
> > > >
> > > > Ken Hunt
> > > > Programming Manager
> > > > Equis International
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Guy Tann [mailto:grt@xxxxxxxxxxxx]
> > > > Sent: Sunday, August 27, 2000 1:50 AM
> > > > To: metastock@xxxxxxxxxxxxx
> > > > Subject: RE: First calculation problem
> > > >
> > > >
> > > > Ken,
> > > >
> > > > Are you saying that I need to add the prec() function to each one of
> my
> > > > variable calculations? Does this problem compound when building
> systems
> > > > using calculations upon calculations or will the prec() function
used
> at
> > > > each level take care of the problem?
> > > >
> > > > Guy
> > > >
> > > > Never be afraid to try something new. Remember, amateurs built the
> ark,
> > > > professionals built the Titanic.
> > > >
> > > > -----Original Message-----
> > > > From: owner-metastock@xxxxxxxxxxxxx
> > > [mailto:owner-metastock@xxxxxxxxxxxxx]On
> > > > Behalf Of PD Manager
> > > > Sent: Friday, August 25, 2000 7:35 AM
> > > > To: 'metastock@xxxxxxxxxxxxx'
> > > > Subject: RE: First calculation problem
> > > >
> > > > MetaStock does indeed use single precision floating point numbers.
As
> > you
> > > > mentioned, going to double precision would literally double the
memory
> > > > requirements for data storage for charts and would also slow down
> > > > calculations. When you get into mathematical calculations, however,
> > going
> > > > to double precision doesn't necessarily make the problem better. PC
> > > > computer hardware still cannot accurately store a number as simple
as
> > 0.1
> > > > whether you are using single or double precision. It is stored as
an
> > > > approximation. When it comes to floating point numbers, the
hardware
> > can
> > > > really only accurately store fractional numbers that are powers of
two
> > > (1/2,
> > > > 1/4, 1/8, 1/16 etc).
> > > >
> > > > Other software packages suffer from the same problem (including VB
and
> > > > Excel) although some manage to mask it better than others. If you
> don't
> > > > believe this, I can submit a set of "simple" calculations that will
> > cause
> > > > Excel to show precision errors also.
> > > >
> > > > Some software packages will use other methods to store and/or
> calculate
> > > > floating point numbers. This usually involves something like BCD
> > encoding
> > > > or some type of integer encoded fixed point real numbers. While
this
> > > > ultimately solves the precision problem, it has other problems with
> > speed
> > > of
> > > > calculations as well as a reduced ability to store large or very
small
> > > > numbers.
> > > >
> > > > We have always been aware of this issue and that is why we added the
> > > > precision function to the formula language. It was put there in an
> > > attempt
> > > > to help those writing formulas to work with the precision they
needed.
> > > >
> > > > Ken Hunt
> > > > Programming Manager
> > > > Equis International
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Kent Rollins [mailto:kentr@xxxxxxxxxxxxxx]
> > > > Sent: Friday, August 25, 2000 12:40 AM
> > > > To: metastock@xxxxxxxxxxxxx
> > > > Subject: Re: First calculation problem
> > > >
> > > >
> > > > Looks like you may have hit the old single-precision problem. PCs
> > > basically
> > > > have 3 native ways of storing floating point numbers:
single-precision
> > (4
> > > > bytes), double-precision (8 bytes), and long double (10 bytes). The
> > > problem
> > > > is that each one of these representations has a limited number of
> > > "numbers"
> > > > that it can represent and from time to time you will hit a
calculation
> > > that
> > > > reveals this limitation in all it's splendor. Single-precision
floats
> > can
> > > > represent approximately 4 billion different numbers. That's a lot
> until
> > > you
> > > > consider that between 0 and 1 there are an infinite number of
floating
> > > point
> > > > numbers. Double-precision has many, many more number that it can
> > > represent
> > > > (4 billion times 4 billion) and you RARELY see the kind of error you
> > have
> > > > hit when you are dealing with numbers on the scale of 1486 with only
2
> > > > places of precision. That leads me to suspect that Equis is using
> > > > single-precision numbers for these calculations (Omega does the same
> > > thing).
> > > > Saves memory, SLIGHTLY faster in computation, loses precision.
There
> is
> > > > really no good reason for using singles in an app like this and
there
> is
> > a
> > > > (now obvious) good reason not to. I would scream and yell at Equis.
> > Tell
> > > > Little Guy that if he convinces Equis to use doubles you'll buy him
a
> > pony
> > > > and then drop him off in the programmer's offices.
> > > >
> > > > Ken Hunt, does MetaStock use single precision for these
calculations?
> > > >
> > > > Kent
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Guy Tann <grt@xxxxxxxxxxxx>
> > > > To: Metastock User Group <metastock-list@xxxxxxxxxxxxx>
> > > > Date: Friday, August 25, 2000 1:29 AM
> > > > Subject: First calculation problem
> > > >
> > > >
> > > > List,
> > > >
> > > > Well, I decided to do a little more work and discovered my first
> > problem.
> > > >
> > > > Somehow, MS came up with the following:
> > > >
> > > > 1486.20
> > > > - 1469.40
> > > > 16.7999 instead of the more commonly expected 16.80
> > > >
> > > > Now the first number is the Close and the second number is the day's
> > low,
> > > so
> > > > we can't blame this on any previous calculation or anything left
over
> > from
> > > > something else. Well, that's not quite true. The Low used in the
> > > > calculation was the result of an IF() statement that made sure that
> the
> > > Low
> > > > was really the Low by our definition (by checking it against the
> > previous
> > > > day's Close).
> > > >
> > > > What internal methodology might cause this excellent bit of
> subtraction?
> > > >
> > > > Granted, in checking out the 170-member dataset, I didn't check them
> > all.
> > > I
> > > > checked the first 20 and the last 20. Now I'll probably have to go
> back
> > > and
> > > > sample some in the middle.
> > > >
> > > > I used my trusty, solar powered calculator to double-check my
Clipper
> > > output
> > > > and they both agree that MS is wrong. Any suggestions?
> > > >
> > > > This is making me very nervous and might force me back to Excel
and/or
> > VB.
> > > > So far I've spent over two week on this relatively simple program
and
> I
> > > have
> > > > to admit that I never thought it would be necessary to go back and
> > > > double-check such basic arithmetic.
> > > >
> > > >
> > > > Guy
> > > >
> > > > Never be afraid to try something new. Remember, amateurs built the
> ark,
> > > > professionals built the Titanic.
> > > >
> > >
> >
|