|
PureBytes Links
Trading Reference Links
|
Tom,
I hit the wrong button on my mouse and the note I was writing
disappeared. You may have it, but I'll do it again in case you don't.
I lost my files on the points test due to a crash, so it will take
some time to generat a new set. I'll try to do that by Friday
evening. In the meantime some observations and comments.
First, the fundamental design basis for the System Tester is wrong.
Even if they eliminated every bug, it will still produce totally
irrelevent results for someone trying to develop an indicator with
one or more paramters. The fundamental concept being used is to
manipulate the parameters to maximize the portfolio return. However,
if that same approach is applied next week, next month, or next year,
it will generate significantly different parameter values each time.
If a portfolio of different stocks is "optimized" the parameter
values will be different than those for the original portfolio. In
short, one cannot use the parameter values on a different portfolio
or at future times and have confidence that they are producing
reliable results.
What is needed is a method that will generate a set of parameter
values that will be essentially unchanged over a period of time with
the same portfolio and will be essentially the same for different
portfolios. Only then can the trading algorithm be trusted to produce
valid and reproducible results day-to-day.
In this situation I am refering to portfolios with a sufficient
number of entries to provide a statistically valid sample of the
entire population.
In order to do this the objective function for the System Tester must
be changed. It can no longer be simply to maximize portfolio
return. What must happen is this. During the calculations by the
System Tester where it tries every combination of parameter values,
the parameter values that maximize the return for each individual
stock must be saved. All values for a given parameter are averaged
and the standard deviation calculated. An optional step would then
be to eliminate all values exceeding 2, 3, or 4 standard deviations,
and then average the remaining values. These average parameter
values can then be used in the trading algorithm to compute portfolio
returns day after day with confidence. They provide the best match
of model to reality that can be achieved with that algorithm. In
short, the objective should be to optimize parameter values, not
portfolio return. It is only with optimized parameter values will
the portfolio be optimized over time.
If the same test is performed one week, one month, or one year later,
the parameter values should be little changed. If a different
portfolio of the same number of stocks were to be tested, the
parameter values would be essentially the same. Again, this is the
best that can be achieved with a given algorithm.
Second. As you may have guessed from the previous paragraphs
inclusion of secondary and tertiary minutia is counter productive.
They will be completely obliterated by the parameter optimization
procedure I have just outlined, while greatly complicating both the
understanding and the application of the System Tester by the user.
Third. My recommendation would be to scape the existing v8.01 System
Tester entirely. As I mentioned above, even if it is totally bug
free, it will still generate totally meaningless and misleading
results for the user. I would further recommend that they take the
existing v7.22 system Tester, encapsulate it in a shell that will
allow it to process a portfolio of stocks and generate optimim
parameter values as I have outlined above.
Will Equis correct their mistake? I think it is highly unlikely.
Big egos made a monumental blunder and are unlikely to admit it and
take corrective action. It is compounded by the fact that they do
not seem to understand what they are trying to do.
David
--- In equismetastock@xxxxxxxxxxxxxxx, "Tom Sprunger"
<tlsprunger@xxxx> wrote:
>
> David, please post what you find in the differences on points only
test.
>
>
> ----- Original Message -----
> From: "equisdvc" <david_cardner@xxxx>
> To: <equismetastock@xxxxxxxxxxxxxxx>
> Sent: Wednesday, December 03, 2003 4:02 PM
> Subject: [EquisMetaStock Group] Re: Bugs in v8.01
>
>
> > Tom,
> > Thanks for taking the time to respond. I agree with everything
you
> > had to say with one exception. I have just run a points only
> > comparison test and it did not match. Sent the results to Equis.
I
> > had all of the settings you recommended. It took about three
months
> > to find them after I bought v8.01.
> > David
> >
> > --- In equismetastock@xxxxxxxxxxxxxxx, "Tom Sprunger"
> > <tlsprunger@xxxx> wrote:
> > > equisdvc, here is a summary of system test issues in 8.01 and
how
> > to get
> > > around them I have posted a few times. Hope it helps you.
> > >
> > > -------------------
> > > I spent a week or three figuring out how the MS v8 System Tester
> > works. I
> > > had problems getting the results from 8.01 to match 7.2. I
had
> > strange
> > > signals, missing trades, multiple trades off one signal, etc.
Had
> > several
> > > emails, phone conversations, etc with Equis. Bottom line is
that
> > they went
> > > overboard in trying to make it realistic, to the point it is
> > unrealistic
> > > (my opinion). Especially if you are trying to develop and test
> > systems.
> > > However, there are two workarounds you can use to get valid
results.
> > >
> > > So I won't forget what I learned I, wrote it up. Below tells
how
> > the tester
> > > currently works and explains why the results give missing
trades,
> > multiple
> > > trades, doesn't use your full equity, etc. and therefore
provides
> > invalid
> > > results.
> > >
> > > Read below how 8.01 works, then look at the two methods to get
> > valid results
> > > if you are trying to develop and test systems.
> > >
> > > Equis has told me they are looking into changing it, but no
commit,
> > and no
> > > timeframe for making the decision.
> > > If you agree with me, send or call them and tell them to change
> > it. At the
> > > bottom, I list what I have recommended they do.
> > >
> > > ----------------------------------------------------------------
----
> > --------
> > >
> > > The Metastock v8.x system tester works very differently than
> > previous
> > > versions. On the up side it can test multiple securities
> > simultaneously.
> > > However, its operation has differences that are very
significant to
> > a system
> > > developer, and in reality are not anywhere near what would
happen
> > in the
> > > real world.
> > >
> > > It is important to understand these differences because the v8
> > tester can
> > > give very different results from the v7 tester when using the
same
> > system on
> > > the same security with the same time periods. In addition,
because
> > of the
> > > v8 "features" you can get different results comparing the
several
> > systems on
> > > the same security from the v7 tester. This also implies
> > Optimization
> > > results could be invalid.
> > >
> > > Later we will see how to get around these issues and
limitations.
> > >
> > > Here's the major differences.
> > >
> > > The v8 tester has several new options that are key. The key
> > differences
> > > result from a "Broker" module which places orders.
> > >
> > > a. "Position Limit" This is accessible from the "General"
screen
> > on each
> > > individual system test description. You will need to be very
> > careful what
> > > you put in here as you will see later. In addition, if you are
> > comparing
> > > different systems, you need to be sure that they all have the
same
> > number of
> > > positions allowed or you may get unfair comparison results.
> > >
> > > b. Trading size. In the v7 tester the system traded 100% of
your
> > equity on
> > > each trade. There was no option to do otherwise (except
> > for "points only"
> > > test). The v8 tester allows you to trade a constant number of
> > shares, total
> > > transaction cost, or a percentage of equity available. We will
> > concentrate
> > > on the "% of equity available, as it is most relevant for system
> > development
> > > and comparison.
> > >
> > > c. Trade Price -- the v7 tester allowed you to select the price
> > field
> > > (O,H,L,C) for entry and exit and the delay for each. In other
> > words you
> > > could have different delays for entry and exit.
> > > The v8 tester does not allow different delays for entry and
exit.
> > It uses
> > > the same delay for all entries and exits. The v8 tester also
allows
> > > different price fields for the long and short entries and exits,
> > which the
> > > v7 did not. But there is more you should know.... read on.
> > >
> > > The biggest difference in v8 is the addition of a button
> > for "Realistic
> > > Market Prices". It is important to understand what happens when
> > you check
> > > this button.
> > >
> > > To explain, let's assume you have "Position Limit" =1, "% of
equity
> > > available"=100 and the "Realistic Market Prices" button checked.
> > >
> > > When you get an enter Long signal, the program calculates the
> > number of
> > > shares that can be purchased using the open price on the signal
bar
> > (day0).
> > > It tries to purchase that number of shares at the open price on
the
> > next bar
> > > (day1). However, if the open price on the next bar (day1) is
> > higher than
> > > the price from the signal bar (day0), the tester does not place
the
> > order
> > > because you do not have enough money. Now let's say you did not
> > have enough
> > > money. If your system is gives one day signals like you would
get
> > from a
> > > cross(x,y) signal, then this trade will never be entered.
> > >
> > > If your system stays on a buy like you might have with a latch
or
> > with a
> > > c>mov(c,20,e) structure, then the tester will still be on a buy
> > signal on
> > > day1. It will then recalculate the number of shares that can be
> > purchase at
> > > the open on day 1 and attempt to buy them on day2. If it can't
buy
> > because
> > > the price is up at the open on day2 then it will not enter. The
> > process
> > > will continue until either the price drops at the open and you
get
> > filled or
> > > the buy signal goes away. Thus it is again possible that you
never
> > enter on
> > > this trade signal. The other possibility is that you will not
get
> > filled at
> > > 100% of your equity.
> > >
> > > Now for a moment assume that you had your "Position Limit" set
at
> > 10, and
> > > your buy signal remains in effect. The tester will continue
each
> > bar to try
> > > to
> > > fill your orders as above, and will add positions until either
your
> > buy
> > > signal goes away, or your equity is used up, or your positions
are
> > equal to
> > > 10. So if you have a buy signal that lasts for 20 bars, you may
get
> > 10
> > > positions filled where the last one may be filled on the last
bar
> > before
> > > the buy signal goes away.
> > >
> > > Note: the reverse happens on enter short signals.
> > >
> > > When the system gets an exit signal (either long or short) it
exits
> > on the
> > > open of the next bar if you have "Realistic Market Prices"
> > checked. Note
> > > that there does not seem to be any issues on the exits. Since
it
> > is an
> > > exit, it just exits regardless of the prices.
> > >
> > > If your system is an always in the market system, and you flip
from
> > a long
> > > to a short, it will exit longs on the next bar, and attempt to
sell
> > short
> > > the number of shares it calculates based upon the open price of
the
> > signal
> > > day. If the price on day1 is higher, it does not enter the short
> > sale.
> > >
> > > So, it is key to realize that if you use "Realistic Market
Prices",
> > > Your tests are not at all realistic because you may not get
filled
> > on the
> > > day of the signal, or you may get filled at all, or you may get
> > filled
> > > multiple times. Implications are:
> > > 1. Can't compare systems on same security.
> > > 2. buy sell arrows represent entry/exit dates and not signal
dates.
> > > 3. This problem applies to all securities including mutual
funds.
> > > 4. Very difficult to develop systems and see valid signals.
> > > 5. Can't compare performance on one security to another with
same or
> > > different systems
> > >
> > > Next let's review how the "Trade Price" from the "Trade
Execution"
> > screen
> > > works if you do not have the "Realistic Prices" option set.
> > >
> > > The system calculates the number of shares to buy based on the
> > price field
> > > you choose and then tries to buy them on the price field you
choose.
> > >
> > > If you have "Buy Price" = Open and Delay =1, the system
calculates
> > the
> > > number of shares to buy based upon the Open price on the signal
bar
> > and
> > > attempts to buy at the open of the next bar. If you have "Buy
> > Price" =
> > > Close and Delay = 1 the system calculates the number of shares
> > based upon
> > > the close of the signal bar and attempts to buy on the close
of
> > the next
> > > bar. Same for Sell Price, Sell Short Price, etc. So if all
you
> > do is
> > > uncheck the "Realistic Market Prices" button and use "Buy on
open
> > with
> > > delay=1", nothing really changes.
> > >
> > > This is not logical at all! If you decided to buy on the open
> > tomorrow
> > > based on a signal today, only an idiot would calculate the
number
> > of shares
> > > to buy based on today's open! There has been a whole day's
worth
> > of price
> > > action transpired already and the open from yesterday
> > > is no longer relevant. You would logically use today's close
> > to "Estimate"
> > > the number of shares that you "Might purchase" the next day.
> > >
> > > Because of all this it is difficult to develop and compare
systems
> > using the
> > > v8 tester that produce valid results.
> > >
> > > Now, it gets even worse. All this tells me the optimization
> > results are not
> > > valid in v8. Since
> > > all this buy, can't buy because price is up, etc etc stuff
occurs,
> > I assume
> > > it occurs on each optimization run. So your optimization
results
> > are not
> > > based on reality. In some trades, you don't get filled because
of
> > this. In
> > > others you get filled late. A $.01 increase on the open would
> > potentially
> > > cause you not to be filled. So it makes the optimization results
> > > inconsistent and unreliable. When you optimize you want to find
> > the best
> > > signals that optimize your performance. The potential non or
late
> > fills
> > > because of a small price change on a given parameter run totally
> > distort
> > > this.
> > >
> > > So what can you do to get around this to develop and compare
> > systems,
> > > compare multiple systems, etc. ?
> > >
> > > Note that the above issues do not seem to effect you if you use
> > a "Points
> > > Only" test. In this mode apparently the v8 works just like the
v7
> > tester.
> > > So you can get valid results that way.
> > >
> > > But if you wish to see an equity curve, or get a return in %
then
> > you have
> > > the above problems.
> > >
> > > There are two ways to get around this and get valid results and
> > signals.
> > >
> > > First, don't use "Realistic Market Prices".
> > > Make sure your "Position Limit" is set to 1.
> > > Set Equity Default size to 100%.
> > > Under Margin Requirements to
> > > Long Initial = 100%
> > > Long Maintenance = 0%
> > > Short Initial = 200% (amazingly, 200% means no margin, but
> > that is
> > > another story)
> > > Short Maintenance = 101% (this means essentially no
> > maintenance. for
> > > some dumb reason they won't let you put in 100%, so you will
never
> > be able
> > > to trade your full equity on a short trade, only 99%.)
> > >
> > > Now you have two options.
> > > 1. Set all your enter and exit positions to "Close" with a
Delay =0.
> > > This will enter and exit all of your equity (except for the
stupid
> > 1% limit
> > > on short sales) at the close. All your signals will show up on
the
> > charts
> > > correctly, your % gains will be correct and it will produce
results
> > very
> > > close to the v7 tester. Of course the drawback is that in the
real
> > world,
> > > you probably won't enter and exit on the same bar as the signal.
> > However, if
> > > you are mainly looking to compare systems or see signal dates,
> > then this
> > > works.
> > >
> > > 2. Set all of your enter and exit conditions to use format of
> > > Ref(enter,-1).
> > > Use "Price Field = Open with Delay =0". (or Close with delay
of 0,
> > or
> > > whatever).
> > > What happens is that the condition that is true on day0 gets
> > evaluated as
> > > true on day1 (the day after the signal and the right day to
> > enter). The
> > > number of shares are calculated and purchased at the same price.
> > The arrows
> > > show up on the chart on the entry/exit day.
> > >
> > > This is as realistic as the v7 tester and actually pretty good.
> > > The downside is you have to rewrite all your systems to use the
Ref
> > ()
> > > function in all the enter and exit conditions. Lots of work and
> > pain.
> > >
> > > Using either of these methods should make the system tests
valid,
> > match 7.2
> > > results, and make the optimization results valid.
> > >
> > > What should be done is that Equis changes the tester. Here is
what
> > I have
> > > recommended they do:
> > >
> > > If the "Realistic Market Prices" button is checked the system
> > should adjust
> > > the number of shares to buy everything allowed by the equity in
the
> > account
> > > at the open the next day.
> > >
> > > When the user selected options for Buy,Sell, etc are checked,
the
> > system
> > > should adjust the number of shares to buy the total $ allowed by
> > the equity
> > > at the price field selected, on the bar selected by the delay
field.
> > >
> > > Of course the other option is to just put it back to the v7
method.
> > >
> > > One other caveat, I have done any testing attempting to use the
> > limit
> > > orders so I have no idea what happens when you try to do that.
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From: "equisdvc" <david_cardner@xxxx>
> > > To: <equismetastock@xxxxxxxxxxxxxxx>
> > > Sent: Sunday, November 23, 2003 7:53 PM
> > > Subject: [EquisMetaStock Group] Bugs in v8.01
> > >
> > >
> > > > I have been working with the Equis Technical Support personnel
> > since
> > > > I purchased v7.0 in June of 2000. The support has been
> > excellent, so
> > > > when v8.0 was released, I bought it. I had developed my own
> > trading
> > > > algorithm and was particularly interested in being able to
> > calibrate
> > > > it against a portfolio of equities with the System Tester.
> > However,
> > > > I found the number of bugs in v8.0 was overwhelming and I had
to
> > > > revert back to using the v7.22 System Tester. My personal
> > observation
> > > > was that version 8.0 was not even qualified for alpha testing
at
> > the
> > > > time of its release.
> > > >
> > > > I was delighted when v8.01 was released six months later. I
> > thought
> > > > the bugs had been corrected. I was wrong. I discovered that
> > v8.01,
> > > > although a big improvement, would barely qualify as a beta
test
> > > > version. I was both upset and embarrassed because I had
talked
> > two
> > > > friends into buying v8.01.
> > > >
> > > > With both v7.22 and v8.01 on the same computer it is easy to
> > compare
> > > > System Tester results. The results between the two versions
> > running
> > > > the identical algorithm are drastically different, regardless
of
> > > > whether they are being run in normal or points-only mode. This
> > occurs
> > > > when v8.01 has been carefully configured to reproduce v7.22
> > results.
> > > > Bottom line is that v7.22 results make sense, v8.01 results do
> > not.
> > > > Technical support has not been able to explain the
differences,
> > nor
> > > > provide a workable solution. I still do not use v8.01. I use
only
> > > > v7.22 because it is the only version that I trust.
> > > >
> > > > What I find unusual is that I can't find any other reference
on
> > the
> > > > internet to the mismatch between v7.22 and v8.01 System Tester
> > > > results. In my 43 years of using computers for problem
solving,
> > and
> > > > having written, debugged, and documented roughly 100,000
lines of
> > > > code, one of the primary requirements when modifying programs
is
> > that
> > > > they must be able to correctly reproduce previously validated
> > > > results. This test was obviously not applied to v8.0 or
v8.01.
> > Yet
> > > > it is the only way to validate the changes made to the
program.
> > > >
> > > > Is there anyone out there that encountered this situation and
> > > > understands its significance?
> > > >
> > > > David
> > > >
> > > >
> > > >
> > > >
> > > > To unsubscribe from this group, send an email to:
> > > > equismetastock-unsubscribe@xxxxxxxxxxxxxxx
> > > >
> > > >
> > > >
> > > > Your use of Yahoo! Groups is subject to
> > http://docs.yahoo.com/info/terms/
> > > >
> > > >
> >
> >
> >
> >
> > To unsubscribe from this group, send an email to:
> > equismetastock-unsubscribe@xxxxxxxxxxxxxxx
> >
> >
> >
> > Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/
> >
> >
------------------------ Yahoo! Groups Sponsor ---------------------~-->
Buy Ink Cartridges or Refill Kits for your HP, Epson, Canon or Lexmark
Printer at MyInks.com. Free s/h on orders $50 or more to the US & Canada.
http://www.c1tracking.com/l.asp?cid=5511
http://us.click.yahoo.com/mOAaAA/3exGAA/qnsNAA/BefplB/TM
---------------------------------------------------------------------~->
To unsubscribe from this group, send an email to:
equismetastock-unsubscribe@xxxxxxxxxxxxxxx
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
|