|
PureBytes Links
Trading Reference Links
|
Tom,
Thanks for taking the time to respond. I agree with everything you
had to say with one exception. I have just run a points only
comparison test and it did not match. Sent the results to Equis. I
had all of the settings you recommended. It took about three months
to find them after I bought v8.01.
David
--- In equismetastock@xxxxxxxxxxxxxxx, "Tom Sprunger"
<tlsprunger@xxxx> wrote:
> equisdvc, here is a summary of system test issues in 8.01 and how
to get
> around them I have posted a few times. Hope it helps you.
>
> -------------------
> I spent a week or three figuring out how the MS v8 System Tester
works. I
> had problems getting the results from 8.01 to match 7.2. I had
strange
> signals, missing trades, multiple trades off one signal, etc. Had
several
> emails, phone conversations, etc with Equis. Bottom line is that
they went
> overboard in trying to make it realistic, to the point it is
unrealistic
> (my opinion). Especially if you are trying to develop and test
systems.
> However, there are two workarounds you can use to get valid results.
>
> So I won't forget what I learned I, wrote it up. Below tells how
the tester
> currently works and explains why the results give missing trades,
multiple
> trades, doesn't use your full equity, etc. and therefore provides
invalid
> results.
>
> Read below how 8.01 works, then look at the two methods to get
valid results
> if you are trying to develop and test systems.
>
> Equis has told me they are looking into changing it, but no commit,
and no
> timeframe for making the decision.
> If you agree with me, send or call them and tell them to change
it. At the
> bottom, I list what I have recommended they do.
>
> --------------------------------------------------------------------
--------
>
> The Metastock v8.x system tester works very differently than
previous
> versions. On the up side it can test multiple securities
simultaneously.
> However, its operation has differences that are very significant to
a system
> developer, and in reality are not anywhere near what would happen
in the
> real world.
>
> It is important to understand these differences because the v8
tester can
> give very different results from the v7 tester when using the same
system on
> the same security with the same time periods. In addition, because
of the
> v8 "features" you can get different results comparing the several
systems on
> the same security from the v7 tester. This also implies
Optimization
> results could be invalid.
>
> Later we will see how to get around these issues and limitations.
>
> Here's the major differences.
>
> The v8 tester has several new options that are key. The key
differences
> result from a "Broker" module which places orders.
>
> a. "Position Limit" This is accessible from the "General" screen
on each
> individual system test description. You will need to be very
careful what
> you put in here as you will see later. In addition, if you are
comparing
> different systems, you need to be sure that they all have the same
number of
> positions allowed or you may get unfair comparison results.
>
> b. Trading size. In the v7 tester the system traded 100% of your
equity on
> each trade. There was no option to do otherwise (except
for "points only"
> test). The v8 tester allows you to trade a constant number of
shares, total
> transaction cost, or a percentage of equity available. We will
concentrate
> on the "% of equity available, as it is most relevant for system
development
> and comparison.
>
> c. Trade Price -- the v7 tester allowed you to select the price
field
> (O,H,L,C) for entry and exit and the delay for each. In other
words you
> could have different delays for entry and exit.
> The v8 tester does not allow different delays for entry and exit.
It uses
> the same delay for all entries and exits. The v8 tester also allows
> different price fields for the long and short entries and exits,
which the
> v7 did not. But there is more you should know.... read on.
>
> The biggest difference in v8 is the addition of a button
for "Realistic
> Market Prices". It is important to understand what happens when
you check
> this button.
>
> To explain, let's assume you have "Position Limit" =1, "% of equity
> available"=100 and the "Realistic Market Prices" button checked.
>
> When you get an enter Long signal, the program calculates the
number of
> shares that can be purchased using the open price on the signal bar
(day0).
> It tries to purchase that number of shares at the open price on the
next bar
> (day1). However, if the open price on the next bar (day1) is
higher than
> the price from the signal bar (day0), the tester does not place the
order
> because you do not have enough money. Now let's say you did not
have enough
> money. If your system is gives one day signals like you would get
from a
> cross(x,y) signal, then this trade will never be entered.
>
> If your system stays on a buy like you might have with a latch or
with a
> c>mov(c,20,e) structure, then the tester will still be on a buy
signal on
> day1. It will then recalculate the number of shares that can be
purchase at
> the open on day 1 and attempt to buy them on day2. If it can't buy
because
> the price is up at the open on day2 then it will not enter. The
process
> will continue until either the price drops at the open and you get
filled or
> the buy signal goes away. Thus it is again possible that you never
enter on
> this trade signal. The other possibility is that you will not get
filled at
> 100% of your equity.
>
> Now for a moment assume that you had your "Position Limit" set at
10, and
> your buy signal remains in effect. The tester will continue each
bar to try
> to
> fill your orders as above, and will add positions until either your
buy
> signal goes away, or your equity is used up, or your positions are
equal to
> 10. So if you have a buy signal that lasts for 20 bars, you may get
10
> positions filled where the last one may be filled on the last bar
before
> the buy signal goes away.
>
> Note: the reverse happens on enter short signals.
>
> When the system gets an exit signal (either long or short) it exits
on the
> open of the next bar if you have "Realistic Market Prices"
checked. Note
> that there does not seem to be any issues on the exits. Since it
is an
> exit, it just exits regardless of the prices.
>
> If your system is an always in the market system, and you flip from
a long
> to a short, it will exit longs on the next bar, and attempt to sell
short
> the number of shares it calculates based upon the open price of the
signal
> day. If the price on day1 is higher, it does not enter the short
sale.
>
> So, it is key to realize that if you use "Realistic Market Prices",
> Your tests are not at all realistic because you may not get filled
on the
> day of the signal, or you may get filled at all, or you may get
filled
> multiple times. Implications are:
> 1. Can't compare systems on same security.
> 2. buy sell arrows represent entry/exit dates and not signal dates.
> 3. This problem applies to all securities including mutual funds.
> 4. Very difficult to develop systems and see valid signals.
> 5. Can't compare performance on one security to another with same or
> different systems
>
> Next let's review how the "Trade Price" from the "Trade Execution"
screen
> works if you do not have the "Realistic Prices" option set.
>
> The system calculates the number of shares to buy based on the
price field
> you choose and then tries to buy them on the price field you choose.
>
> If you have "Buy Price" = Open and Delay =1, the system calculates
the
> number of shares to buy based upon the Open price on the signal bar
and
> attempts to buy at the open of the next bar. If you have "Buy
Price" =
> Close and Delay = 1 the system calculates the number of shares
based upon
> the close of the signal bar and attempts to buy on the close of
the next
> bar. Same for Sell Price, Sell Short Price, etc. So if all you
do is
> uncheck the "Realistic Market Prices" button and use "Buy on open
with
> delay=1", nothing really changes.
>
> This is not logical at all! If you decided to buy on the open
tomorrow
> based on a signal today, only an idiot would calculate the number
of shares
> to buy based on today's open! There has been a whole day's worth
of price
> action transpired already and the open from yesterday
> is no longer relevant. You would logically use today's close
to "Estimate"
> the number of shares that you "Might purchase" the next day.
>
> Because of all this it is difficult to develop and compare systems
using the
> v8 tester that produce valid results.
>
> Now, it gets even worse. All this tells me the optimization
results are not
> valid in v8. Since
> all this buy, can't buy because price is up, etc etc stuff occurs,
I assume
> it occurs on each optimization run. So your optimization results
are not
> based on reality. In some trades, you don't get filled because of
this. In
> others you get filled late. A $.01 increase on the open would
potentially
> cause you not to be filled. So it makes the optimization results
> inconsistent and unreliable. When you optimize you want to find
the best
> signals that optimize your performance. The potential non or late
fills
> because of a small price change on a given parameter run totally
distort
> this.
>
> So what can you do to get around this to develop and compare
systems,
> compare multiple systems, etc. ?
>
> Note that the above issues do not seem to effect you if you use
a "Points
> Only" test. In this mode apparently the v8 works just like the v7
tester.
> So you can get valid results that way.
>
> But if you wish to see an equity curve, or get a return in % then
you have
> the above problems.
>
> There are two ways to get around this and get valid results and
signals.
>
> First, don't use "Realistic Market Prices".
> Make sure your "Position Limit" is set to 1.
> Set Equity Default size to 100%.
> Under Margin Requirements to
> Long Initial = 100%
> Long Maintenance = 0%
> Short Initial = 200% (amazingly, 200% means no margin, but
that is
> another story)
> Short Maintenance = 101% (this means essentially no
maintenance. for
> some dumb reason they won't let you put in 100%, so you will never
be able
> to trade your full equity on a short trade, only 99%.)
>
> Now you have two options.
> 1. Set all your enter and exit positions to "Close" with a Delay =0.
> This will enter and exit all of your equity (except for the stupid
1% limit
> on short sales) at the close. All your signals will show up on the
charts
> correctly, your % gains will be correct and it will produce results
very
> close to the v7 tester. Of course the drawback is that in the real
world,
> you probably won't enter and exit on the same bar as the signal.
However, if
> you are mainly looking to compare systems or see signal dates,
then this
> works.
>
> 2. Set all of your enter and exit conditions to use format of
> Ref(enter,-1).
> Use "Price Field = Open with Delay =0". (or Close with delay of 0,
or
> whatever).
> What happens is that the condition that is true on day0 gets
evaluated as
> true on day1 (the day after the signal and the right day to
enter). The
> number of shares are calculated and purchased at the same price.
The arrows
> show up on the chart on the entry/exit day.
>
> This is as realistic as the v7 tester and actually pretty good.
> The downside is you have to rewrite all your systems to use the Ref
()
> function in all the enter and exit conditions. Lots of work and
pain.
>
> Using either of these methods should make the system tests valid,
match 7.2
> results, and make the optimization results valid.
>
> What should be done is that Equis changes the tester. Here is what
I have
> recommended they do:
>
> If the "Realistic Market Prices" button is checked the system
should adjust
> the number of shares to buy everything allowed by the equity in the
account
> at the open the next day.
>
> When the user selected options for Buy,Sell, etc are checked, the
system
> should adjust the number of shares to buy the total $ allowed by
the equity
> at the price field selected, on the bar selected by the delay field.
>
> Of course the other option is to just put it back to the v7 method.
>
> One other caveat, I have done any testing attempting to use the
limit
> orders so I have no idea what happens when you try to do that.
>
>
>
> ----- Original Message -----
> From: "equisdvc" <david_cardner@xxxx>
> To: <equismetastock@xxxxxxxxxxxxxxx>
> Sent: Sunday, November 23, 2003 7:53 PM
> Subject: [EquisMetaStock Group] Bugs in v8.01
>
>
> > I have been working with the Equis Technical Support personnel
since
> > I purchased v7.0 in June of 2000. The support has been
excellent, so
> > when v8.0 was released, I bought it. I had developed my own
trading
> > algorithm and was particularly interested in being able to
calibrate
> > it against a portfolio of equities with the System Tester.
However,
> > I found the number of bugs in v8.0 was overwhelming and I had to
> > revert back to using the v7.22 System Tester. My personal
observation
> > was that version 8.0 was not even qualified for alpha testing at
the
> > time of its release.
> >
> > I was delighted when v8.01 was released six months later. I
thought
> > the bugs had been corrected. I was wrong. I discovered that
v8.01,
> > although a big improvement, would barely qualify as a beta test
> > version. I was both upset and embarrassed because I had talked
two
> > friends into buying v8.01.
> >
> > With both v7.22 and v8.01 on the same computer it is easy to
compare
> > System Tester results. The results between the two versions
running
> > the identical algorithm are drastically different, regardless of
> > whether they are being run in normal or points-only mode. This
occurs
> > when v8.01 has been carefully configured to reproduce v7.22
results.
> > Bottom line is that v7.22 results make sense, v8.01 results do
not.
> > Technical support has not been able to explain the differences,
nor
> > provide a workable solution. I still do not use v8.01. I use only
> > v7.22 because it is the only version that I trust.
> >
> > What I find unusual is that I can't find any other reference on
the
> > internet to the mismatch between v7.22 and v8.01 System Tester
> > results. In my 43 years of using computers for problem solving,
and
> > having written, debugged, and documented roughly 100,000 lines of
> > code, one of the primary requirements when modifying programs is
that
> > they must be able to correctly reproduce previously validated
> > results. This test was obviously not applied to v8.0 or v8.01.
Yet
> > it is the only way to validate the changes made to the program.
> >
> > Is there anyone out there that encountered this situation and
> > understands its significance?
> >
> > David
> >
> >
> >
> >
> > To unsubscribe from this group, send an email to:
> > equismetastock-unsubscribe@xxxxxxxxxxxxxxx
> >
> >
> >
> > Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/
> >
> >
------------------------ Yahoo! Groups Sponsor ---------------------~-->
Buy Ink Cartridges or Refill Kits for your HP, Epson, Canon or Lexmark
Printer at MyInks.com. Free s/h on orders $50 or more to the US & Canada.
http://www.c1tracking.com/l.asp?cid=5511
http://us.click.yahoo.com/mOAaAA/3exGAA/qnsNAA/BefplB/TM
---------------------------------------------------------------------~->
To unsubscribe from this group, send an email to:
equismetastock-unsubscribe@xxxxxxxxxxxxxxx
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
|