|
PureBytes Links
Trading Reference Links
|
equisdvc, here is a summary of system test issues in 8.01 and how to get
around them I have posted a few times. Hope it helps you.
-------------------
I spent a week or three figuring out how the MS v8 System Tester works. I
had problems getting the results from 8.01 to match 7.2. I had strange
signals, missing trades, multiple trades off one signal, etc. Had several
emails, phone conversations, etc with Equis. Bottom line is that they went
overboard in trying to make it realistic, to the point it is unrealistic
(my opinion). Especially if you are trying to develop and test systems.
However, there are two workarounds you can use to get valid results.
So I won't forget what I learned I, wrote it up. Below tells how the tester
currently works and explains why the results give missing trades, multiple
trades, doesn't use your full equity, etc. and therefore provides invalid
results.
Read below how 8.01 works, then look at the two methods to get valid results
if you are trying to develop and test systems.
Equis has told me they are looking into changing it, but no commit, and no
timeframe for making the decision.
If you agree with me, send or call them and tell them to change it. At the
bottom, I list what I have recommended they do.
----------------------------------------------------------------------------
The Metastock v8.x system tester works very differently than previous
versions. On the up side it can test multiple securities simultaneously.
However, its operation has differences that are very significant to a system
developer, and in reality are not anywhere near what would happen in the
real world.
It is important to understand these differences because the v8 tester can
give very different results from the v7 tester when using the same system on
the same security with the same time periods. In addition, because of the
v8 "features" you can get different results comparing the several systems on
the same security from the v7 tester. This also implies Optimization
results could be invalid.
Later we will see how to get around these issues and limitations.
Here's the major differences.
The v8 tester has several new options that are key. The key differences
result from a "Broker" module which places orders.
a. "Position Limit" This is accessible from the "General" screen on each
individual system test description. You will need to be very careful what
you put in here as you will see later. In addition, if you are comparing
different systems, you need to be sure that they all have the same number of
positions allowed or you may get unfair comparison results.
b. Trading size. In the v7 tester the system traded 100% of your equity on
each trade. There was no option to do otherwise (except for "points only"
test). The v8 tester allows you to trade a constant number of shares, total
transaction cost, or a percentage of equity available. We will concentrate
on the "% of equity available, as it is most relevant for system development
and comparison.
c. Trade Price -- the v7 tester allowed you to select the price field
(O,H,L,C) for entry and exit and the delay for each. In other words you
could have different delays for entry and exit.
The v8 tester does not allow different delays for entry and exit. It uses
the same delay for all entries and exits. The v8 tester also allows
different price fields for the long and short entries and exits, which the
v7 did not. But there is more you should know.... read on.
The biggest difference in v8 is the addition of a button for "Realistic
Market Prices". It is important to understand what happens when you check
this button.
To explain, let's assume you have "Position Limit" =1, "% of equity
available"=100 and the "Realistic Market Prices" button checked.
When you get an enter Long signal, the program calculates the number of
shares that can be purchased using the open price on the signal bar (day0).
It tries to purchase that number of shares at the open price on the next bar
(day1). However, if the open price on the next bar (day1) is higher than
the price from the signal bar (day0), the tester does not place the order
because you do not have enough money. Now let's say you did not have enough
money. If your system is gives one day signals like you would get from a
cross(x,y) signal, then this trade will never be entered.
If your system stays on a buy like you might have with a latch or with a
c>mov(c,20,e) structure, then the tester will still be on a buy signal on
day1. It will then recalculate the number of shares that can be purchase at
the open on day 1 and attempt to buy them on day2. If it can't buy because
the price is up at the open on day2 then it will not enter. The process
will continue until either the price drops at the open and you get filled or
the buy signal goes away. Thus it is again possible that you never enter on
this trade signal. The other possibility is that you will not get filled at
100% of your equity.
Now for a moment assume that you had your "Position Limit" set at 10, and
your buy signal remains in effect. The tester will continue each bar to try
to
fill your orders as above, and will add positions until either your buy
signal goes away, or your equity is used up, or your positions are equal to
10. So if you have a buy signal that lasts for 20 bars, you may get 10
positions filled where the last one may be filled on the last bar before
the buy signal goes away.
Note: the reverse happens on enter short signals.
When the system gets an exit signal (either long or short) it exits on the
open of the next bar if you have "Realistic Market Prices" checked. Note
that there does not seem to be any issues on the exits. Since it is an
exit, it just exits regardless of the prices.
If your system is an always in the market system, and you flip from a long
to a short, it will exit longs on the next bar, and attempt to sell short
the number of shares it calculates based upon the open price of the signal
day. If the price on day1 is higher, it does not enter the short sale.
So, it is key to realize that if you use "Realistic Market Prices",
Your tests are not at all realistic because you may not get filled on the
day of the signal, or you may get filled at all, or you may get filled
multiple times. Implications are:
1. Can't compare systems on same security.
2. buy sell arrows represent entry/exit dates and not signal dates.
3. This problem applies to all securities including mutual funds.
4. Very difficult to develop systems and see valid signals.
5. Can't compare performance on one security to another with same or
different systems
Next let's review how the "Trade Price" from the "Trade Execution" screen
works if you do not have the "Realistic Prices" option set.
The system calculates the number of shares to buy based on the price field
you choose and then tries to buy them on the price field you choose.
If you have "Buy Price" = Open and Delay =1, the system calculates the
number of shares to buy based upon the Open price on the signal bar and
attempts to buy at the open of the next bar. If you have "Buy Price" =
Close and Delay = 1 the system calculates the number of shares based upon
the close of the signal bar and attempts to buy on the close of the next
bar. Same for Sell Price, Sell Short Price, etc. So if all you do is
uncheck the "Realistic Market Prices" button and use "Buy on open with
delay=1", nothing really changes.
This is not logical at all! If you decided to buy on the open tomorrow
based on a signal today, only an idiot would calculate the number of shares
to buy based on today's open! There has been a whole day's worth of price
action transpired already and the open from yesterday
is no longer relevant. You would logically use today's close to "Estimate"
the number of shares that you "Might purchase" the next day.
Because of all this it is difficult to develop and compare systems using the
v8 tester that produce valid results.
Now, it gets even worse. All this tells me the optimization results are not
valid in v8. Since
all this buy, can't buy because price is up, etc etc stuff occurs, I assume
it occurs on each optimization run. So your optimization results are not
based on reality. In some trades, you don't get filled because of this. In
others you get filled late. A $.01 increase on the open would potentially
cause you not to be filled. So it makes the optimization results
inconsistent and unreliable. When you optimize you want to find the best
signals that optimize your performance. The potential non or late fills
because of a small price change on a given parameter run totally distort
this.
So what can you do to get around this to develop and compare systems,
compare multiple systems, etc. ?
Note that the above issues do not seem to effect you if you use a "Points
Only" test. In this mode apparently the v8 works just like the v7 tester.
So you can get valid results that way.
But if you wish to see an equity curve, or get a return in % then you have
the above problems.
There are two ways to get around this and get valid results and signals.
First, don't use "Realistic Market Prices".
Make sure your "Position Limit" is set to 1.
Set Equity Default size to 100%.
Under Margin Requirements to
Long Initial = 100%
Long Maintenance = 0%
Short Initial = 200% (amazingly, 200% means no margin, but that is
another story)
Short Maintenance = 101% (this means essentially no maintenance. for
some dumb reason they won't let you put in 100%, so you will never be able
to trade your full equity on a short trade, only 99%.)
Now you have two options.
1. Set all your enter and exit positions to "Close" with a Delay =0.
This will enter and exit all of your equity (except for the stupid 1% limit
on short sales) at the close. All your signals will show up on the charts
correctly, your % gains will be correct and it will produce results very
close to the v7 tester. Of course the drawback is that in the real world,
you probably won't enter and exit on the same bar as the signal. However, if
you are mainly looking to compare systems or see signal dates, then this
works.
2. Set all of your enter and exit conditions to use format of
Ref(enter,-1).
Use "Price Field = Open with Delay =0". (or Close with delay of 0, or
whatever).
What happens is that the condition that is true on day0 gets evaluated as
true on day1 (the day after the signal and the right day to enter). The
number of shares are calculated and purchased at the same price. The arrows
show up on the chart on the entry/exit day.
This is as realistic as the v7 tester and actually pretty good.
The downside is you have to rewrite all your systems to use the Ref()
function in all the enter and exit conditions. Lots of work and pain.
Using either of these methods should make the system tests valid, match 7.2
results, and make the optimization results valid.
What should be done is that Equis changes the tester. Here is what I have
recommended they do:
If the "Realistic Market Prices" button is checked the system should adjust
the number of shares to buy everything allowed by the equity in the account
at the open the next day.
When the user selected options for Buy,Sell, etc are checked, the system
should adjust the number of shares to buy the total $ allowed by the equity
at the price field selected, on the bar selected by the delay field.
Of course the other option is to just put it back to the v7 method.
One other caveat, I have done any testing attempting to use the limit
orders so I have no idea what happens when you try to do that.
----- Original Message -----
From: "equisdvc" <david_cardner@xxxxxxxxxxx>
To: <equismetastock@xxxxxxxxxxxxxxx>
Sent: Sunday, November 23, 2003 7:53 PM
Subject: [EquisMetaStock Group] Bugs in v8.01
> I have been working with the Equis Technical Support personnel since
> I purchased v7.0 in June of 2000. The support has been excellent, so
> when v8.0 was released, I bought it. I had developed my own trading
> algorithm and was particularly interested in being able to calibrate
> it against a portfolio of equities with the System Tester. However,
> I found the number of bugs in v8.0 was overwhelming and I had to
> revert back to using the v7.22 System Tester. My personal observation
> was that version 8.0 was not even qualified for alpha testing at the
> time of its release.
>
> I was delighted when v8.01 was released six months later. I thought
> the bugs had been corrected. I was wrong. I discovered that v8.01,
> although a big improvement, would barely qualify as a beta test
> version. I was both upset and embarrassed because I had talked two
> friends into buying v8.01.
>
> With both v7.22 and v8.01 on the same computer it is easy to compare
> System Tester results. The results between the two versions running
> the identical algorithm are drastically different, regardless of
> whether they are being run in normal or points-only mode. This occurs
> when v8.01 has been carefully configured to reproduce v7.22 results.
> Bottom line is that v7.22 results make sense, v8.01 results do not.
> Technical support has not been able to explain the differences, nor
> provide a workable solution. I still do not use v8.01. I use only
> v7.22 because it is the only version that I trust.
>
> What I find unusual is that I can't find any other reference on the
> internet to the mismatch between v7.22 and v8.01 System Tester
> results. In my 43 years of using computers for problem solving, and
> having written, debugged, and documented roughly 100,000 lines of
> code, one of the primary requirements when modifying programs is that
> they must be able to correctly reproduce previously validated
> results. This test was obviously not applied to v8.0 or v8.01. Yet
> it is the only way to validate the changes made to the program.
>
> Is there anyone out there that encountered this situation and
> understands its significance?
>
> David
>
>
>
>
> To unsubscribe from this group, send an email to:
> equismetastock-unsubscribe@xxxxxxxxxxxxxxx
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>
------------------------ Yahoo! Groups Sponsor ---------------------~-->
Buy Ink Cartridges or Refill Kits for your HP, Epson, Canon or Lexmark
Printer at MyInks.com. Free s/h on orders $50 or more to the US & Canada.
http://www.c1tracking.com/l.asp?cid=5511
http://us.click.yahoo.com/mOAaAA/3exGAA/qnsNAA/BefplB/TM
---------------------------------------------------------------------~->
To unsubscribe from this group, send an email to:
equismetastock-unsubscribe@xxxxxxxxxxxxxxx
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
|