|
PureBytes Links
Trading Reference Links
|
Is there a reason why you use a points only test? Is this more appropriate
for futures? If using percentage for a systems test, is there a translation
for your 2 examples? Thanks in advance.
-----Original Message-----
From: owner-metastock@xxxxxxxxxxxxx
[mailto:owner-metastock@xxxxxxxxxxxxx]On Behalf Of Dave Nadeau
Sent: Thursday, March 01, 2001 12:05 AM
To: metastock@xxxxxxxxxxxxx
Subject: Why Optimization Won't Kill My System and Will It Work In The
Future?
OK, this is a work in process, but I wanted to put these ideas out here for
comment. I'd love to hear any corrections or additional thoughts on this
approach:
____________________________________________________________
I've often asked myself this question about over optimizing my systems and
have avoided trying to find optimal parameters, probably to my system's
detriment.
I am reading a book, The Encyclopedia of Trading Strategies by Jeffrey Katz
and Donna McCormick, which provides an excellent treatment of this
subject and offers and answer to this nagging question.
(Any mathematicians or those better versed in statistics, please feel free
to chime in orcorrect any misstatements that I might make. I am basically
talking through something I've just learned...)
By performing a t-test on the resulting set of trading Profit and Losses,
you can tell what is the likelihood that you have a workable system or just
one that you happened upon by chance....or have over-fitted with
optimization.
I will use two different real world examples to demonstrate this in
Metastock:
Example 1:
I will run two tests over the same data set, a continuous contract of the
S&P 500 futures, daily bars, from April 1982 to present. I'll start with
the
built in Equis: Negative Volume Index w/ Opt system in the System Tester.
This system runs a total of 20 optimization steps varying the Moving
Average from 10 to 200 and shows a profitable "Points Only" result of 827
points (incl slippage & commissions) over the period versus a buy and hold
profit of 747 points. Looks good!!!
In the Results.... dialogue, I will right click and copy all of my
individual trades for the in-sample test period, in this case 136 trades.
These I will
paste into Excel, then sort and delete the "Out" trading positions, just
keeping the Longs and Shorts.
I'll continue my analysis in Excel by finding the mean value of the set of
Profit and Loss data generated by my trades, cell formula: =AVERAGE
(X1:X137) for example. I'll also calculate the Standard Deviation, e.g.
cell formula: =STDEV(X1:X137). Also figure the total number of results
that
you have, e.g. cell formula =COUNT(X1:X137).
Now the t-value is just the MEAN divided by the Standard Deviation divided
by the square root of the number of results, e.g. (MEAN/(STDEV/
(SQRT(COUNT)))). Higher t-values are better. Using the function
=TDIST(t-value,COUNT,1) will give you a probability. This t-test is testing
for a
statistical difference between two means, in this case, the mean of the P/L
and zero, or no profits.
Now, I end up with a probability of 10%, so what??? Well, this tells me
that there is about a 10% chance that my results or profits were random, or
just profitable by chance or luck. The system also has about a nine in ten
chance of continuing to work in the out of sample period or in actual
trading. But wait, there's more....
Optimization: If I take the extra step and consider the optimization on the
system, then the results get worse. This optimization consisted of 20
steps, so I'd take (1-.10)=90% chance of success in the future or out of
sample data raised to the power of 20, (.90^20)=11% !!! So much for a good
chance of profits in the future! This is a great example of
overoptimization.
Example 2:
I'll compare the above system to the Equis: Moving Average Crossovers w/
Opt. Over the same data with the same commission and slippage
allowances, the net Points Only profit is 1109 versus Buy and Hold of 747
points. Total trades were 291 for this system. However, this system test
used 60 optimization steps to arrive at this improved result...3 times as
many as Example 1.
The results in this analysis yield a t-statistic of 2.64. The probability
of success as calculated above is 99.5%.
After adjusting this for the 60 optimization steps, the lowered probability
of success or profits is 92%.
So even though this system has gone through greater optimization, it looks
like it has a greater probability of holding up in the future.
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
|