|
PureBytes Links
Trading Reference Links
|
OK, this is a work in process, but I wanted to put these ideas out here for comment. I'd love to hear any corrections or additional thoughts on this
approach:
____________________________________________________________
I've often asked myself this question about over optimizing my systems and have avoided trying to find optimal parameters, probably to my system's
detriment.
I am reading a book, The Encyclopedia of Trading Strategies by Jeffrey Katz and Donna McCormick, which provides an excellent treatment of this
subject and offers and answer to this nagging question.
(Any mathematicians or those better versed in statistics, please feel free to chime in orcorrect any misstatements that I might make. I am basically
talking through something I've just learned...)
By performing a t-test on the resulting set of trading Profit and Losses, you can tell what is the likelihood that you have a workable system or just
one that you happened upon by chance....or have over-fitted with optimization.
I will use two different real world examples to demonstrate this in Metastock:
Example 1:
I will run two tests over the same data set, a continuous contract of the S&P 500 futures, daily bars, from April 1982 to present. I'll start with the
built in Equis: Negative Volume Index w/ Opt system in the System Tester. This system runs a total of 20 optimization steps varying the Moving
Average from 10 to 200 and shows a profitable "Points Only" result of 827 points (incl slippage & commissions) over the period versus a buy and hold
profit of 747 points. Looks good!!!
In the Results.... dialogue, I will right click and copy all of my individual trades for the in-sample test period, in this case 136 trades. These I will
paste into Excel, then sort and delete the "Out" trading positions, just keeping the Longs and Shorts.
I'll continue my analysis in Excel by finding the mean value of the set of Profit and Loss data generated by my trades, cell formula: =AVERAGE
(X1:X137) for example. I'll also calculate the Standard Deviation, e.g. cell formula: =STDEV(X1:X137). Also figure the total number of results that
you have, e.g. cell formula =COUNT(X1:X137).
Now the t-value is just the MEAN divided by the Standard Deviation divided by the square root of the number of results, e.g. (MEAN/(STDEV/
(SQRT(COUNT)))). Higher t-values are better. Using the function =TDIST(t-value,COUNT,1) will give you a probability. This t-test is testing for a
statistical difference between two means, in this case, the mean of the P/L and zero, or no profits.
Now, I end up with a probability of 10%, so what??? Well, this tells me that there is about a 10% chance that my results or profits were random, or
just profitable by chance or luck. The system also has about a nine in ten chance of continuing to work in the out of sample period or in actual
trading. But wait, there's more....
Optimization: If I take the extra step and consider the optimization on the system, then the results get worse. This optimization consisted of 20
steps, so I'd take (1-.10)=90% chance of success in the future or out of sample data raised to the power of 20, (.90^20)=11% !!! So much for a good
chance of profits in the future! This is a great example of overoptimization.
Example 2:
I'll compare the above system to the Equis: Moving Average Crossovers w/ Opt. Over the same data with the same commission and slippage
allowances, the net Points Only profit is 1109 versus Buy and Hold of 747 points. Total trades were 291 for this system. However, this system test
used 60 optimization steps to arrive at this improved result...3 times as many as Example 1.
The results in this analysis yield a t-statistic of 2.64. The probability of success as calculated above is 99.5%.
After adjusting this for the 60 optimization steps, the lowered probability of success or profits is 92%.
So even though this system has gone through greater optimization, it looks like it has a greater probability of holding up in the future.
_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
|