|
PureBytes Links
Trading Reference Links
|
Hi Dave
A very easy to understand discussion. Congratulations.
I had looked at the book, you've inspired me to re-look at it.
Best regards
Walter
----- Original Message -----
From: Dave Nadeau <dave_nadeau@xxxxxxxxx>
To: <metastock@xxxxxxxxxxxxx>
Sent: Thursday, March 01, 2001 12:04 AM
Subject: Why Optimization Won't Kill My System and Will It Work In The
Future?
>
> OK, this is a work in process, but I wanted to put these ideas out here
for comment. I'd love to hear any corrections or additional thoughts on
this
> approach:
> ____________________________________________________________
> I've often asked myself this question about over optimizing my systems and
have avoided trying to find optimal parameters, probably to my system's
> detriment.
>
> I am reading a book, The Encyclopedia of Trading Strategies by Jeffrey
Katz and Donna McCormick, which provides an excellent treatment of this
> subject and offers and answer to this nagging question.
>
> (Any mathematicians or those better versed in statistics, please feel free
to chime in orcorrect any misstatements that I might make. I am basically
> talking through something I've just learned...)
>
> By performing a t-test on the resulting set of trading Profit and Losses,
you can tell what is the likelihood that you have a workable system or just
> one that you happened upon by chance....or have over-fitted with
optimization.
>
> I will use two different real world examples to demonstrate this in
Metastock:
>
> Example 1:
>
> I will run two tests over the same data set, a continuous contract of the
S&P 500 futures, daily bars, from April 1982 to present. I'll start with
the
> built in Equis: Negative Volume Index w/ Opt system in the System Tester.
This system runs a total of 20 optimization steps varying the Moving
> Average from 10 to 200 and shows a profitable "Points Only" result of 827
points (incl slippage & commissions) over the period versus a buy and hold
> profit of 747 points. Looks good!!!
>
> In the Results.... dialogue, I will right click and copy all of my
individual trades for the in-sample test period, in this case 136 trades.
These I will
> paste into Excel, then sort and delete the "Out" trading positions, just
keeping the Longs and Shorts.
> I'll continue my analysis in Excel by finding the mean value of the set of
Profit and Loss data generated by my trades, cell formula: =AVERAGE
> (X1:X137) for example. I'll also calculate the Standard Deviation, e.g.
cell formula: =STDEV(X1:X137). Also figure the total number of results
that
> you have, e.g. cell formula =COUNT(X1:X137).
>
> Now the t-value is just the MEAN divided by the Standard Deviation divided
by the square root of the number of results, e.g. (MEAN/(STDEV/
> (SQRT(COUNT)))). Higher t-values are better. Using the function
=TDIST(t-value,COUNT,1) will give you a probability. This t-test is testing
for a
> statistical difference between two means, in this case, the mean of the
P/L and zero, or no profits.
> Now, I end up with a probability of 10%, so what??? Well, this tells me
that there is about a 10% chance that my results or profits were random, or
> just profitable by chance or luck. The system also has about a nine in
ten chance of continuing to work in the out of sample period or in actual
> trading. But wait, there's more....
>
> Optimization: If I take the extra step and consider the optimization on
the system, then the results get worse. This optimization consisted of 20
> steps, so I'd take (1-.10)=90% chance of success in the future or out of
sample data raised to the power of 20, (.90^20)=11% !!! So much for a good
> chance of profits in the future! This is a great example of
overoptimization.
>
> Example 2:
>
> I'll compare the above system to the Equis: Moving Average Crossovers w/
Opt. Over the same data with the same commission and slippage
> allowances, the net Points Only profit is 1109 versus Buy and Hold of 747
points. Total trades were 291 for this system. However, this system test
> used 60 optimization steps to arrive at this improved result...3 times as
many as Example 1.
>
> The results in this analysis yield a t-statistic of 2.64. The probability
of success as calculated above is 99.5%.
>
> After adjusting this for the 60 optimization steps, the lowered
probability of success or profits is 92%.
>
> So even though this system has gone through greater optimization, it looks
like it has a greater probability of holding up in the future.
>
>
> _________________________________________________________
> Do You Yahoo!?
> Get your free @yahoo.com address at http://mail.yahoo.com
>
|