Re: [amibroker] Re: Is the Walk forward study useful?, AmiBroker Email List Archive

On Thu, Oct 15, 2009 at 6:08 AM, Richard N Park <rnpark@xxxxxxxxxxxxx> wrote:

Bravo, Howard!

You should write a book. J

From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Howard B
Sent: Thursday, October 15, 2009 7:46 AM
To: amibroker@xxxxxxxxxxxxxxx

Subject: Re: [amibroker] Re: Is the Walk forward study useful?

Hi Bing, and all --

I think we need a reality check.

First -- computing the t-test, or any other metric using the results from in-sample runs has no value. Almost any trading system can have the parameters, logic, time frame, and reporting period adjusted so that the in-sample results are very profitable. In my speeches, I show a series of slides, each slide displayed in two steps. Step 1 is the in-sample equity after optimizing over an in-sample period. The chart ends at the end of the in-sample period. It always looks good. When the results of testing a system do not look good, we do not spend any more time on that system, but go on to others where it does look good. Step 2 adds an out-of-sample data period, applies the same system, and shows the out-of-sample equity. For some systems the equity continues to rise, for some it goes flat, for others it drops sharply. There is no way to tell what will happen in the out-of-sample period without testing the out-of-sample period. That is -- there is no information in the in-sample results that predict the out-of-sample results. There is no way to know whether the model has fit itself to the signal component of the data that contains the pattern leading to profitable trades or to some noise component of the data that does not exist in the out-of-sample data.

Second -- if someone has a trading system that has produced a truly out-of-sample set of closed trades where the t-test, or any other fitness metric, would be embarrassingly high without limiting the value used as the number of data points, he or she should call me. I can help them find a semitrailer large enough to carry all the money they will make trading that system. Limiting the number used as N is a non-issue, in addition to being bad procedure.

Tharp admits that he is not a statistician, and that he finds some of the mathematics involved in position sizing and fitness function analysis to be at his comfort limit. It is apparent to me that the systems he shows as standards for SQNs of 2, 3, 4, 5, and so forth are not the result of trading system runs -- either in-sample or out-of-sample -- to which he has applied his metric; they are artificial examples constructed so the results turn out as he wants them to. That is not a bad thing in itself, but when he suggests that we should go on searching for systems with SQNs of 10 or more (page 279 of "Definitive Guide to Position Sizing"), that is completely unrealistic and will send naive systems developers off on windmill-tilting quests that will never be successful.

One of the difficulties using Tharp's data sets is that the standard deviation of losing trades is zero for some of them. That is not only unrealistic, but it makes computation of metrics that include standard deviation of losing trades, such as Sortino ratio, difficult.

Do the following experiment. Put together a data set that represents potentially realistic trading results that you hope to achieve -- be optimistic, but realistic. If you would expect one trade a week, a set of 52 data points represents a year. 252 data points if you would have a trading result every day. Each data point is the number of dollars gained or lost from that closed trade, based on trading a single unit -- one futures contract, one hundred shares of stock, or whatever your one unit is. Compute the mean, standard deviation, and t-test score -- where the t-test is "what is the level of confidence that the mean is greater than 0". Put the data into a text file and use it as input to Equity Monaco, ProSizer, or Market System Analyzer. Run some Monte Carlo simulations to see what the equity could look like from trading one year. Systems with t-test scores, based on expectancy, of 3 will make you incredible amounts of money. Dummy up a data set with a t-test / SQN of 10 and you will see how unrealistic that is.

The critical elements needed to apply aggressive position sizing are: trading something that can be scaled up as desired, positive expectancy, limited semi-deviation (standard deviation of losing trades), and frequent trading.

Being able to limit the amount lost on a losing trade is essential. Risk of bankruptcy (and all trading systems have a non-zero probability of going bankrupt) increases dramatically as the standard deviation of losing trades increases. Dummy up some data and runs some more tests. The effect the amount lost on losing trades has on system performance is scary.

Now go back to your trading system development platform (AmiBroker, of course) and test your trading systems, this time focusing on limiting the standard deviation of losing trades. If your system holds a long time (more than a week or two), pay attention to Maximum Adverse Excursion. If you attempt to apply aggressive position sizing to a system that has reasonable results for closed trades, but has large MAE intra-trade, you will get stopped out (or scared out) intra-trade.

Remember -- if you are applying position sizing to your trading system, your largest loss will come when you have your largest position. Read Ralph Vince.

As usual -- be careful to base your analysis and decision whether to trade any system on truly out-of-sample results. Decisions based on in-sample or contaminated out-of-sample results seriously underestimate the probability of bankruptcy.

Repeat after me -- keep a positive expectancy, limit losing trades, trade frequently.

Thanks for listening,
Howard

On Thu, Oct 15, 2009 at 1:11 AM, bingk66 <bing.kwok@xxxxxxxxxxxxxxx> wrote:

Hi Howard,

If there are no means to limit the number of transactions in the calcs, then one seriously runs the risk of challenging the mystical t-test score of 7 that you spoke about previously.

As an example, if the OOS test was run over a 5 year period with 5000 transactions (a mere 1000 transaction/year, which is not excessive, especially for very short term trades), sqrt(5000) alone would yield in excess of 70 for the multiplier. This would leave expectancy/StdDev of R with just a target of 0.1, to reach the 7 t-tests score.

Now, if you had 1,000,000 tranasctions in your OOS test....

The concept of limiting the trade count does make sense to me. Maybe 100 is too low, and should be set higher. There does come a point whereby the sqrt(N) part of the equation will render the rest of the equation irrelevant once N gets too large.

$0.02

Bing

--- In amibroker@xxxxxxxxxxxxxxx, Howard B <howardbandy@xxx> wrote:
>

> Hi Zozu --
>
> I must disagree with Van Tharp on this.
>
> If the runs are truly out-of-sample, then each and every one contributes to
> the computation. It makes no sense to limit the count to 100. It is poor
> procedure to limit the count. It is bad science to limit the count. Do not
> limit the count.
>
> If the runs are in-sample, then the test has no meaning anyway. Computing
> the t-test statistic using any N will be misleading. Do not even do the
> computation. If a decision to trade a system is made after computing the
> t-test statistic on trades that came solely from in-sample results, there is
> an extremely high probability that a Type I error will be committed. That
> is, the trader will believe that his system is better than random, when it
> is in fact not better than random. Type I errors result in loss of money.
>
> Thanks,
> Howard
>
>

> On Tue, Oct 13, 2009 at 10:54 AM, zozuzoza <zozuka@xxx> wrote:
>
> >
> >
> > Hi Howard,
> >
> > Limiting the number of N doesn't mean that you are not using all trades for
> > the calculation of SQN. Only the sqrt(N) part of the formula is limited in
> > order not to distort the results if there are many trades. It makes sense.
> > The other part of the formula does count on all the trades.
> >
> > Zozu
> >
> >
> >
>