Re: [amibroker] Re: Is the Walk forward study useful?, AmiBroker Email List Archive

On Fri, Oct 16, 2009 at 3:12 AM, bingk66 <bing.kwok@xxxxxxxxxxxxxxx> wrote:

Hi Howard,

Many thanks for your detailed response.

You'll be pleased to know that I am still repeating "keep a positive expectancy, limit losing trades, trade frequently"..... lol

Given your point that risk of bankruptcy increases dramatically as the standard deviation of losing trades increases, would it be fair to say that you prefer to use standard deviation of losing trades as the denominator for the t-test calculation, as opposed to the normal standard deviation ?

Also would appreciate your opinion about using the standard deviation of trades that are below the mean, as opposed to just the losing trades ?

Bing

--- In amibroker@xxxxxxxxxxxxxxx, Howard B <howardbandy@xxx> wrote:
>
> Hi Bing, and all --
>
> I think we need a reality check.
>
> First -- computing the t-test, or any other metric using the results from
> in-sample runs has no value. Almost any trading system can have the
> parameters, logic, time frame, and reporting period adjusted so that the
> in-sample results are very profitable. In my speeches, I show a series of
> slides, each slide displayed in two steps. Step 1 is the in-sample equity
> after optimizing over an in-sample period. The chart ends at the end of the
> in-sample period. It always looks good. When the results of testing a
> system do not look good, we do not spend any more time on that system, but
> go on to others where it does look good. Step 2 adds an out-of-sample data
> period, applies the same system, and shows the out-of-sample equity. For
> some systems the equity continues to rise, for some it goes flat, for others
> it drops sharply. There is no way to tell what will happen in the
> out-of-sample period without testing the out-of-sample period. That is --
> there is no information in the in-sample results that predict the
> out-of-sample results. There is no way to know whether the model has fit
> itself to the signal component of the data that contains the pattern leading
> to profitable trades or to some noise component of the data that does not
> exist in the out-of-sample data.
>
> Second -- if someone has a trading system that has produced a truly
> out-of-sample set of closed trades where the t-test, or any other fitness
> metric, would be embarrassingly high without limiting the value used as the
> number of data points, he or she should call me. I can help them find a
> semitrailer large enough to carry all the money they will make trading that
> system. Limiting the number used as N is a non-issue, in addition to being
> bad procedure.
>
> Tharp admits that he is not a statistician, and that he finds some of the
> mathematics involved in position sizing and fitness function analysis to be
> at his comfort limit. It is apparent to me that the systems he shows as
> standards for SQNs of 2, 3, 4, 5, and so forth are not the result of trading
> system runs -- either in-sample or out-of-sample -- to which he has applied
> his metric; they are artificial examples constructed so the results turn out
> as he wants them to. That is not a bad thing in itself, but when he
> suggests that we should go on searching for systems with SQNs of 10 or more
> (page 279 of "Definitive Guide to Position Sizing"), that is completely
> unrealistic and will send naive systems developers off on windmill-tilting
> quests that will never be successful.
>
> One of the difficulties using Tharp's data sets is that the standard
> deviation of losing trades is zero for some of them. That is not only
> unrealistic, but it makes computation of metrics that include standard
> deviation of losing trades, such as Sortino ratio, difficult.
>
> Do the following experiment. Put together a data set that represents
> potentially realistic trading results that you hope to achieve -- be
> optimistic, but realistic. If you would expect one trade a week, a set of
> 52 data points represents a year. 252 data points if you would have a
> trading result every day. Each data point is the number of dollars gained
> or lost from that closed trade, based on trading a single unit -- one
> futures contract, one hundred shares of stock, or whatever your one unit
> is. Compute the mean, standard deviation, and t-test score -- where the
> t-test is "what is the level of confidence that the mean is greater than
> 0". Put the data into a text file and use it as input to Equity Monaco,
> ProSizer, or Market System Analyzer. Run some Monte Carlo simulations to
> see what the equity could look like from trading one year. Systems with
> t-test scores, based on expectancy, of 3 will make you incredible amounts of
> money. Dummy up a data set with a t-test / SQN of 10 and you will see how
> unrealistic that is.
>
> The critical elements needed to apply aggressive position sizing are:
> trading something that can be scaled up as desired, positive expectancy,
> limited semi-deviation (standard deviation of losing trades), and frequent
> trading.
>
> Being able to limit the amount lost on a losing trade is essential. Risk of
> bankruptcy (and all trading systems have a non-zero probability of going
> bankrupt) increases dramatically as the standard deviation of losing trades
> increases. Dummy up some data and runs some more tests. The effect the
> amount lost on losing trades has on system performance is scary.
>
> Now go back to your trading system development platform (AmiBroker, of
> course) and test your trading systems, this time focusing on limiting the
> standard deviation of losing trades. If your system holds a long time (more
> than a week or two), pay attention to Maximum Adverse Excursion. If you
> attempt to apply aggressive position sizing to a system that has reasonable
> results for closed trades, but has large MAE intra-trade, you will get
> stopped out (or scared out) intra-trade.
>
> Remember -- if you are applying position sizing to your trading system, your
> largest loss will come when you have your largest position. Read Ralph
> Vince.
>
> As usual -- be careful to base your analysis and decision whether to trade
> any system on truly out-of-sample results. Decisions based on in-sample or
> contaminated out-of-sample results seriously underestimate the probability
> of bankruptcy.
>
> Repeat after me -- keep a positive expectancy, limit losing trades, trade
> frequently.
>
> Thanks for listening,
> Howard
>
>
> On Thu, Oct 15, 2009 at 1:11 AM, bingk66 <bing.kwok@xxx> wrote:
>
> >
> >
> > Hi Howard,
> >
> > If there are no means to limit the number of transactions in the calcs,
> > then one seriously runs the risk of challenging the mystical t-test score of
> > 7 that you spoke about previously.
> >
> > As an example, if the OOS test was run over a 5 year period with 5000
> > transactions (a mere 1000 transaction/year, which is not excessive,
> > especially for very short term trades), sqrt(5000) alone would yield in
> > excess of 70 for the multiplier. This would leave expectancy/StdDev of R
> > with just a target of 0.1, to reach the 7 t-tests score.
> >
> > Now, if you had 1,000,000 tranasctions in your OOS test....
> >
> > The concept of limiting the trade count does make sense to me. Maybe 100 is
> > too low, and should be set higher. There does come a point whereby the
> > sqrt(N) part of the equation will render the rest of the equation irrelevant
> > once N gets too large.
> >
> > $0.02
> >
> > Bing
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%40yahoogroups.com>, Howard B

> > <howardbandy@> wrote:
> > >
> > > Hi Zozu --
> > >
> > > I must disagree with Van Tharp on this.
> > >
> > > If the runs are truly out-of-sample, then each and every one contributes
> > to
> > > the computation. It makes no sense to limit the count to 100. It is poor
> > > procedure to limit the count. It is bad science to limit the count. Do
> > not
> > > limit the count.
> > >
> > > If the runs are in-sample, then the test has no meaning anyway. Computing
> > > the t-test statistic using any N will be misleading. Do not even do the
> > > computation. If a decision to trade a system is made after computing the
> > > t-test statistic on trades that came solely from in-sample results, there
> > is
> > > an extremely high probability that a Type I error will be committed. That
> > > is, the trader will believe that his system is better than random, when
> > it
> > > is in fact not better than random. Type I errors result in loss of money.
> > >
> > > Thanks,
> > > Howard
> > >
> > >
> > > On Tue, Oct 13, 2009 at 10:54 AM, zozuzoza <zozuka@> wrote:
> > >
> > > >
> > > >
> > > > Hi Howard,
> > > >
> > > > Limiting the number of N doesn't mean that you are not using all trades
> > for
> > > > the calculation of SQN. Only the sqrt(N) part of the formula is limited
> > in
> > > > order not to distort the results if there are many trades. It makes
> > sense.
> > > > The other part of the formula does count on all the trades.
> > > >
> > > > Zozu
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>