Re: [amibroker] Re: Another tough question..., AmiBroker Email List Archive

On Tue, Apr 8, 2008 at 10:37 PM, Mike <sfclimbers@xxxxxxxxx> wrote:

Howard's comments are consistent with those of Robert Pardo (The
Evaluation and Optimization of Trading Strategies, Wiley 2008), with
respect to training periods.

Pardo recognizes that there is a tradeoff between more robust
strategies which require longer in sample training periods, require
fewer reoptimizations, trade for longer out of sample periods and are
generally less profitable, vs. more responsive strategies which
require shorter in sample training periods, require more frequent
reoptimizations, can only trade for shorter out of sample periods and
are generally more profitable.

Pardo suggests that strategies generating more frequent signals can
use shorter in sample training windows since they generate the
minimum 30+ trades sooner than strategies that generate less frequent
signals. But, that in any case, one should try to use an in sample
period sufficiently long to capture bull, bear, and sideways markets.

Further, when first trying to evaluate the worth of the strategy,
Pardo suggests backtesting the in sample history in segments rather
than one shot (e.g. 10 year history divided into five 2 year
segments). This gives you better insight as to whether the results
are due to a single segment or are consistent accross segments, and
provides insight to your eventual in sample/out of sample periods for
Walk Forward Optimization.

Finally, Pardo suggests that regardless of whether a long or short
training period is used, a rule of thumb for in sample vs. out of
sample is for out of sample to be between 1/8 to 1/3 of the in sample
period (e.g. 24/8 = 3 and 24/3 = 8, so it would be "safe" to trade
out of sample for 3 - 8 months based on a system backtested over 24
months.

Yet another good book covering the topic. I reccomend it.

Mike

--- In amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@xxx> wrote:
>
> Hi Louis, and all --
>
> I know David Aronson, respect him, and like and recommend his book.
>
> My view is that the in-sample period should be as short as
practical. My
> thought is that: the system we are testing / trading is trying to
recognize
> the signal from among the noise; and the signal patterns are
changing over
> time. So the length of the in-sample period is a tradeoff -- short
to be
> able to change as the characteristics of the underlying market
change, but
> not so short that the system is over-fit to the noise rather than
learns the
> signal.
>
> You can test this in AmiBroker. Have your system ready to buy and
sell. In
> the Automatic Analysis window, use Settings and set up the Walk
Forward
> parameters. Try an in-sample period of 10 years, an out-of-sample
period of
> 6 months or 1 year. Run Optimize > Walk Forward and look at the in-
sample
> and out-of-sample equity curves. Shorten the length of the in-
sample period
> to 9, then 8, then 7, ... then 1 year, keeping the out-of-sample
period
> unchanged. Depending on your system and the market it is trading,
you may
> find that there is a sweet spot in the length of the in-sample
data. If so,
> that is the amount of data that allows your system to recognize the
signal
> without being overwhelmed by the noise.
>
> Thanks,
> Howard
>
>
> On Tue, Apr 8, 2008 at 8:56 AM, Louis Préfontaine <rockprog80@xxx>

> wrote:
>
> > Hi,
> >
> > I've been thinking a lot lately, and here is something I would
like to
> > have your opinion on.
> >
> > I've been introduced to automated systems by a trend following
book which
> > related how some trend followers built their systems in the 70s
or 80s and
> > got rich with them, and how their system did not really change
all this
> > time. They didn't change their system because they say the
market does NOT
> > change. They looked at historic market data from the 1800s and
the market
> > was as it is right now. So they say.
> >
> > On the other side, lately I have been introduced to the concept of
> > ever-changing markets and have had a hard time trying to build my
system.
> > Got a very promising start with a system getting around 15-20%
average for
> > April 2007 to April 2008 (with little drawdown, which mean that
with
> > leverage I can boost this a lot). In any variation over
thousands of stocks
> > the results were nearly all positives. But then, I tested that
same system
> > for the years 2000 to 2008, and that was disappointing. Even more
> > disappointing from 2001 to 2003, another troubled market like the
one we are
> > in right now.
> >
> > So here I am, wondering where to go from now. Aronson's
excellent book
> > talk about the importance of having a very large sample of data.
But the
> > problem is: the larger the data, the more "historic" it gets and
the less it
> > seems to work.
> >
> > Is my system not working, or did the markets really change? Do I
need to
> > make it more robust (that is, it MUST make profit even from 2001
to 2003),
> > or can I have complete faith in what happened in the last year?
> >
> > All those questions... Would be nice to read what you think
about this.
> >
> > Louis
> >
> >
>