--- In
amibroker@xxxxxxxxxxxxxxx, "Mike" <sfclimbers@xxx> wrote:
>
> Howard,
>
> I agree that reoptimization is valid, necessary even, at any time
> after going live. Specifically; Any time measured performance
values
> stray significantly from what was expected.
>
> I did not mean to imply, nor do I think that Pardo was suggesting,
> that the system had to be traded for the duration of the calculated
> out of sample (OOS) period before reoptimization could occur.
>
> I also subscribe to your advice, of iterating over data periods, to
> find the most suitable combination of in sample (IS) and OOS.
>
> I take Pardo's 1/8 to 1/3 ratio rule of thumb at face value (i.e. a
> suggested range in which to perform the above iterations). Faced
with
> so many possible combinations, we have to start somewhere. However,
> if my tests are showing noticably improving results closer to a 1/3
> ratio than a 1/8 ratio, I would not hesitate to additionally try
1/2,
> 3/4, etc.
>
> You do raise a new question though;
>
> Are you suggesting that you would be open to changing the duration
of
> your IS lookback window at subsequent reoptimizations? If so,
> wouldn't that be changing the underlying nature of the strategy?
>
> In other words, if the targeted signal is no longer sufficiently
> prevalent using the original IS lookback span, might that not be a
> red flag that the system as a whole is failing, as opposed to
simply
> no longer being in sync with the market?
>
> Or, do you consider the IS lookback period to be optimizable at
each
> reoptimization in the same vein as any other parameter (e.g.
> comparable to extending the lookback of a moving average parameter)?
>
> Thanks,
>
> Mike
>
> --- In
amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@> wrote:
> >
> > Hi Mike --
> >
> > I have read both of Bob Pardo's books. Bob and I had a telephone
> > conversation several years ago, and we have exchanged emails
> recently. I
> > agree with much of what he writes, but have different views in
some
> areas.
> >
> > In my opinion, the only way to determine how long the in-sample
> period
> > should be is to run tests, varying the length of the in-sample
> period and
> > observing the performance of the system on the following out-of-
> sample
> > data. The "sweet spot" will depend on both the trading system
> logic and the
> > data series that it is processing.
> >
> > The data being processed is composed of one or more of the
following
> > components: long-term trend, long-term cycle, short term cycle,
> pattern,
> > seasonality, and noise. There may be other components as well,
> just include
> > them in the list. Long-term and short-term are relative. The
> trading
> > system logic is written to identify some component (usually just
> one of the
> > components) that precedes profitable trading opportunities, hope
> that those
> > features persist beyond the in-sample period over which the
system
> is
> > developed, and can be profitably traded. The feature(s) being
> identified
> > are the signal portion of the data. Everything else is noise.
> Even if some
> > part of the "everything else" contains features that some other
> trading
> > system can identify and profitably trade, it is noise to any
system
> being
> > tested that does not identify and remove it or compensate for it.
> >
> > To shorten the explanation, the system is looking for the signal
> among the
> > noise. The ease with which the signal can be identified depends
> on both
> > the logic of the system and the characteristics of the data. It
is
> not
> > possible to generalize without knowing both.
> >
> > Similarly, the only way to determine how long the out-of-sample
> period
> > should be is to run tests. The systems we write are static.
They
> may have
> > logic that allows the parameters and the logic to adjust
> themselves, but the
> > system does not change. The characteristics of the data being
> processed are
> > dynamic. A trading system remains profitable only as long as the
> system and
> > the data it models remain in synchronization. Clever systems can
> (but not
> > always will) remain synchronized better than simple systems, but
the
> > synchronization is required for the trades to be profitable.
The
> period of
> > time that the system remains profitable is the period of time
that
> the
> > system and the market remain in sync. That period determines the
> schedule
> > for re-optimization (the maximum time between re-optimizations)
and
> that is
> > the length of the out-of-sample period. There is no way to
> determine that
> > length without testing the specific system on the specific data.
> The length
> > of out-of-sample profitability will not remain constant, but will
> vary.
> > There is no relationship between the length of the in-sample
period
> and the
> > length of the out-of-sample period.
> >
> > The best we can hope for is a high level of confidence that our
> newly
> > designed, newly optimized, newly re-optimized system performs
well
> in real
> > trading. There are no guarantees. The best way to gain
confidence
> is to
> > observe as many in-sample to out-of-sample transitions as
possible
> and learn
> > what to expect. The best way to do that is to run automated walk
> forward
> > testing with fairly short out-of-sample periods. In an automated
> walk
> > forward test, the length of the out-of-sample period is often the
> same as
> > the re-optimization schedule.
> >
> > Once the system has passed the validation procedure, and the
> designer
> > understands what to expect in the period immediately following the
> > re-optimization, re-optimization is permitted at any time. There
> is no need
> > to wait the previously defined out-of-sample length time.
> >
> > Although you did not raise the question in your posting, there is
> another
> > component of system design and testing that is critically
> important. My
> > opinion is that the whole process begins with the person or
> organization
> > that is going to trade the system defining the criteria by which
the
> > acceptability of each system, or alternative system, is judged.
> Choice of
> > this "objective function" is very personal, and it incorporates
> most of
> > those features that the psychology of trading experts talk about
> when they
> > help us learn to accept a trading system. With the correct
choice
> of the
> > objective function, every system that passes as acceptable is
> already one
> > that the trader will be comfortable with.
> >
> > And, importantly, it is the score on the objective function that
> determines
> > which of the alternative systems will be chosen as "best" and
used
> to trade
> > forward in the out-of-sample period.
> >
> > Consequently, I recommend that the objective function be chosen
> first. Then
> > the system designed, tested, and validated using the walk forward
> process,
> > and letting the system and the data it is reading determine how
> long the
> > in-sample period is and how long the out-of-sample period is.
> >
> > Thanks for listening,
> > Howard
> >
www.quantitativetradingsystems.com
> >
> >
> >
> >
> > On Wed, Apr 9, 2008 at 10:08 AM, Howard B <howardbandy@> wrote:
> >
> > > Hi Louis --
> > >
> > > I agree that there is a serious problem when the only data that
is
> > > available contains no period that is similar to what is
expected
> in the
> > > future.
> > >
> > > Artificial data has no value.
> > >
> > > Using data that is earlier in time than the in-sample period
has
> limited
> > > value. You can test earlier data, but you will over-estimate
the
> > > performance that you can expect in the future.
> > >
> > > Are there other tickers that are closely related that have data
> for the
> > > periods you would like to test?
> > >
> > > In the end, you will need to make a decision on whether to
place
> actual
> > > trades. And that decision must be based on your understanding
of
> and
> > > confidence in your system. The only way to gain that
confidence
> is by
> > > observing the transitions from in-sample testing to out-of-
sample
> simulated
> > > trading.
> > >
> > > Thanks,
> > > Howard
> > >
> > > On Tue, Apr 8, 2008 at 10:37 PM, Mike <sfclimbers@> wrote:
> > >
> > > > Howard's comments are consistent with those of Robert Pardo
> (The
> > > > Evaluation and Optimization of Trading Strategies, Wiley
2008),
> with
> > > > respect to training periods.
> > > >
> > > > Pardo recognizes that there is a tradeoff between more robust
> > > > strategies which require longer in sample training periods,
> require
> > > > fewer reoptimizations, trade for longer out of sample periods
> and are
> > > > generally less profitable, vs. more responsive strategies
which
> > > > require shorter in sample training periods, require more
> frequent
> > > > reoptimizations, can only trade for shorter out of sample
> periods and
> > > > are generally more profitable.
> > > >
> > > > Pardo suggests that strategies generating more frequent
signals
> can
> > > > use shorter in sample training windows since they generate the
> > > > minimum 30+ trades sooner than strategies that generate less
> frequent
> > > > signals. But, that in any case, one should try to use an in
> sample
> > > > period sufficiently long to capture bull, bear, and sideways
> markets.
> > > >
> > > > Further, when first trying to evaluate the worth of the
> strategy,
> > > > Pardo suggests backtesting the in sample history in segments
> rather
> > > > than one shot (e.g. 10 year history divided into five 2 year
> > > > segments). This gives you better insight as to whether the
> results
> > > > are due to a single segment or are consistent accross
segments,
> and
> > > > provides insight to your eventual in sample/out of sample
> periods for
> > > > Walk Forward Optimization.
> > > >
> > > > Finally, Pardo suggests that regardless of whether a long or
> short
> > > > training period is used, a rule of thumb for in sample vs.
out
> of
> > > > sample is for out of sample to be between 1/8 to 1/3 of the
in
> sample
> > > > period (e.g. 24/8 = 3 and 24/3 = 8, so it would be "safe" to
> trade
> > > > out of sample for 3 - 8 months based on a system backtested
> over 24
> > > > months.
> > > >
> > > > Yet another good book covering the topic. I reccomend it.
> > > >
> > > > Mike
> > > >
> > > >
> > > > --- In
amibroker@xxxxxxxxxxxxxxx <amibroker%
>
40yahoogroups.com>, "Howard
> > > > B" <howardbandy@> wrote:
> > > > >
> > > > > Hi Louis, and all --
> > > > >
> > > > > I know David Aronson, respect him, and like and recommend
his
> book.
> > > > >
> > > > > My view is that the in-sample period should be as short as
> > > > practical. My
> > > > > thought is that: the system we are testing / trading is
> trying to
> > > > recognize
> > > > > the signal from among the noise; and the signal patterns are
> > > > changing over
> > > > > time. So the length of the in-sample period is a tradeoff --
> short
> > > > to be
> > > > > able to change as the characteristics of the underlying
market
> > > > change, but
> > > > > not so short that the system is over-fit to the noise
rather
> than
> > > > learns the
> > > > > signal.
> > > > >
> > > > > You can test this in AmiBroker. Have your system ready to
buy
> and
> > > > sell. In
> > > > > the Automatic Analysis window, use Settings and set up the
> Walk
> > > > Forward
> > > > > parameters. Try an in-sample period of 10 years, an out-of-
> sample
> > > > period of
> > > > > 6 months or 1 year. Run Optimize > Walk Forward and look at
> the in-
> > > > sample
> > > > > and out-of-sample equity curves. Shorten the length of the
in-
> > > > sample period
> > > > > to 9, then 8, then 7, ... then 1 year, keeping the out-of-
> sample
> > > > period
> > > > > unchanged. Depending on your system and the market it is
> trading,
> > > > you may
> > > > > find that there is a sweet spot in the length of the in-
sample
> > > > data. If so,
> > > > > that is the amount of data that allows your system to
> recognize the
> > > > signal
> > > > > without being overwhelmed by the noise.
> > > > >
> > > > > Thanks,
> > > > > Howard
> > > > >
> > > > >
> > > > > On Tue, Apr 8, 2008 at 8:56 AM, Louis Préfontaine
> <rockprog80@>
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I've been thinking a lot lately, and here is something I
> would
> > > > like to
> > > > > > have your opinion on.
> > > > > >
> > > > > > I've been introduced to automated systems by a trend
> following
> > > > book which
> > > > > > related how some trend followers built their systems in
the
> 70s
> > > > or 80s and
> > > > > > got rich with them, and how their system did not really
> change
> > > > all this
> > > > > > time. They didn't change their system because they say the
> > > > market does NOT
> > > > > > change. They looked at historic market data from the
1800s
> and
> > > > the market
> > > > > > was as it is right now. So they say.
> > > > > >
> > > > > > On the other side, lately I have been introduced to the
> concept of
> > > > > > ever-changing markets and have had a hard time trying to
> build my
> > > > system.
> > > > > > Got a very promising start with a system getting around
15-
> 20%
> > > > average for
> > > > > > April 2007 to April 2008 (with little drawdown, which
mean
> that
> > > > with
> > > > > > leverage I can boost this a lot). In any variation over
> > > > thousands of stocks
> > > > > > the results were nearly all positives. But then, I tested
> that
> > > > same system
> > > > > > for the years 2000 to 2008, and that was disappointing.
> Even more
> > > > > > disappointing from 2001 to 2003, another troubled market
> like the
> > > > one we are
> > > > > > in right now.
> > > > > >
> > > > > > So here I am, wondering where to go from now. Aronson's
> > > > excellent book
> > > > > > talk about the importance of having a very large sample
of
> data.
> > > > But the
> > > > > > problem is: the larger the data, the more "historic" it
> gets and
> > > > the less it
> > > > > > seems to work.
> > > > > >
> > > > > > Is my system not working, or did the markets really
change?
> Do I
> > > > need to
> > > > > > make it more robust (that is, it MUST make profit even
from
> 2001
> > > > to 2003),
> > > > > > or can I have complete faith in what happened in the last
> year?
> > > > > >
> > > > > > All those questions... Would be nice to read what you
think
> > > > about this.
> > > > > >
> > > > > > Louis
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
>