> Hi Mike --
>
> I have read both of Bob Pardo's books. Bob and I had a telephone
> conversation several years ago, and we have exchanged emails
recently. I
> agree with much of what he writes, but have different views in some
areas.
>
> In my opinion, the only way to determine how long the in-sample
period
> should be is to run tests, varying the length of the in-sample
period and
> observing the performance of the system on the following out-of-
sample
> data. The "sweet spot" will depend on both the trading system
logic and the
> data series that it is processing.
>
> The data being processed is composed of one or more of the following
> components: long-term trend, long-term cycle, short term cycle,
pattern,
> seasonality, and noise. There may be other components as well,
just include
> them in the list. Long-term and short-term are relative. The
trading
> system logic is written to identify some component (usually just
one of the
> components) that precedes profitable trading opportunities, hope
that those
> features persist beyond the in-sample period over which the system
is
> developed, and can be profitably traded. The feature(s) being
identified
> are the signal portion of the data. Everything else is noise.
Even if some
> part of the "everything else" contains features that some other
trading
> system can identify and profitably trade, it is noise to any system
being
> tested that does not identify and remove it or compensate for it.
>
> To shorten the explanation, the system is looking for the signal
among the
> noise. The ease with which the signal can be identified depends
on both
> the logic of the system and the characteristics of the data. It is
not
> possible to generalize without knowing both.
>
> Similarly, the only way to determine how long the out-of-sample
period
> should be is to run tests. The systems we write are static. They
may have
> logic that allows the parameters and the logic to adjust
themselves, but the
> system does not change. The characteristics of the data being
processed are
> dynamic. A trading system remains profitable only as long as the
system and
> the data it models remain in synchronization. Clever systems can
(but not
> always will) remain synchronized better than simple systems, but the
> synchronization is required for the trades to be profitable. The
period of
> time that the system remains profitable is the period of time that
the
> system and the market remain in sync. That period determines the
schedule
> for re-optimization (the maximum time between re-optimizations) and
that is
> the length of the out-of-sample period. There is no way to
determine that
> length without testing the specific system on the specific data.
The length
> of out-of-sample profitability will not remain constant, but will
vary.
> There is no relationship between the length of the in-sample period
and the
> length of the out-of-sample period.
>
> The best we can hope for is a high level of confidence that our
newly
> designed, newly optimized, newly re-optimized system performs well
in real
> trading. There are no guarantees. The best way to gain confidence
is to
> observe as many in-sample to out-of-sample transitions as possible
and learn
> what to expect. The best way to do that is to run automated walk
forward
> testing with fairly short out-of-sample periods. In an automated
walk
> forward test, the length of the out-of-sample period is often the
same as
> the re-optimization schedule.
>
> Once the system has passed the validation procedure, and the
designer
> understands what to expect in the period immediately following the
> re-optimization, re-optimization is permitted at any time. There
is no need
> to wait the previously defined out-of-sample length time.
>
> Although you did not raise the question in your posting, there is
another
> component of system design and testing that is critically
important. My
> opinion is that the whole process begins with the person or
organization
> that is going to trade the system defining the criteria by which the
> acceptability of each system, or alternative system, is judged.
Choice of
> this "objective function" is very personal, and it incorporates
most of
> those features that the psychology of trading experts talk about
when they
> help us learn to accept a trading system. With the correct choice
of the
> objective function, every system that passes as acceptable is
already one
> that the trader will be comfortable with.
>
> And, importantly, it is the score on the objective function that
determines
> which of the alternative systems will be chosen as "best" and used
to trade
> forward in the out-of-sample period.
>
> Consequently, I recommend that the objective function be chosen
first. Then
> the system designed, tested, and validated using the walk forward
process,
> and letting the system and the data it is reading determine how
long the
> in-sample period is and how long the out-of-sample period is.
>
> Thanks for listening,
> Howard
>
www.quantitativetradingsystems.com
>
>
>
>