Re: [amibroker] Re: Another tough question..., AmiBroker Email List Archive

On Mon, Apr 14, 2008 at 8:08 PM, Mike <sfclimbers@xxxxxxxxx> wrote:

Howard,

Thanks for your reply. If I may reuse your example, I would like to
ask for a few clarifications.

As per your example, let us assume that there is an exploitable 20
day cycle present in the data beginning sometime prior to 1995 which
persists from 1995 until 2000 at which time it changes to a 10 day
cycle.

Let us assume that I am designing a new system today. To leave myself
room for validation, I opt to design based on year 2000 data.

Let us assume that my design efforts against year 2000 data reveal
the cycle in its 10 day form. I capture this as a parameter named
CycleLength.

Finally, for simplicity below, let us assume that after some
experimentation I conclude that a 1 year IS period paired with a 1
year OOS period gives the most consistent walk forward results.

As per your example, we see our walk forwards selecting a CycleLength
optimum of 10 days for use in 2001 OOS and beyond.

At this point, I have 7 OOS data points (1 year OOS for each of 2001 -
2007) with which to judge the robustness of my system.

Question: Given that I have not yet traded this system, and the
initial selection of year 2000 data was arbitrary, is it your
contention that to now perform walk forward beginning with IS of 1994
and OOS of 1995 is trading yesterday's chart?

e.g.
1994 IS for 12 mths.
1995 OOS for 12 mths.
1995 IS for 12 mths.
1996 OOS for 12 mths.
...
2000 IS for 12 mths.
2001 OOS for 12 mths.
...

In both cases the walk forward OOS periods would be calculated based
on IS optimizations that preceded each of those OOS periods. The
earlier years would end up recognizing the CycleLength value as 20
days and the subsequent years would conclude 10 days.

The results would be exactly what would have resulted had I opted to
design against 1994 data in the first place, rather than against 2000
data. And, in this example, the results from the OOS data points
beginning with 2001 would be identical to the 7 found in the case of
above of working only with data from 2000 and beyond.

I understand that with the passing of time, more and more
inefficiencies are removed from the market by more and more trading.
As such, I accept that earlier OOS periods may be less representitive
than later OOS periods.

Yet, at the same time, wouldn't there be value in having the larger
sample size of 13 OOS data points (i.e. 1 year OOS for each of 1995 -
2007) rather than just 7 (2001 - 2007)?

Would you consider the possibly exagerated earlier OOS returns to be
more damaging than the risk of chance distorting the smaller sample
size?

Even if exagerated, with respect to size of returns, might not the
statistic of winning periods vs. losing periods still be a strong
indicator of robustness (assuming the trend of winning periods was
not downward sloping as the OOS periods get closer and closer to
today's date)?

Thanks,

Mike

--- In amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@xxx> wrote:
>
> Hi Mike --
>
> On having the out-of-sample data period earlier than the in-sample
data
> period ---
>
> It is my contention that a successful (profitable) trading system is
> successful because it identifies inefficiencies in the market it is
> trading; and every profitable trade removes a portion of those
> inefficiencies. So assume I discover a trading system in the year
2000 and
> begin to trade it. As this system becomes more widely used and
makes
> profitable trades, it will become less profitable since there is
less
> inefficiency remaining to remove.
>
> When I perform the walk forward testing, each step simulates the
process I
> will go through when I eventually move the system from development
to
> production. Optimize over some in-sample period, then trade for
some length
> of time. Repeating the optimize -- trade cycle as necessary. In
no case
> can I ever perform the system development over one period of time
and then
> trade an earlier period. That would be trading yesterday's chart.
>
> So I contend that the OOS period must be later than the IS period
for two
> reasons.
> 1. The process of walking forward is intended to give us
confidence for the
> process we will eventually go through when trading. Since trading
is done
> on data that is more recent than development, the validation
process should
> use data that is in that same time relationship.
> 2. Using the example above of a system developed in 2000.
Beginning in
> 2001, the fact that that system exists makes it more difficult to be
> profitable in the future -- some of the inefficiency has been
removed from
> the data. But the historical data for 1999 covers a period when
that system
> did not exist and the full inefficiency is in the data. Using an
earlier
> period to test will over-estimate the likely future performance.
The only
> way to determine the magnitude of the over-estimate is to test over
a data
> period that is more recent than the development period.
>
> Continuing on.
>
> Longer IS periods do give a trading system more data to learn.
Whether that
> is an advantage or a disadvantage depends on the system and the
data. If
> there is some fundamental reason the system is profitable, and that
reason
> remains valid through out the period, then more data gives better
learning
> of the signal portion of the data -- the signal to noise ratio
improves.
> But, as soon as the characteristics of the data change, using more
data
> creates a problem, since there are now multiple conditions to
learn. If the
> system code is clever, it may be able to learn more than one
thing. If it
> is not clever, then it will still try to learn, but it will learn
one
> condition or another or a compromise; and being profitable in the
future
> will be due to luck as much as to proper system design.
>
> For example -- a very simplistic example, but it will illustrate my
point.
>
> Assume the data contains a 20 day cycle for five years, after which
the
> period changes to 10 days for five more years. If a system is
asked to
> learn the cycle and the IS period contains only 20 day cycles, the
system
> will synchronize itself to the 20 days and be profitable as long as
the
> cycle remains at 20 days. During the transition from 20 to 10
days, the
> system will not be profitable. Then during the 10 day section it
will again
> be in sync and profitable.
>
> On the other hand, is the system is given all ten years of data and
asked to
> identify the cycle length, it will be inaccurate for both the 20
day portion
> and the 10 day portion -- it cannot learn the length because the
length
> changes within the IS period.
>
> My recommendation is that the in-sample period should be as short
as is
> practical. Long enough to allow the system to learn the current
> characteristics of the data, and long enough that the system learns
the
> signal rather than the noise. And short enough to allow the system
to learn
> the current conditions as well as possible. There is no way to
estimate in
> advance what that ideal length should be -- it depends on both the
system
> and the data.
>
> And you are correct, there are a couple of areas where I disagree
with Bob
> Pardo.
>
> Thanks for listening,
> Howard
>
>
>
>
> On Sat, Apr 12, 2008 at 11:45 PM, Mike <sfclimbers@xxx> wrote:
>
> > > I did not mean to imply, nor do I think that Pardo was
suggesting,
> > > that the system had to be traded for the duration of the
calculated
> > > out of sample (OOS) period before reoptimization could occur.
> >
> > I stand corrected. The chapter I had been summarizing(Chapter 6)
did
> > not express that the OOS period should be fixed. However, upon
review
> > of Chapter 11, Pardo does say that the OOS period should remain
> > constant and traded to completion before reoptimization.
> >
> > If you are still reading Howard, I wonder if you would care to
> > comment on another point?
> >
> > In Chapter 18 of your book, "In Sample Out of Sample", you have
> > stated that using an out of sample period that predates the in
sample
> > period will have no predictive value towards the future. You go
on to
> > say that a "jackknife" approach over 10 periods really only has 1
out
> > of sample period rather than 10.
> >
> > I'm trying to compare this to Pardo's suggestion of starting with
> > reasonably recent data for initial testing. Then doing walk
forward
> > over at least 10 - 20 years of history in an attempt to get as
many
> > OOS periods as possible for summary analysis.
> >
> > Is your assertion regarding jackknife testing a difference of
opinion
> > with Pardo's practice of starting a walk forward earlier in time
than
> > the time period upon which the strategy was first devised?
> >
> > Or, is the nature of walk forward sufficiently different from
> > jackknife in that all walk forward IS periods are prior to each
OOS
> > period, whereas jackknife would have a mix of prior and later IS
> > periods when calculating most of the OOS periods?
> >
> > Thanks.
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com>, "Mike"

> > <sfclimbers@> wrote:
> > >
> > > Howard,
> > >
> > > I agree that reoptimization is valid, necessary even, at any
time
> > > after going live. Specifically; Any time measured performance
> > values
> > > stray significantly from what was expected.
> > >
> > > I did not mean to imply, nor do I think that Pardo was
suggesting,
> > > that the system had to be traded for the duration of the
calculated
> > > out of sample (OOS) period before reoptimization could occur.
> > >
> > > I also subscribe to your advice, of iterating over data
periods, to
> > > find the most suitable combination of in sample (IS) and OOS.
> > >
> > > I take Pardo's 1/8 to 1/3 ratio rule of thumb at face value
(i.e. a
> > > suggested range in which to perform the above iterations). Faced
> > with
> > > so many possible combinations, we have to start somewhere.
However,
> > > if my tests are showing noticably improving results closer to a
1/3
> > > ratio than a 1/8 ratio, I would not hesitate to additionally try
> > 1/2,
> > > 3/4, etc.
> > >
> > > You do raise a new question though;
> > >
> > > Are you suggesting that you would be open to changing the
duration
> > of
> > > your IS lookback window at subsequent reoptimizations? If so,
> > > wouldn't that be changing the underlying nature of the strategy?
> > >
> > > In other words, if the targeted signal is no longer sufficiently
> > > prevalent using the original IS lookback span, might that not
be a
> > > red flag that the system as a whole is failing, as opposed to
> > simply
> > > no longer being in sync with the market?
> > >
> > > Or, do you consider the IS lookback period to be optimizable at
> > each
> > > reoptimization in the same vein as any other parameter (e.g.
> > > comparable to extending the lookback of a moving average
parameter)?
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com>, "Howard
> > B" <howardbandy@> wrote:
> > > >
> > > > Hi Mike --
> > > >
> > > > I have read both of Bob Pardo's books. Bob and I had a
telephone
> > > > conversation several years ago, and we have exchanged emails
> > > recently. I
> > > > agree with much of what he writes, but have different views in
> > some
> > > areas.
> > > >
> > > > In my opinion, the only way to determine how long the in-
sample
> > > period
> > > > should be is to run tests, varying the length of the in-sample
> > > period and
> > > > observing the performance of the system on the following out-
of-
> > > sample
> > > > data. The "sweet spot" will depend on both the trading system
> > > logic and the
> > > > data series that it is processing.
> > > >
> > > > The data being processed is composed of one or more of the
> > following
> > > > components: long-term trend, long-term cycle, short term
cycle,
> > > pattern,
> > > > seasonality, and noise. There may be other components as well,
> > > just include
> > > > them in the list. Long-term and short-term are relative. The
> > > trading
> > > > system logic is written to identify some component (usually
just
> > > one of the
> > > > components) that precedes profitable trading opportunities,
hope
> > > that those
> > > > features persist beyond the in-sample period over which the
> > system
> > > is
> > > > developed, and can be profitably traded. The feature(s) being
> > > identified
> > > > are the signal portion of the data. Everything else is noise.
> > > Even if some
> > > > part of the "everything else" contains features that some
other
> > > trading
> > > > system can identify and profitably trade, it is noise to any
> > system
> > > being
> > > > tested that does not identify and remove it or compensate for
it.
> > > >
> > > > To shorten the explanation, the system is looking for the
signal
> > > among the
> > > > noise. The ease with which the signal can be identified
depends
> > > on both
> > > > the logic of the system and the characteristics of the data.
It
> > is
> > > not
> > > > possible to generalize without knowing both.
> > > >
> > > > Similarly, the only way to determine how long the out-of-
sample
> > > period
> > > > should be is to run tests. The systems we write are static.
> > They
> > > may have
> > > > logic that allows the parameters and the logic to adjust
> > > themselves, but the
> > > > system does not change. The characteristics of the data being
> > > processed are
> > > > dynamic. A trading system remains profitable only as long as
the
> > > system and
> > > > the data it models remain in synchronization. Clever systems
can
> > > (but not
> > > > always will) remain synchronized better than simple systems,
but
> > the
> > > > synchronization is required for the trades to be profitable.
> > The
> > > period of
> > > > time that the system remains profitable is the period of time
> > that
> > > the
> > > > system and the market remain in sync. That period determines
the
> > > schedule
> > > > for re-optimization (the maximum time between re-
optimizations)
> > and
> > > that is
> > > > the length of the out-of-sample period. There is no way to
> > > determine that
> > > > length without testing the specific system on the specific
data.
> > > The length
> > > > of out-of-sample profitability will not remain constant, but
will
> > > vary.
> > > > There is no relationship between the length of the in-sample
> > period
> > > and the
> > > > length of the out-of-sample period.
> > > >
> > > > The best we can hope for is a high level of confidence that
our
> > > newly
> > > > designed, newly optimized, newly re-optimized system performs
> > well
> > > in real
> > > > trading. There are no guarantees. The best way to gain
> > confidence
> > > is to
> > > > observe as many in-sample to out-of-sample transitions as
> > possible
> > > and learn
> > > > what to expect. The best way to do that is to run automated
walk
> > > forward
> > > > testing with fairly short out-of-sample periods. In an
automated
> > > walk
> > > > forward test, the length of the out-of-sample period is often
the
> > > same as
> > > > the re-optimization schedule.
> > > >
> > > > Once the system has passed the validation procedure, and the
> > > designer
> > > > understands what to expect in the period immediately
following the
> > > > re-optimization, re-optimization is permitted at any time.
There
> > > is no need
> > > > to wait the previously defined out-of-sample length time.
> > > >
> > > > Although you did not raise the question in your posting,
there is
> > > another
> > > > component of system design and testing that is critically
> > > important. My
> > > > opinion is that the whole process begins with the person or
> > > organization
> > > > that is going to trade the system defining the criteria by
which
> > the
> > > > acceptability of each system, or alternative system, is
judged.
> > > Choice of
> > > > this "objective function" is very personal, and it
incorporates
> > > most of
> > > > those features that the psychology of trading experts talk
about
> > > when they
> > > > help us learn to accept a trading system. With the correct
> > choice
> > > of the
> > > > objective function, every system that passes as acceptable is
> > > already one
> > > > that the trader will be comfortable with.
> > > >
> > > > And, importantly, it is the score on the objective function
that
> > > determines
> > > > which of the alternative systems will be chosen as "best" and
> > used
> > > to trade
> > > > forward in the out-of-sample period.
> > > >
> > > > Consequently, I recommend that the objective function be
chosen
> > > first. Then
> > > > the system designed, tested, and validated using the walk
forward
> > > process,
> > > > and letting the system and the data it is reading determine
how
> > > long the
> > > > in-sample period is and how long the out-of-sample period is.
> > > >
> > > > Thanks for listening,
> > > > Howard
> > > > www.quantitativetradingsystems.com
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Apr 9, 2008 at 10:08 AM, Howard B <howardbandy@>
wrote:
> > > >
> > > > > Hi Louis --
> > > > >
> > > > > I agree that there is a serious problem when the only data
that
> > is
> > > > > available contains no period that is similar to what is
> > expected
> > > in the
> > > > > future.
> > > > >
> > > > > Artificial data has no value.
> > > > >
> > > > > Using data that is earlier in time than the in-sample period
> > has
> > > limited
> > > > > value. You can test earlier data, but you will over-estimate
> > the
> > > > > performance that you can expect in the future.
> > > > >
> > > > > Are there other tickers that are closely related that have
data
> > > for the
> > > > > periods you would like to test?
> > > > >
> > > > > In the end, you will need to make a decision on whether to
> > place
> > > actual
> > > > > trades. And that decision must be based on your
understanding
> > of
> > > and
> > > > > confidence in your system. The only way to gain that
> > confidence
> > > is by
> > > > > observing the transitions from in-sample testing to out-of-
> > sample
> > > simulated
> > > > > trading.
> > > > >
> > > > > Thanks,
> > > > > Howard
> > > > >
> > > > > On Tue, Apr 8, 2008 at 10:37 PM, Mike <sfclimbers@> wrote:
> > > > >
> > > > > > Howard's comments are consistent with those of Robert
Pardo
> > > (The
> > > > > > Evaluation and Optimization of Trading Strategies, Wiley
> > 2008),
> > > with
> > > > > > respect to training periods.
> > > > > >
> > > > > > Pardo recognizes that there is a tradeoff between more
robust
> > > > > > strategies which require longer in sample training
periods,
> > > require
> > > > > > fewer reoptimizations, trade for longer out of sample
periods
> > > and are
> > > > > > generally less profitable, vs. more responsive strategies
> > which
> > > > > > require shorter in sample training periods, require more
> > > frequent
> > > > > > reoptimizations, can only trade for shorter out of sample
> > > periods and
> > > > > > are generally more profitable.
> > > > > >
> > > > > > Pardo suggests that strategies generating more frequent
> > signals
> > > can
> > > > > > use shorter in sample training windows since they
generate the
> > > > > > minimum 30+ trades sooner than strategies that generate
less
> > > frequent
> > > > > > signals. But, that in any case, one should try to use an
in
> > > sample
> > > > > > period sufficiently long to capture bull, bear, and
sideways
> > > markets.
> > > > > >
> > > > > > Further, when first trying to evaluate the worth of the
> > > strategy,
> > > > > > Pardo suggests backtesting the in sample history in
segments
> > > rather
> > > > > > than one shot (e.g. 10 year history divided into five 2
year
> > > > > > segments). This gives you better insight as to whether the
> > > results
> > > > > > are due to a single segment or are consistent accross
> > segments,
> > > and
> > > > > > provides insight to your eventual in sample/out of sample
> > > periods for
> > > > > > Walk Forward Optimization.
> > > > > >
> > > > > > Finally, Pardo suggests that regardless of whether a long
or
> > > short
> > > > > > training period is used, a rule of thumb for in sample vs.
> > out
> > > of
> > > > > > sample is for out of sample to be between 1/8 to 1/3 of
the
> > in
> > > sample
> > > > > > period (e.g. 24/8 = 3 and 24/3 = 8, so it would be "safe"
to
> > > trade
> > > > > > out of sample for 3 - 8 months based on a system
backtested
> > > over 24
> > > > > > months.
> > > > > >
> > > > > > Yet another good book covering the topic. I reccomend it.
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%

40yahoogroups.com><amibroker%

> > > 40yahoogroups.com>, "Howard
> > > > > > B" <howardbandy@> wrote:
> > > > > > >
> > > > > > > Hi Louis, and all --
> > > > > > >
> > > > > > > I know David Aronson, respect him, and like and
recommend
> > his
> > > book.
> > > > > > >
> > > > > > > My view is that the in-sample period should be as short
as
> > > > > > practical. My
> > > > > > > thought is that: the system we are testing / trading is
> > > trying to
> > > > > > recognize
> > > > > > > the signal from among the noise; and the signal
patterns are
> > > > > > changing over
> > > > > > > time. So the length of the in-sample period is a
tradeoff --
> >
> > > short
> > > > > > to be
> > > > > > > able to change as the characteristics of the underlying
> > market
> > > > > > change, but
> > > > > > > not so short that the system is over-fit to the noise
> > rather
> > > than
> > > > > > learns the
> > > > > > > signal.
> > > > > > >
> > > > > > > You can test this in AmiBroker. Have your system ready
to
> > buy
> > > and
> > > > > > sell. In
> > > > > > > the Automatic Analysis window, use Settings and set up
the
> > > Walk
> > > > > > Forward
> > > > > > > parameters. Try an in-sample period of 10 years, an out-
of-
> > > sample
> > > > > > period of
> > > > > > > 6 months or 1 year. Run Optimize > Walk Forward and
look at
> > > the in-
> > > > > > sample
> > > > > > > and out-of-sample equity curves. Shorten the length of
the
> > in-
> > > > > > sample period
> > > > > > > to 9, then 8, then 7, ... then 1 year, keeping the out-
of-
> > > sample
> > > > > > period
> > > > > > > unchanged. Depending on your system and the market it is
> > > trading,
> > > > > > you may
> > > > > > > find that there is a sweet spot in the length of the in-
> > sample
> > > > > > data. If so,
> > > > > > > that is the amount of data that allows your system to
> > > recognize the
> > > > > > signal
> > > > > > > without being overwhelmed by the noise.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Howard
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 8, 2008 at 8:56 AM, Louis Préfontaine
> > > <rockprog80@>
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I've been thinking a lot lately, and here is
something I
> > > would
> > > > > > like to
> > > > > > > > have your opinion on.
> > > > > > > >
> > > > > > > > I've been introduced to automated systems by a trend
> > > following
> > > > > > book which
> > > > > > > > related how some trend followers built their systems
in
> > the
> > > 70s
> > > > > > or 80s and
> > > > > > > > got rich with them, and how their system did not
really
> > > change
> > > > > > all this
> > > > > > > > time. They didn't change their system because they
say the
> > > > > > market does NOT
> > > > > > > > change. They looked at historic market data from the
> > 1800s
> > > and
> > > > > > the market
> > > > > > > > was as it is right now. So they say.
> > > > > > > >
> > > > > > > > On the other side, lately I have been introduced to
the
> > > concept of
> > > > > > > > ever-changing markets and have had a hard time trying
to
> > > build my
> > > > > > system.
> > > > > > > > Got a very promising start with a system getting
around
> > 15-
> > > 20%
> > > > > > average for
> > > > > > > > April 2007 to April 2008 (with little drawdown, which
> > mean
> > > that
> > > > > > with
> > > > > > > > leverage I can boost this a lot). In any variation
over
> > > > > > thousands of stocks
> > > > > > > > the results were nearly all positives. But then, I
tested
> > > that
> > > > > > same system
> > > > > > > > for the years 2000 to 2008, and that was
disappointing.
> > > Even more
> > > > > > > > disappointing from 2001 to 2003, another troubled
market
> > > like the
> > > > > > one we are
> > > > > > > > in right now.
> > > > > > > >
> > > > > > > > So here I am, wondering where to go from now.
Aronson's
> > > > > > excellent book
> > > > > > > > talk about the importance of having a very large
sample
> > of
> > > data.
> > > > > > But the
> > > > > > > > problem is: the larger the data, the more "historic"
it
> > > gets and
> > > > > > the less it
> > > > > > > > seems to work.
> > > > > > > >
> > > > > > > > Is my system not working, or did the markets really
> > change?
> > > Do I
> > > > > > need to
> > > > > > > > make it more robust (that is, it MUST make profit even
> > from
> > > 2001
> > > > > > to 2003),
> > > > > > > > or can I have complete faith in what happened in the
last
> > > year?
> > > > > > > >
> > > > > > > > All those questions... Would be nice to read what you
> > think
> > > > > > about this.
> > > > > > > >
> > > > > > > > Louis
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
>