[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] Re: Another tough question...



PureBytes Links

Trading Reference Links

Hi Mike --

On having the out-of-sample data period earlier than the in-sample data period ---

It is my contention that a successful (profitable) trading system is successful because it identifies inefficiencies in the market it is trading;  and every profitable trade removes a portion of those inefficiencies.  So assume I discover a trading system in the year 2000 and begin to trade it.  As this system becomes more widely used and makes profitable trades, it will become less profitable since there is less inefficiency remaining to remove. 

When I perform the walk forward testing, each step simulates the process I will go through when I eventually move the system from development to production.  Optimize over some in-sample period, then trade for some length of time.  Repeating the optimize -- trade cycle as necessary.  In no case can I ever perform the system development over one period of time and then trade an earlier period.  That would be trading yesterday's chart. 

So I contend that the OOS period must be later than the IS period for two reasons.
1.  The process of walking forward is intended to give us confidence for the process we will eventually go through when trading.  Since trading is done on data that is more recent than development, the validation process should use data that is in that same time relationship.
2.  Using the example above of a system developed in 2000.  Beginning in 2001, the fact that that system exists makes it more difficult to be profitable in the future -- some of the inefficiency has been removed from the data.  But the historical data for 1999 covers a period when that system did not exist and the full inefficiency is in the data.  Using an earlier period to test will over-estimate the likely future performance.  The only way to determine the magnitude of the over-estimate is to test over a data period that is more recent than the development period. 

Continuing on. 

Longer IS periods do give a trading system more data to learn.  Whether that is an advantage or a disadvantage depends on the system and the data.  If there is some fundamental reason the system is profitable, and that reason remains valid through out the period, then more data gives better learning of the signal portion of the data -- the signal to noise ratio improves.  But, as soon as the characteristics of the data change, using more data creates a problem, since there are now multiple conditions to learn.  If the system code is clever, it may be able to learn more than one thing.  If it is not clever, then it will still try to learn, but it will learn one condition or another or a compromise; and being profitable in the future will be due to luck as much as to proper system design.

For example -- a very simplistic example, but it will illustrate my point. 

Assume the data contains a 20 day cycle for five years, after which the period changes to 10 days for five more years.  If a system is asked to learn the cycle and the IS period contains only 20 day cycles, the system will synchronize itself to the 20 days and be profitable as long as the cycle remains at 20 days.  During the transition from 20 to 10 days, the system will not be profitable.  Then during the 10 day section it will again be in sync and profitable. 

On the other hand, is the system is given all ten years of data and asked to identify the cycle length, it will be inaccurate for both the 20 day portion and the 10 day portion -- it cannot learn the length because the length changes within the IS period.

My recommendation is that the in-sample period should be as short as is practical.  Long enough to allow the system to learn the current characteristics of the data, and long enough that the system learns the signal rather than the noise.  And short enough to allow the system to learn the current conditions as well as possible.  There is no way to estimate in advance what that ideal length should be -- it depends on both the system and the data.

And you are correct, there are a couple of areas where I disagree with Bob Pardo.

Thanks for listening,
Howard
 



On Sat, Apr 12, 2008 at 11:45 PM, Mike <sfclimbers@xxxxxxxxx> wrote:

> I did not mean to imply, nor do I think that Pardo was suggesting,
> that the system had to be traded for the duration of the calculated
> out of sample (OOS) period before reoptimization could occur.

I stand corrected. The chapter I had been summarizing(Chapter 6) did
not express that the OOS period should be fixed. However, upon review
of Chapter 11, Pardo does say that the OOS period should remain
constant and traded to completion before reoptimization.

If you are still reading Howard, I wonder if you would care to
comment on another point?

In Chapter 18 of your book, "In Sample Out of Sample", you have
stated that using an out of sample period that predates the in sample
period will have no predictive value towards the future. You go on to
say that a "jackknife" approach over 10 periods really only has 1 out
of sample period rather than 10.

I'm trying to compare this to Pardo's suggestion of starting with
reasonably recent data for initial testing. Then doing walk forward
over at least 10 - 20 years of history in an attempt to get as many
OOS periods as possible for summary analysis.

Is your assertion regarding jackknife testing a difference of opinion
with Pardo's practice of starting a walk forward earlier in time than
the time period upon which the strategy was first devised?

Or, is the nature of walk forward sufficiently different from
jackknife in that all walk forward IS periods are prior to each OOS
period, whereas jackknife would have a mix of prior and later IS
periods when calculating most of the OOS periods?

Thanks.


--- In amibroker@xxxxxxxxxxxxxxx, "Mike" <sfclimbers@xxx> wrote:
>
> Howard,
>
> I agree that reoptimization is valid, necessary even, at any time
> after going live. Specifically; Any time measured performance
values
> stray significantly from what was expected.
>
> I did not mean to imply, nor do I think that Pardo was suggesting,
> that the system had to be traded for the duration of the calculated
> out of sample (OOS) period before reoptimization could occur.
>
> I also subscribe to your advice, of iterating over data periods, to
> find the most suitable combination of in sample (IS) and OOS.
>
> I take Pardo's 1/8 to 1/3 ratio rule of thumb at face value (i.e. a
> suggested range in which to perform the above iterations). Faced
with
> so many possible combinations, we have to start somewhere. However,
> if my tests are showing noticably improving results closer to a 1/3
> ratio than a 1/8 ratio, I would not hesitate to additionally try
1/2,
> 3/4, etc.
>
> You do raise a new question though;
>
> Are you suggesting that you would be open to changing the duration
of
> your IS lookback window at subsequent reoptimizations? If so,
> wouldn't that be changing the underlying nature of the strategy?
>
> In other words, if the targeted signal is no longer sufficiently
> prevalent using the original IS lookback span, might that not be a
> red flag that the system as a whole is failing, as opposed to
simply
> no longer being in sync with the market?
>
> Or, do you consider the IS lookback period to be optimizable at
each
> reoptimization in the same vein as any other parameter (e.g.
> comparable to extending the lookback of a moving average parameter)?
>
> Thanks,
>
> Mike
>
> --- In amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@> wrote:
> >
> > Hi Mike --
> >
> > I have read both of Bob Pardo's books. Bob and I had a telephone
> > conversation several years ago, and we have exchanged emails
> recently. I
> > agree with much of what he writes, but have different views in
some
> areas.
> >
> > In my opinion, the only way to determine how long the in-sample
> period
> > should be is to run tests, varying the length of the in-sample
> period and
> > observing the performance of the system on the following out-of-
> sample
> > data. The "sweet spot" will depend on both the trading system
> logic and the
> > data series that it is processing.
> >
> > The data being processed is composed of one or more of the
following
> > components: long-term trend, long-term cycle, short term cycle,
> pattern,
> > seasonality, and noise. There may be other components as well,
> just include
> > them in the list. Long-term and short-term are relative. The
> trading
> > system logic is written to identify some component (usually just
> one of the
> > components) that precedes profitable trading opportunities, hope
> that those
> > features persist beyond the in-sample period over which the
system
> is
> > developed, and can be profitably traded. The feature(s) being
> identified
> > are the signal portion of the data. Everything else is noise.
> Even if some
> > part of the "everything else" contains features that some other
> trading
> > system can identify and profitably trade, it is noise to any
system
> being
> > tested that does not identify and remove it or compensate for it.
> >
> > To shorten the explanation, the system is looking for the signal
> among the
> > noise. The ease with which the signal can be identified depends
> on both
> > the logic of the system and the characteristics of the data. It
is
> not
> > possible to generalize without knowing both.
> >
> > Similarly, the only way to determine how long the out-of-sample
> period
> > should be is to run tests. The systems we write are static.
They
> may have
> > logic that allows the parameters and the logic to adjust
> themselves, but the
> > system does not change. The characteristics of the data being
> processed are
> > dynamic. A trading system remains profitable only as long as the
> system and
> > the data it models remain in synchronization. Clever systems can
> (but not
> > always will) remain synchronized better than simple systems, but
the
> > synchronization is required for the trades to be profitable.
The
> period of
> > time that the system remains profitable is the period of time
that
> the
> > system and the market remain in sync. That period determines the
> schedule
> > for re-optimization (the maximum time between re-optimizations)
and
> that is
> > the length of the out-of-sample period. There is no way to
> determine that
> > length without testing the specific system on the specific data.
> The length
> > of out-of-sample profitability will not remain constant, but will
> vary.
> > There is no relationship between the length of the in-sample
period
> and the
> > length of the out-of-sample period.
> >
> > The best we can hope for is a high level of confidence that our
> newly
> > designed, newly optimized, newly re-optimized system performs
well
> in real
> > trading. There are no guarantees. The best way to gain
confidence
> is to
> > observe as many in-sample to out-of-sample transitions as
possible
> and learn
> > what to expect. The best way to do that is to run automated walk
> forward
> > testing with fairly short out-of-sample periods. In an automated
> walk
> > forward test, the length of the out-of-sample period is often the
> same as
> > the re-optimization schedule.
> >
> > Once the system has passed the validation procedure, and the
> designer
> > understands what to expect in the period immediately following the
> > re-optimization, re-optimization is permitted at any time. There
> is no need
> > to wait the previously defined out-of-sample length time.
> >
> > Although you did not raise the question in your posting, there is
> another
> > component of system design and testing that is critically
> important. My
> > opinion is that the whole process begins with the person or
> organization
> > that is going to trade the system defining the criteria by which
the
> > acceptability of each system, or alternative system, is judged.
> Choice of
> > this "objective function" is very personal, and it incorporates
> most of
> > those features that the psychology of trading experts talk about
> when they
> > help us learn to accept a trading system. With the correct
choice
> of the
> > objective function, every system that passes as acceptable is
> already one
> > that the trader will be comfortable with.
> >
> > And, importantly, it is the score on the objective function that
> determines
> > which of the alternative systems will be chosen as "best" and
used
> to trade
> > forward in the out-of-sample period.
> >
> > Consequently, I recommend that the objective function be chosen
> first. Then
> > the system designed, tested, and validated using the walk forward
> process,
> > and letting the system and the data it is reading determine how
> long the
> > in-sample period is and how long the out-of-sample period is.
> >
> > Thanks for listening,
> > Howard
> > www.quantitativetradingsystems.com
> >
> >
> >
> >
> > On Wed, Apr 9, 2008 at 10:08 AM, Howard B <howardbandy@> wrote:
> >
> > > Hi Louis --
> > >
> > > I agree that there is a serious problem when the only data that
is
> > > available contains no period that is similar to what is
expected
> in the
> > > future.
> > >
> > > Artificial data has no value.
> > >
> > > Using data that is earlier in time than the in-sample period
has
> limited
> > > value. You can test earlier data, but you will over-estimate
the
> > > performance that you can expect in the future.
> > >
> > > Are there other tickers that are closely related that have data
> for the
> > > periods you would like to test?
> > >
> > > In the end, you will need to make a decision on whether to
place
> actual
> > > trades. And that decision must be based on your understanding
of
> and
> > > confidence in your system. The only way to gain that
confidence
> is by
> > > observing the transitions from in-sample testing to out-of-
sample
> simulated
> > > trading.
> > >
> > > Thanks,
> > > Howard
> > >
> > > On Tue, Apr 8, 2008 at 10:37 PM, Mike <sfclimbers@> wrote:
> > >
> > > > Howard's comments are consistent with those of Robert Pardo
> (The
> > > > Evaluation and Optimization of Trading Strategies, Wiley
2008),
> with
> > > > respect to training periods.
> > > >
> > > > Pardo recognizes that there is a tradeoff between more robust
> > > > strategies which require longer in sample training periods,
> require
> > > > fewer reoptimizations, trade for longer out of sample periods
> and are
> > > > generally less profitable, vs. more responsive strategies
which
> > > > require shorter in sample training periods, require more
> frequent
> > > > reoptimizations, can only trade for shorter out of sample
> periods and
> > > > are generally more profitable.
> > > >
> > > > Pardo suggests that strategies generating more frequent
signals
> can
> > > > use shorter in sample training windows since they generate the
> > > > minimum 30+ trades sooner than strategies that generate less
> frequent
> > > > signals. But, that in any case, one should try to use an in
> sample
> > > > period sufficiently long to capture bull, bear, and sideways
> markets.
> > > >
> > > > Further, when first trying to evaluate the worth of the
> strategy,
> > > > Pardo suggests backtesting the in sample history in segments
> rather
> > > > than one shot (e.g. 10 year history divided into five 2 year
> > > > segments). This gives you better insight as to whether the
> results
> > > > are due to a single segment or are consistent accross
segments,
> and
> > > > provides insight to your eventual in sample/out of sample
> periods for
> > > > Walk Forward Optimization.
> > > >
> > > > Finally, Pardo suggests that regardless of whether a long or
> short
> > > > training period is used, a rule of thumb for in sample vs.
out
> of
> > > > sample is for out of sample to be between 1/8 to 1/3 of the
in
> sample
> > > > period (e.g. 24/8 = 3 and 24/3 = 8, so it would be "safe" to
> trade
> > > > out of sample for 3 - 8 months based on a system backtested
> over 24
> > > > months.
> > > >
> > > > Yet another good book covering the topic. I reccomend it.
> > > >
> > > > Mike
> > > >
> > > >
> > > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
> 40yahoogroups.com>, "Howard
> > > > B" <howardbandy@> wrote:
> > > > >
> > > > > Hi Louis, and all --
> > > > >
> > > > > I know David Aronson, respect him, and like and recommend
his
> book.
> > > > >
> > > > > My view is that the in-sample period should be as short as
> > > > practical. My
> > > > > thought is that: the system we are testing / trading is
> trying to
> > > > recognize
> > > > > the signal from among the noise; and the signal patterns are
> > > > changing over
> > > > > time. So the length of the in-sample period is a tradeoff --

> short
> > > > to be
> > > > > able to change as the characteristics of the underlying
market
> > > > change, but
> > > > > not so short that the system is over-fit to the noise
rather
> than
> > > > learns the
> > > > > signal.
> > > > >
> > > > > You can test this in AmiBroker. Have your system ready to
buy
> and
> > > > sell. In
> > > > > the Automatic Analysis window, use Settings and set up the
> Walk
> > > > Forward
> > > > > parameters. Try an in-sample period of 10 years, an out-of-
> sample
> > > > period of
> > > > > 6 months or 1 year. Run Optimize > Walk Forward and look at
> the in-
> > > > sample
> > > > > and out-of-sample equity curves. Shorten the length of the
in-
> > > > sample period
> > > > > to 9, then 8, then 7, ... then 1 year, keeping the out-of-
> sample
> > > > period
> > > > > unchanged. Depending on your system and the market it is
> trading,
> > > > you may
> > > > > find that there is a sweet spot in the length of the in-
sample
> > > > data. If so,
> > > > > that is the amount of data that allows your system to
> recognize the
> > > > signal
> > > > > without being overwhelmed by the noise.
> > > > >
> > > > > Thanks,
> > > > > Howard
> > > > >
> > > > >
> > > > > On Tue, Apr 8, 2008 at 8:56 AM, Louis Préfontaine
> <rockprog80@>
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I've been thinking a lot lately, and here is something I
> would
> > > > like to
> > > > > > have your opinion on.
> > > > > >
> > > > > > I've been introduced to automated systems by a trend
> following
> > > > book which
> > > > > > related how some trend followers built their systems in
the
> 70s
> > > > or 80s and
> > > > > > got rich with them, and how their system did not really
> change
> > > > all this
> > > > > > time. They didn't change their system because they say the
> > > > market does NOT
> > > > > > change. They looked at historic market data from the
1800s
> and
> > > > the market
> > > > > > was as it is right now. So they say.
> > > > > >
> > > > > > On the other side, lately I have been introduced to the
> concept of
> > > > > > ever-changing markets and have had a hard time trying to
> build my
> > > > system.
> > > > > > Got a very promising start with a system getting around
15-
> 20%
> > > > average for
> > > > > > April 2007 to April 2008 (with little drawdown, which
mean
> that
> > > > with
> > > > > > leverage I can boost this a lot). In any variation over
> > > > thousands of stocks
> > > > > > the results were nearly all positives. But then, I tested
> that
> > > > same system
> > > > > > for the years 2000 to 2008, and that was disappointing.
> Even more
> > > > > > disappointing from 2001 to 2003, another troubled market
> like the
> > > > one we are
> > > > > > in right now.
> > > > > >
> > > > > > So here I am, wondering where to go from now. Aronson's
> > > > excellent book
> > > > > > talk about the importance of having a very large sample
of
> data.
> > > > But the
> > > > > > problem is: the larger the data, the more "historic" it
> gets and
> > > > the less it
> > > > > > seems to work.
> > > > > >
> > > > > > Is my system not working, or did the markets really
change?
> Do I
> > > > need to
> > > > > > make it more robust (that is, it MUST make profit even
from
> 2001
> > > > to 2003),
> > > > > > or can I have complete faith in what happened in the last
> year?
> > > > > >
> > > > > > All those questions... Would be nice to read what you
think
> > > > about this.
> > > > > >
> > > > > > Louis
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
>


__._,_.___

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html




Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___