|
PureBytes Links
Trading Reference Links
|
Howard,
I agree that reoptimization is valid, necessary even, at any time
after going live. Specifically; Any time measured performance values
stray significantly from what was expected.
I did not mean to imply, nor do I think that Pardo was suggesting,
that the system had to be traded for the duration of the calculated
out of sample (OOS) period before reoptimization could occur.
I also subscribe to your advice, of iterating over data periods, to
find the most suitable combination of in sample (IS) and OOS.
I take Pardo's 1/8 to 1/3 ratio rule of thumb at face value (i.e. a
suggested range in which to perform the above iterations). Faced with
so many possible combinations, we have to start somewhere. However,
if my tests are showing noticably improving results closer to a 1/3
ratio than a 1/8 ratio, I would not hesitate to additionally try 1/2,
3/4, etc.
You do raise a new question though;
Are you suggesting that you would be open to changing the duration of
your IS lookback window at subsequent reoptimizations? If so,
wouldn't that be changing the underlying nature of the strategy?
In other words, if the targeted signal is no longer sufficiently
prevalent using the original IS lookback span, might that not be a
red flag that the system as a whole is failing, as opposed to simply
no longer being in sync with the market?
Or, do you consider the IS lookback period to be optimizable at each
reoptimization in the same vein as any other parameter (e.g.
comparable to extending the lookback of a moving average parameter)?
Thanks,
Mike
--- In amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@xxx> wrote:
>
> Hi Mike --
>
> I have read both of Bob Pardo's books. Bob and I had a telephone
> conversation several years ago, and we have exchanged emails
recently. I
> agree with much of what he writes, but have different views in some
areas.
>
> In my opinion, the only way to determine how long the in-sample
period
> should be is to run tests, varying the length of the in-sample
period and
> observing the performance of the system on the following out-of-
sample
> data. The "sweet spot" will depend on both the trading system
logic and the
> data series that it is processing.
>
> The data being processed is composed of one or more of the following
> components: long-term trend, long-term cycle, short term cycle,
pattern,
> seasonality, and noise. There may be other components as well,
just include
> them in the list. Long-term and short-term are relative. The
trading
> system logic is written to identify some component (usually just
one of the
> components) that precedes profitable trading opportunities, hope
that those
> features persist beyond the in-sample period over which the system
is
> developed, and can be profitably traded. The feature(s) being
identified
> are the signal portion of the data. Everything else is noise.
Even if some
> part of the "everything else" contains features that some other
trading
> system can identify and profitably trade, it is noise to any system
being
> tested that does not identify and remove it or compensate for it.
>
> To shorten the explanation, the system is looking for the signal
among the
> noise. The ease with which the signal can be identified depends
on both
> the logic of the system and the characteristics of the data. It is
not
> possible to generalize without knowing both.
>
> Similarly, the only way to determine how long the out-of-sample
period
> should be is to run tests. The systems we write are static. They
may have
> logic that allows the parameters and the logic to adjust
themselves, but the
> system does not change. The characteristics of the data being
processed are
> dynamic. A trading system remains profitable only as long as the
system and
> the data it models remain in synchronization. Clever systems can
(but not
> always will) remain synchronized better than simple systems, but the
> synchronization is required for the trades to be profitable. The
period of
> time that the system remains profitable is the period of time that
the
> system and the market remain in sync. That period determines the
schedule
> for re-optimization (the maximum time between re-optimizations) and
that is
> the length of the out-of-sample period. There is no way to
determine that
> length without testing the specific system on the specific data.
The length
> of out-of-sample profitability will not remain constant, but will
vary.
> There is no relationship between the length of the in-sample period
and the
> length of the out-of-sample period.
>
> The best we can hope for is a high level of confidence that our
newly
> designed, newly optimized, newly re-optimized system performs well
in real
> trading. There are no guarantees. The best way to gain confidence
is to
> observe as many in-sample to out-of-sample transitions as possible
and learn
> what to expect. The best way to do that is to run automated walk
forward
> testing with fairly short out-of-sample periods. In an automated
walk
> forward test, the length of the out-of-sample period is often the
same as
> the re-optimization schedule.
>
> Once the system has passed the validation procedure, and the
designer
> understands what to expect in the period immediately following the
> re-optimization, re-optimization is permitted at any time. There
is no need
> to wait the previously defined out-of-sample length time.
>
> Although you did not raise the question in your posting, there is
another
> component of system design and testing that is critically
important. My
> opinion is that the whole process begins with the person or
organization
> that is going to trade the system defining the criteria by which the
> acceptability of each system, or alternative system, is judged.
Choice of
> this "objective function" is very personal, and it incorporates
most of
> those features that the psychology of trading experts talk about
when they
> help us learn to accept a trading system. With the correct choice
of the
> objective function, every system that passes as acceptable is
already one
> that the trader will be comfortable with.
>
> And, importantly, it is the score on the objective function that
determines
> which of the alternative systems will be chosen as "best" and used
to trade
> forward in the out-of-sample period.
>
> Consequently, I recommend that the objective function be chosen
first. Then
> the system designed, tested, and validated using the walk forward
process,
> and letting the system and the data it is reading determine how
long the
> in-sample period is and how long the out-of-sample period is.
>
> Thanks for listening,
> Howard
> www.quantitativetradingsystems.com
>
>
>
>
> On Wed, Apr 9, 2008 at 10:08 AM, Howard B <howardbandy@xxx> wrote:
>
> > Hi Louis --
> >
> > I agree that there is a serious problem when the only data that is
> > available contains no period that is similar to what is expected
in the
> > future.
> >
> > Artificial data has no value.
> >
> > Using data that is earlier in time than the in-sample period has
limited
> > value. You can test earlier data, but you will over-estimate the
> > performance that you can expect in the future.
> >
> > Are there other tickers that are closely related that have data
for the
> > periods you would like to test?
> >
> > In the end, you will need to make a decision on whether to place
actual
> > trades. And that decision must be based on your understanding of
and
> > confidence in your system. The only way to gain that confidence
is by
> > observing the transitions from in-sample testing to out-of-sample
simulated
> > trading.
> >
> > Thanks,
> > Howard
> >
> > On Tue, Apr 8, 2008 at 10:37 PM, Mike <sfclimbers@xxx> wrote:
> >
> > > Howard's comments are consistent with those of Robert Pardo
(The
> > > Evaluation and Optimization of Trading Strategies, Wiley 2008),
with
> > > respect to training periods.
> > >
> > > Pardo recognizes that there is a tradeoff between more robust
> > > strategies which require longer in sample training periods,
require
> > > fewer reoptimizations, trade for longer out of sample periods
and are
> > > generally less profitable, vs. more responsive strategies which
> > > require shorter in sample training periods, require more
frequent
> > > reoptimizations, can only trade for shorter out of sample
periods and
> > > are generally more profitable.
> > >
> > > Pardo suggests that strategies generating more frequent signals
can
> > > use shorter in sample training windows since they generate the
> > > minimum 30+ trades sooner than strategies that generate less
frequent
> > > signals. But, that in any case, one should try to use an in
sample
> > > period sufficiently long to capture bull, bear, and sideways
markets.
> > >
> > > Further, when first trying to evaluate the worth of the
strategy,
> > > Pardo suggests backtesting the in sample history in segments
rather
> > > than one shot (e.g. 10 year history divided into five 2 year
> > > segments). This gives you better insight as to whether the
results
> > > are due to a single segment or are consistent accross segments,
and
> > > provides insight to your eventual in sample/out of sample
periods for
> > > Walk Forward Optimization.
> > >
> > > Finally, Pardo suggests that regardless of whether a long or
short
> > > training period is used, a rule of thumb for in sample vs. out
of
> > > sample is for out of sample to be between 1/8 to 1/3 of the in
sample
> > > period (e.g. 24/8 = 3 and 24/3 = 8, so it would be "safe" to
trade
> > > out of sample for 3 - 8 months based on a system backtested
over 24
> > > months.
> > >
> > > Yet another good book covering the topic. I reccomend it.
> > >
> > > Mike
> > >
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com>, "Howard
> > > B" <howardbandy@> wrote:
> > > >
> > > > Hi Louis, and all --
> > > >
> > > > I know David Aronson, respect him, and like and recommend his
book.
> > > >
> > > > My view is that the in-sample period should be as short as
> > > practical. My
> > > > thought is that: the system we are testing / trading is
trying to
> > > recognize
> > > > the signal from among the noise; and the signal patterns are
> > > changing over
> > > > time. So the length of the in-sample period is a tradeoff --
short
> > > to be
> > > > able to change as the characteristics of the underlying market
> > > change, but
> > > > not so short that the system is over-fit to the noise rather
than
> > > learns the
> > > > signal.
> > > >
> > > > You can test this in AmiBroker. Have your system ready to buy
and
> > > sell. In
> > > > the Automatic Analysis window, use Settings and set up the
Walk
> > > Forward
> > > > parameters. Try an in-sample period of 10 years, an out-of-
sample
> > > period of
> > > > 6 months or 1 year. Run Optimize > Walk Forward and look at
the in-
> > > sample
> > > > and out-of-sample equity curves. Shorten the length of the in-
> > > sample period
> > > > to 9, then 8, then 7, ... then 1 year, keeping the out-of-
sample
> > > period
> > > > unchanged. Depending on your system and the market it is
trading,
> > > you may
> > > > find that there is a sweet spot in the length of the in-sample
> > > data. If so,
> > > > that is the amount of data that allows your system to
recognize the
> > > signal
> > > > without being overwhelmed by the noise.
> > > >
> > > > Thanks,
> > > > Howard
> > > >
> > > >
> > > > On Tue, Apr 8, 2008 at 8:56 AM, Louis Préfontaine
<rockprog80@>
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I've been thinking a lot lately, and here is something I
would
> > > like to
> > > > > have your opinion on.
> > > > >
> > > > > I've been introduced to automated systems by a trend
following
> > > book which
> > > > > related how some trend followers built their systems in the
70s
> > > or 80s and
> > > > > got rich with them, and how their system did not really
change
> > > all this
> > > > > time. They didn't change their system because they say the
> > > market does NOT
> > > > > change. They looked at historic market data from the 1800s
and
> > > the market
> > > > > was as it is right now. So they say.
> > > > >
> > > > > On the other side, lately I have been introduced to the
concept of
> > > > > ever-changing markets and have had a hard time trying to
build my
> > > system.
> > > > > Got a very promising start with a system getting around 15-
20%
> > > average for
> > > > > April 2007 to April 2008 (with little drawdown, which mean
that
> > > with
> > > > > leverage I can boost this a lot). In any variation over
> > > thousands of stocks
> > > > > the results were nearly all positives. But then, I tested
that
> > > same system
> > > > > for the years 2000 to 2008, and that was disappointing.
Even more
> > > > > disappointing from 2001 to 2003, another troubled market
like the
> > > one we are
> > > > > in right now.
> > > > >
> > > > > So here I am, wondering where to go from now. Aronson's
> > > excellent book
> > > > > talk about the importance of having a very large sample of
data.
> > > But the
> > > > > problem is: the larger the data, the more "historic" it
gets and
> > > the less it
> > > > > seems to work.
> > > > >
> > > > > Is my system not working, or did the markets really change?
Do I
> > > need to
> > > > > make it more robust (that is, it MUST make profit even from
2001
> > > to 2003),
> > > > > or can I have complete faith in what happened in the last
year?
> > > > >
> > > > > All those questions... Would be nice to read what you think
> > > about this.
> > > > >
> > > > > Louis
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> >
> >
>
------------------------------------
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|