|
PureBytes Links
Trading Reference Links
|
I think you would agree that the points of difference are often more
important than the points of agreement (that is where we learn the
most).
I am taking notice of our points of difference and thinking about
them.
> I'll not advise anyone to use artificial data for anything -- I
>continue to
> assert that artificial data has no value whatsoever for training a
>trading
> system.
What I have used if for is to train me (mathematically), rather than
a system e.g. knowing what the mean is in advance allowed me to
compare predicted metrics (using stats equations) to observed metrics
(as produced by the RandomGenerator).
I only used it to study break-even systems (black swans), and the
variance they were capable of producing.
Since a RG will produce a pretty fair 'random walk (an assumption on
my part!) I thought that was fair enough.
Keep in mind that as a naive mathematician and self-confessed
intuitive I have to find ways to take my maths medicine that works
for me.
I definitely would not use it to train trading systems - my trading
philosophy is based on the view that participants behaviour skews the
market, which provides us with our opportunities and you can't
replicate that in synthetic data.
Anyway, I think enough of my RG excel sheets to post an example in
the future so hopefully you will be available to make some specific
criticisms then - I would welcome that.
Thanks for your answers.
brian_z
--- In amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@xxx> wrote:
>
> Hi Brian --
>
> I'll not advise anyone to use artificial data for anything -- I
continue to
> assert that artificial data has no value whatsoever for training a
trading
> system. If I had the ability to design an artificial data series
that
> represented the real data so closely that it had value as training
data, I
> would have no need for it. I would simply code those rules into my
trading
> system.
>
> Thanks,
> Howard
>
>
> On Wed, Apr 9, 2008 at 5:41 PM, Howard B <howardbandy@xxx> wrote:
>
> > Hi Mike --
> >
> > I suggest that the in-sample period be selected, then fixed for
the
> > duration of the use of the trading system. It is the length of
time between
> > re-optimizations that can be any length you wish.
> >
> > Thanks,
> > Howard
> >
> >
> >
> > On Wed, Apr 9, 2008 at 5:07 PM, Mike <sfclimbers@xxx> wrote:
> >
> > > Howard,
> > >
> > > I agree that reoptimization is valid, necessary even, at any
time
> > > after going live. Specifically; Any time measured performance
values
> > > stray significantly from what was expected.
> > >
> > > I did not mean to imply, nor do I think that Pardo was
suggesting,
> > > that the system had to be traded for the duration of the
calculated
> > > out of sample (OOS) period before reoptimization could occur.
> > >
> > > I also subscribe to your advice, of iterating over data
periods, to
> > > find the most suitable combination of in sample (IS) and OOS.
> > >
> > > I take Pardo's 1/8 to 1/3 ratio rule of thumb at face value
(i.e. a
> > > suggested range in which to perform the above iterations).
Faced with
> > > so many possible combinations, we have to start somewhere.
However,
> > > if my tests are showing noticably improving results closer to a
1/3
> > > ratio than a 1/8 ratio, I would not hesitate to additionally
try 1/2,
> > > 3/4, etc.
> > >
> > > You do raise a new question though;
> > >
> > > Are you suggesting that you would be open to changing the
duration of
> > > your IS lookback window at subsequent reoptimizations? If so,
> > > wouldn't that be changing the underlying nature of the strategy?
> > >
> > > In other words, if the targeted signal is no longer sufficiently
> > > prevalent using the original IS lookback span, might that not
be a
> > > red flag that the system as a whole is failing, as opposed to
simply
> > > no longer being in sync with the market?
> > >
> > > Or, do you consider the IS lookback period to be optimizable at
each
> > > reoptimization in the same vein as any other parameter (e.g.
> > > comparable to extending the lookback of a moving average
parameter)?
> > >
> > > Thanks,
> > >
> > >
> > > Mike
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com>, "Howard
> > > B" <howardbandy@> wrote:
> > > >
> > > > Hi Mike --
> > > >
> > > > I have read both of Bob Pardo's books. Bob and I had a
telephone
> > > > conversation several years ago, and we have exchanged emails
> > > recently. I
> > > > agree with much of what he writes, but have different views
in some
> > > areas.
> > > >
> > > > In my opinion, the only way to determine how long the in-
sample
> > > period
> > > > should be is to run tests, varying the length of the in-sample
> > > period and
> > > > observing the performance of the system on the following out-
of-
> > > sample
> > > > data. The "sweet spot" will depend on both the trading system
> > > logic and the
> > > > data series that it is processing.
> > > >
> > > > The data being processed is composed of one or more of the
following
> > > > components: long-term trend, long-term cycle, short term
cycle,
> > > pattern,
> > > > seasonality, and noise. There may be other components as well,
> > > just include
> > > > them in the list. Long-term and short-term are relative. The
> > > trading
> > > > system logic is written to identify some component (usually
just
> > > one of the
> > > > components) that precedes profitable trading opportunities,
hope
> > > that those
> > > > features persist beyond the in-sample period over which the
system
> > > is
> > > > developed, and can be profitably traded. The feature(s) being
> > > identified
> > > > are the signal portion of the data. Everything else is noise.
> > > Even if some
> > > > part of the "everything else" contains features that some
other
> > > trading
> > > > system can identify and profitably trade, it is noise to any
system
> > > being
> > > > tested that does not identify and remove it or compensate for
it.
> > > >
> > > > To shorten the explanation, the system is looking for the
signal
> > > among the
> > > > noise. The ease with which the signal can be identified
depends
> > > on both
> > > > the logic of the system and the characteristics of the data.
It is
> > > not
> > > > possible to generalize without knowing both.
> > > >
> > > > Similarly, the only way to determine how long the out-of-
sample
> > > period
> > > > should be is to run tests. The systems we write are static.
They
> > > may have
> > > > logic that allows the parameters and the logic to adjust
> > > themselves, but the
> > > > system does not change. The characteristics of the data being
> > > processed are
> > > > dynamic. A trading system remains profitable only as long as
the
> > > system and
> > > > the data it models remain in synchronization. Clever systems
can
> > > (but not
> > > > always will) remain synchronized better than simple systems,
but the
> > > > synchronization is required for the trades to be profitable.
The
> > > period of
> > > > time that the system remains profitable is the period of time
that
> > > the
> > > > system and the market remain in sync. That period determines
the
> > > schedule
> > > > for re-optimization (the maximum time between re-
optimizations) and
> > > that is
> > > > the length of the out-of-sample period. There is no way to
> > > determine that
> > > > length without testing the specific system on the specific
data.
> > > The length
> > > > of out-of-sample profitability will not remain constant, but
will
> > > vary.
> > > > There is no relationship between the length of the in-sample
period
> > > and the
> > > > length of the out-of-sample period.
> > > >
> > > > The best we can hope for is a high level of confidence that
our
> > > newly
> > > > designed, newly optimized, newly re-optimized system performs
well
> > > in real
> > > > trading. There are no guarantees. The best way to gain
confidence
> > > is to
> > > > observe as many in-sample to out-of-sample transitions as
possible
> > > and learn
> > > > what to expect. The best way to do that is to run automated
walk
> > > forward
> > > > testing with fairly short out-of-sample periods. In an
automated
> > > walk
> > > > forward test, the length of the out-of-sample period is often
the
> > > same as
> > > > the re-optimization schedule.
> > > >
> > > > Once the system has passed the validation procedure, and the
> > > designer
> > > > understands what to expect in the period immediately
following the
> > > > re-optimization, re-optimization is permitted at any time.
There
> > > is no need
> > > > to wait the previously defined out-of-sample length time.
> > > >
> > > > Although you did not raise the question in your posting,
there is
> > > another
> > > > component of system design and testing that is critically
> > > important. My
> > > > opinion is that the whole process begins with the person or
> > > organization
> > > > that is going to trade the system defining the criteria by
which the
> > > > acceptability of each system, or alternative system, is
judged.
> > > Choice of
> > > > this "objective function" is very personal, and it
incorporates
> > > most of
> > > > those features that the psychology of trading experts talk
about
> > > when they
> > > > help us learn to accept a trading system. With the correct
choice
> > > of the
> > > > objective function, every system that passes as acceptable is
> > > already one
> > > > that the trader will be comfortable with.
> > > >
> > > > And, importantly, it is the score on the objective function
that
> > > determines
> > > > which of the alternative systems will be chosen as "best" and
used
> > > to trade
> > > > forward in the out-of-sample period.
> > > >
> > > > Consequently, I recommend that the objective function be
chosen
> > > first. Then
> > > > the system designed, tested, and validated using the walk
forward
> > > process,
> > > > and letting the system and the data it is reading determine
how
> > > long the
> > > > in-sample period is and how long the out-of-sample period is.
> > > >
> > > > Thanks for listening,
> > > > Howard
> > > > www.quantitativetradingsystems.com
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Apr 9, 2008 at 10:08 AM, Howard B <howardbandy@>
wrote:
> > > >
> > > > > Hi Louis --
> > > > >
> > > > > I agree that there is a serious problem when the only data
that is
> > > > > available contains no period that is similar to what is
expected
> > > in the
> > > > > future.
> > > > >
> > > > > Artificial data has no value.
> > > > >
> > > > > Using data that is earlier in time than the in-sample
period has
> > > limited
> > > > > value. You can test earlier data, but you will over-
estimate the
> > > > > performance that you can expect in the future.
> > > > >
> > > > > Are there other tickers that are closely related that have
data
> > > for the
> > > > > periods you would like to test?
> > > > >
> > > > > In the end, you will need to make a decision on whether to
place
> > > actual
> > > > > trades. And that decision must be based on your
understanding of
> > > and
> > > > > confidence in your system. The only way to gain that
confidence
> > > is by
> > > > > observing the transitions from in-sample testing to out-of-
sample
> > > simulated
> > > > > trading.
> > > > >
> > > > > Thanks,
> > > > > Howard
> > > > >
> > > > > On Tue, Apr 8, 2008 at 10:37 PM, Mike <sfclimbers@> wrote:
> > > > >
> > > > > > Howard's comments are consistent with those of Robert
Pardo
> > > (The
> > > > > > Evaluation and Optimization of Trading Strategies, Wiley
2008),
> > > with
> > > > > > respect to training periods.
> > > > > >
> > > > > > Pardo recognizes that there is a tradeoff between more
robust
> > > > > > strategies which require longer in sample training
periods,
> > > require
> > > > > > fewer reoptimizations, trade for longer out of sample
periods
> > > and are
> > > > > > generally less profitable, vs. more responsive strategies
which
> > > > > > require shorter in sample training periods, require more
> > > frequent
> > > > > > reoptimizations, can only trade for shorter out of sample
> > > periods and
> > > > > > are generally more profitable.
> > > > > >
> > > > > > Pardo suggests that strategies generating more frequent
signals
> > > can
> > > > > > use shorter in sample training windows since they
generate the
> > > > > > minimum 30+ trades sooner than strategies that generate
less
> > > frequent
> > > > > > signals. But, that in any case, one should try to use an
in
> > > sample
> > > > > > period sufficiently long to capture bull, bear, and
sideways
> > > markets.
> > > > > >
> > > > > > Further, when first trying to evaluate the worth of the
> > > strategy,
> > > > > > Pardo suggests backtesting the in sample history in
segments
> > > rather
> > > > > > than one shot (e.g. 10 year history divided into five 2
year
> > > > > > segments). This gives you better insight as to whether the
> > > results
> > > > > > are due to a single segment or are consistent accross
segments,
> > > and
> > > > > > provides insight to your eventual in sample/out of sample
> > > periods for
> > > > > > Walk Forward Optimization.
> > > > > >
> > > > > > Finally, Pardo suggests that regardless of whether a long
or
> > > short
> > > > > > training period is used, a rule of thumb for in sample
vs. out
> > > of
> > > > > > sample is for out of sample to be between 1/8 to 1/3 of
the in
> > > sample
> > > > > > period (e.g. 24/8 = 3 and 24/3 = 8, so it would be "safe"
to
> > > trade
> > > > > > out of sample for 3 - 8 months based on a system
backtested
> > > over 24
> > > > > > months.
> > > > > >
> > > > > > Yet another good book covering the topic. I reccomend it.
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com><amibroker%
> > > 40yahoogroups.com>, "Howard
> > >
> > > > > > B" <howardbandy@> wrote:
> > > > > > >
> > > > > > > Hi Louis, and all --
> > > > > > >
> > > > > > > I know David Aronson, respect him, and like and
recommend his
> > > book.
> > > > > > >
> > > > > > > My view is that the in-sample period should be as short
as
> > > > > > practical. My
> > > > > > > thought is that: the system we are testing / trading is
> > > trying to
> > > > > > recognize
> > > > > > > the signal from among the noise; and the signal
patterns are
> > > > > > changing over
> > > > > > > time. So the length of the in-sample period is a
tradeoff --
> > > short
> > > > > > to be
> > > > > > > able to change as the characteristics of the underlying
market
> > > > > > change, but
> > > > > > > not so short that the system is over-fit to the noise
rather
> > > than
> > > > > > learns the
> > > > > > > signal.
> > > > > > >
> > > > > > > You can test this in AmiBroker. Have your system ready
to buy
> > > and
> > > > > > sell. In
> > > > > > > the Automatic Analysis window, use Settings and set up
the
> > > Walk
> > > > > > Forward
> > > > > > > parameters. Try an in-sample period of 10 years, an out-
of-
> > > sample
> > > > > > period of
> > > > > > > 6 months or 1 year. Run Optimize > Walk Forward and
look at
> > > the in-
> > > > > > sample
> > > > > > > and out-of-sample equity curves. Shorten the length of
the in-
> > > > > > sample period
> > > > > > > to 9, then 8, then 7, ... then 1 year, keeping the out-
of-
> > > sample
> > > > > > period
> > > > > > > unchanged. Depending on your system and the market it is
> > > trading,
> > > > > > you may
> > > > > > > find that there is a sweet spot in the length of the in-
sample
> > > > > > data. If so,
> > > > > > > that is the amount of data that allows your system to
> > > recognize the
> > > > > > signal
> > > > > > > without being overwhelmed by the noise.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Howard
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 8, 2008 at 8:56 AM, Louis Préfontaine
> > > <rockprog80@>
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I've been thinking a lot lately, and here is
something I
> > > would
> > > > > > like to
> > > > > > > > have your opinion on.
> > > > > > > >
> > > > > > > > I've been introduced to automated systems by a trend
> > > following
> > > > > > book which
> > > > > > > > related how some trend followers built their systems
in the
> > > 70s
> > > > > > or 80s and
> > > > > > > > got rich with them, and how their system did not
really
> > > change
> > > > > > all this
> > > > > > > > time. They didn't change their system because they
say the
> > > > > > market does NOT
> > > > > > > > change. They looked at historic market data from the
1800s
> > > and
> > > > > > the market
> > > > > > > > was as it is right now. So they say.
> > > > > > > >
> > > > > > > > On the other side, lately I have been introduced to
the
> > > concept of
> > > > > > > > ever-changing markets and have had a hard time trying
to
> > > build my
> > > > > > system.
> > > > > > > > Got a very promising start with a system getting
around 15-
> > > 20%
> > > > > > average for
> > > > > > > > April 2007 to April 2008 (with little drawdown, which
mean
> > > that
> > > > > > with
> > > > > > > > leverage I can boost this a lot). In any variation
over
> > > > > > thousands of stocks
> > > > > > > > the results were nearly all positives. But then, I
tested
> > > that
> > > > > > same system
> > > > > > > > for the years 2000 to 2008, and that was
disappointing.
> > > Even more
> > > > > > > > disappointing from 2001 to 2003, another troubled
market
> > > like the
> > > > > > one we are
> > > > > > > > in right now.
> > > > > > > >
> > > > > > > > So here I am, wondering where to go from now.
Aronson's
> > > > > > excellent book
> > > > > > > > talk about the importance of having a very large
sample of
> > > data.
> > > > > > But the
> > > > > > > > problem is: the larger the data, the more "historic"
it
> > > gets and
> > > > > > the less it
> > > > > > > > seems to work.
> > > > > > > >
> > > > > > > > Is my system not working, or did the markets really
change?
> > > Do I
> > > > > > need to
> > > > > > > > make it more robust (that is, it MUST make profit
even from
> > > 2001
> > > > > > to 2003),
> > > > > > > > or can I have complete faith in what happened in the
last
> > > year?
> > > > > > > >
> > > > > > > > All those questions... Would be nice to read what you
think
> > > > > > about this.
> > > > > > > >
> > > > > > > > Louis
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> >
> >
>
------------------------------------
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|