Re: [amibroker] Re: Difference betwee OOS and IS, AmiBroker Email List Archive

2008/4/18, Mike <sfclimbers@xxxxxxxxx>:

Louis,

I don't think that you are understanding how walk forward works. You
always want the *best* values for use in the next OOS. What you have
to understand is that after the last IS period, you will have a new
set of parameter values that you start trading *live* tomorrow.

You don't want 2nd best. You don't care if last month you were
trading 20/50. What you care about is that starting tomorrow you want
to trade 10/25 (for example). 20/50 is *no longer best*, so why do
you care if it's 2nd, 3rd or 100th best? It's no longer meaningful
going forward. This is why it is critical that you use an objective
function that represents your definition of "best".

As for your comments regarding random tests. It sounds like you are
confusing two issues. Monte Carlo is not Walk Forward.

If you want to apply Monte Carlo when first developing your strategy,
that's fine. Monte Carlo is used to see how much your performance
differs when adding noise to the *same set of data* as the original
optimization. Was it pure chance that your strategy perfomed well
over that data, or did your strategy still do well even when noise
was added to the data (i.e. strategy recognized the signal and was
able to ignore the noise)?

Walk forward is used to validate that your strategy will adapt to the
market and continue to give positive returns on data that it has
*never seen before*.

If you want to add sensitivity analysis to your walk forward process
such that you end up with parameter values that are surrounded by
other well performing values (as opposed to values isolated at the
top of a peak surrounded by poor performing values), then you can use
the walk forward capabilities of IO (See Intelligent Optimizer in
Files section of this group).

http://www.amibroker.org/userkb/?s=intelligent+optimizer

Mike

--- In amibroker@xxxxxxxxxxxxxxx, "Louis Préfontaine"
<rockprog80@xxx> wrote:
>
> Hi Mike (and everyone),
>
> The problem I have with actual walk-forward is this: each
parameters are
> tested only once. As an example, if I choose a Cross ma (c,20), ma
(c,50);
> and optimize the two variables, each time it will only be tested
once. But
> there could be many results with the same variables; I would prefer
to be
> able to do a random test of the same variables let's say 1000 times
to get
> an average of the results. But with walk-forward that would take
WEEKS to
> get it done!
>
> The other problem is this one. Let's say that an IS and OOS
results show
> that 20 and 50 (in the previous example) are the two best results
(based on
> only one test, as I talked previously, but still...). Then, the
next IS and
> OOS shows something different (let's say 10 and 25). But maybe 20
and 50 is
> still good, been the second best of the list. But the problem is
that I
> only see the ultimate best results in the walk-forward! I'd like
to see,
> let's say the 5 best results so if a parameter is in the 5 best
every time I
> have something stable even if it's not the best parameter at the
particular
> IS-OOS time!
>
> Sorry again if I gace someone a headache here. I wish I could
explain
> better what I mean, but maybe you understand my problem.
>
> Thanks,
>
> Louis
>
> 2008/4/18, Mike <sfclimbers@xxx>:

> >
> > Louis,
> >
> > That's the whole point of walk forward analysis! The goal is to
enter
> > each new OOS period with parameter values *most relevant* to what
the
> > market is about to offer. When parameter values change, it is not
a
> > sign of failure. It is a sign of adaptation to changing market
> > conditions.
> >
> > If your parameter values stay the same, that suggests that the
market
> > is continuing as it had been. If your parameter values change,
that
> > suggests that the market has changed and that your strategy has
> > *adapted* to the change.
> >
> > The frequency at which your parameter values change will largely
be
> > dependant upon the length of your IS period. If you have a 10
year IS
> > period and a 2 week OOS period, it will be a loooooooong time
before
> > your parameter values change from one OOS to the next OOS.
> >
> > That is why Howard advised you in your earlier thread to
experiment
> > with different IS period lengths to determine what the ideal IS
> > period length is for your strategy.
> >
> > If you are unable to find a suitable IS:OOS period lengths
> > combination, that suggests that your strategy is poor and needs
to be
> > reworked or abandoned.
> >
> > Mike
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com>, "Louis

> > Préfontaine"
> > <rockprog80@> wrote:
> > >
> > > Hi Thomas,
> > >
> > > I understand what you mean and I agree with you on what you say.
> > My concern
> > > is more about what to do when each IS and the following OOS uses
> > some
> > > parameters and then each new IS-OOS uses another parameter.
> > Moreover, all
> > > those parameters are (as far as I know) the best-of-many and
they
> > are not
> > > resampled, so I see a potential problem and I have trouble
> > identifying where
> > > my system fails because if I use a walk-forward of 2 weeks each
2
> > weeks has
> > > new parameters...
> > >
> > > Do you understand what I mean? Sorry if I am confusing...
English
> > is not
> > > my first language.
> > >
> > > Louis
> > >
> > > 2008/4/18, Thomas Ludwig <Thomas.Ludwig@>:
> >
> > > >
> > > > > The problem with that is the following: let's say my signal
is
> > a MA
> > > > > crossover, and I optimized each MA. I apply a walk-forward
of 3
> > > > > months, and each time the MA Crossover is different. So, in
> > the end,
> > > > > if the OOS is worse than IS, I don't know much more because
> > each time
> > > > > the walk-forward was acting on different MA Crossover.
> > > >
> > > >
> > > > I disagree. Look at it this way: You perform an optimization
ovet
> > your
> > > > IS period, and then you usually select the best parameter
> > combination
> > > > based on the metric you chose (like K-ratio). The basic
> > *assumtion* is
> > > > that this "best" parameter combination gives you a trading
system
> > that
> > > > can generalize - i.e., it will be able to be profitable even
if
> > market
> > > > conditions change somewhat. But you can't be sure before you
> > apply this
> > > > optimized system to NEW data different from the IS data -
that's
> > the
> > > > OOS period. That's exactly what walk-forward is doing. It just
> > adds one
> > > > more assumption: That you normally wouldn't stick to the same
> > > > parameters over several years. That's why it walks forward
though
> > the
> > > > time comparing the optimized result of the respective IS
period
> > with
> > > > the NEW data of the subsequent OOS period. Now, if you compare
> > your IS
> > > > results with the OOS results and you find the latter ones
> > considerably
> > > > worse that shows that your system is not robust. Think about
the
> > 3D
> > > > optimization graph available in Amibroker: Such a poor system
> > would
> > > > probably produce a 3D surface plot with big changes (spikes,
> > drops)
> > > > instead of stable plateaus - i.e. it is *very* dependent on
the
> > > > parameters you chose. A small change in the parameters for the
> > tested
> > > > period would lead to poor results as the system has
> > only "learned" the
> > > > market structure in that period and will fail if that changes
over
> > > > time.
> > > >
> > > >
> > > > > Do you
> > > > > understand what I mean? And because the walk-forward only
> > shows the
> > > > > BEST result of ONE optimized portfolio result (RAR, RRR,
CAR,
> > etc.) I
> > > > > just can't make it better because I can't see what is result
> > number
> > > > > 2, or 3. In that way, optimization seems superior, but
maybe I
> > am
> > > > > not using walk-forward correctly. That is my main concern!
> > > > >
> > > > > Louis
> > > > >
> > > > > 2008/4/17, Louis Préfontaine <rockprog80@>:
> > > > > > Hi Thomas,
> > > > > >
> > > > > > Do you use Walk-forward as a random optimizer like Monte
Carlo
> > > > > > Simulation or do you use it as a parameter optimization
> > (let's say
> > > > > > you want to know the best numbers for a MA crossover). Or
> > maybe
> > > > > > both?
> > > > > >
> > > > > > I ask this because my feeling is that if I use it only as
a
> > > > > > parameter optimization, each parameter would be tested
only
> > one
> > > > > > time in each IS or OOS, hence this could not be
> > significative. I
> > > > > > tried to add a random simulation 5 times to get the best
out
> > of 5
> > > > > > results, but I was wondering if this was correct or
simply a
> > waste
> > > > > > of time.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Louis
> > > > > >
> > > > > > 2008/4/16, Thomas Ludwig <Thomas.Ludwig@>:
> > > > > > > > Thank you very much Thomas. So in fact the walk-
forward
> > > > > > > > measures the data-mining bias in some way? I will read
> > what
> > > > > > > > you say I should read, and I will look at chapter 20
in
> > > > > > > > Howard's book...
> > > > > > >
> > > > > > > Yes, it's explained there in detail. It's great that
> > Amibroker
> > > > > > > now automates this process (that wasn't the case when
> > Howard's
> > > > > > > book was published).
> > > > > > >
> > > > > > > > But still, so far I get the impression that if I
> > backtested
> > > > > > > > let's say cross (ma,ma...) for 2000 to 2008 and I take
> > this
> > > > > > > > best result and it is 100% CAR, then eben if OOS is
50%
> > CAR I
> > > > > > > > guess there can still be place for data-ming bias (or
> > > > > > > > curve-fitting) because the optimization was done with
the
> > best
> > > > > > > > result.
> > > > > > >
> > > > > > > If your IS period is 2000-2008 I'm afraid that there is
no
> > time
> > > > > > > left for the OOS period ;-) Both periods must not
overlap!
> > In
> > > > > > > the book (and you can configure Amibroker accordingly)
the
> > IS
> > > > > > > period is chosen to be 2 years and OOS is one year. If
you
> > start
> > > > > > > the process in 2000, the first IS period would be 2000-
2001
> > and
> > > > > > > the first OOS period would be 2002. The second IS
period is
> > > > > > > 2001-2002 and OOS is 2003 - and so forth. In my
> > understanding
> > > > > > > this approach simulates the fact that you normally don't
> > trade a
> > > > > > > sytem with the same parameters unchanged for many years
> > (unless
> > > > > > > it's a really long-term system). Rather, you would re-
> > optimize
> > > > > > > the system every 1-2 years to adjust it to changing
market
> > > > > > > conditions. That's what the walk-forward test is
actually
> > doing.
> > > > > > > Every new optimization is compared with the results of
an
> > OOS
> > > > > > > period. If the OOS results are considerably worse than
the
> > IS
> > > > > > > results this is a strong hint that the sytem will not
work
> > in
> > > > > > > real trading.
> > > > > > >
> > > > > > > > Anyway, if I understand correctly what you say, the
OOS
> > will
> > > > > > > > ALWAYS be less than IS because the IS is optimized -
that
> > is,
> > > > > > > > it will take the best-of-100 (or 200, etc.) result,
and
> > compare
> > > > > > > > it with a more random result that would occure in real
> > life.
> > > > > > > > Am I wrong?
> > > > > > >
> > > > > > > The OOS results are not necessarily worse - but most
often
> > they
> > > > > > > are. I've analyzed a couple of systems (which I don't
> > trade) that
> > > > > > > had beautiful IS equity curves - they all failed the
walk-
> > forward
> > > > > > > test spectacularly. So it seems they were all overfitted
> > and were
> > > > > > > not able to generalize.
> > > > > > >
> > > > > > > Greetings,
> > > > > > >
> > > > > > >
> > > > > > > Thomas
> > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Louis
> > > > > > > >
> > > > > > > > 2008/4/16, Thomas Ludwig <Thomas.Ludwig@>:
> > > > > > > > > Louis,
> > > > > > > > >
> > > > > > > > > in the IS period your system is optimized, then the
best
> > > > > > > > > values from the optimization are used to perform a
test
> > over
> > > > > > > > > the IS and OOS periods.
> > > > > > > > >
> > > > > > > > > If the OOS results are worse than the IS results,
this
> > means
> > > > > > > > > that the system doesn't generalize well enough. BTW:
> > This
> > > > > > > > > topic is very well explained in chapter 20 of
Howard's
> > book.
> > > > > > > > > I also suggest to look at
> > > > > > > > >
> > > > > > > > > http://www.amibroker.com/kb/2008/02/12/getting-
started-
> > with-a
> > > >
> > > > > > > > >utomat ic-walk-forward-optimization/.
> > > >
> > > > > > > > >
> > > > > > > > > I must say, that walk forward testing has completely
> > changed
> > > > > > > > > my way of thinking. It's much easier to see now if a
> > trading
> > > > > > > > > system is worth a second look.
> > > > > > > > >
> > > > > > > > > Greetings,
> > > > > > > > >
> > > > > > > > > Thomas
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I've been experimenting with walking-forward, and
I
> > have
> > > > > > > > > > some questions regarding how it works.
> > > > > > > > > >
> > > > > > > > > > I ran a complete random optimization or
buying/selling
> > > > > > > > > > using the variables I set (a MCS in fact), and
> > > > > > > > > > systematically OOS results were worst than IS. I
> > don't
> > > > > > > > > > understand how it works, because whatever if the
> > sampling
> > > > > > > > > > is IS or OOS it is always the same variables that
are
> > in
> > > > > > > > > > place.
> > > > > > > > > >
> > > > > > > > > > Anyone could explain how this work?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Louis
> > > > > > > > >
> > > > > > > > > ------------------------------------
> > > > > > > > >
> > > > > > > > > Please note that this group is for discussion
between
> > users
> > > > > > > > > only.
> > > > > > > > >
> > > > > > > > > To get support from AmiBroker please send an e-mail
> > directly
> > > > > > > > > to SUPPORT {at} amibroker.com
> > > > > > > > >
> > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always
> > check
> > > > > > > > > DEVLOG: http://www.amibroker.com/devlog/
> > > > > > > > >
> > > > > > > > > For other support material please check also:
> > > > > > > > > http://www.amibroker.com/support.html
> > > > > > > > > Yahoo! Groups Links
> > > > > > >
> > > > > > > ------------------------------------
> > > > > > >
> > > > > > > Please note that this group is for discussion between
users
> > only.
> > > > > > >
> > > > > > > To get support from AmiBroker please send an e-mail
> > directly to
> > > > > > > SUPPORT {at} amibroker.com
> > > > > > >
> > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always
check
> > DEVLOG:
> > > > > > > http://www.amibroker.com/devlog/
> > > > > > >
> > > > > > > For other support material please check also:
> > > > > > > http://www.amibroker.com/support.html
> > > > > > > Yahoo! Groups Links
> > > >
> > > >
> > > >
> > > > ------------------------------------
> > > >
> > > > Please note that this group is for discussion between users
only.
> > > >
> > > > To get support from AmiBroker please send an e-mail directly
to
> > > > SUPPORT {at} amibroker.com
> > > >
> > > > For NEW RELEASE ANNOUNCEMENTS and other news always check
DEVLOG:
> > > > http://www.amibroker.com/devlog/
> > > >
> > > > For other support material please check also:
> > > > http://www.amibroker.com/support.html
> > > > Yahoo! Groups Links
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>