Re: [amibroker] Difference betwee OOS and IS, AmiBroker Email List Archive

2008/4/18, Thomas Ludwig <Thomas.Ludwig@xxxxxx>:

> The problem with that is the following:  let's say my signal is a MA
> crossover, and I optimized each MA.  I apply a walk-forward of 3
> months, and each time the MA Crossover is different.  So, in the end,
> if the OOS is worse than IS, I don't know much more because each time
> the walk-forward was acting on different MA Crossover.

I disagree. Look at it this way: You perform an optimization ovet your
IS period, and then you usually select the best parameter combination
based on the metric you chose (like K-ratio). The basic *assumtion* is
that this "best" parameter combination gives you a trading system that
can generalize - i.e., it will be able to be profitable even if market
conditions change somewhat. But you can't be sure before you apply this
optimized system to NEW data different from the IS data - that's the
OOS period. That's exactly what walk-forward is doing. It just adds one
more assumption: That you normally wouldn't stick to the same
parameters over several years. That's why it walks forward though the
time comparing the optimized result of the respective IS period with
the NEW data of the subsequent OOS period. Now, if you compare your IS
results with the OOS results and you find the latter ones considerably
worse that shows that your system is not robust. Think about the 3D
optimization graph available in Amibroker: Such a poor system would
probably produce a 3D surface plot with big changes (spikes, drops)
instead of stable plateaus - i.e. it is *very* dependent on the
parameters you chose. A small change in the parameters for the tested
period would lead to poor results as the system has only "learned" the
market structure in that period and will fail if that changes over
time.

> Do you
> understand what I mean?  And because the walk-forward only shows the
> BEST result of ONE optimized portfolio result (RAR, RRR, CAR, etc.) I
> just can't make it better because I can't see what is result number
> 2, or 3.  In that way, optimization seems superior, but maybe I am
> not using walk-forward correctly.  That is my main concern!
>
> Louis
>
> 2008/4/17, Louis Préfontaine <rockprog80@xxxxxxxxx>:
> > Hi Thomas,
> >
> > Do you use Walk-forward as a random optimizer like Monte Carlo
> > Simulation or do you use it as a parameter optimization (let's say
> > you want to know the best numbers for a MA crossover).  Or maybe
> > both?
> >
> > I ask this because my feeling is that if I use it only as a
> > parameter optimization, each parameter would be tested only one
> > time in each IS or OOS, hence this could not be significative.  I
> > tried to add a random simulation 5 times to get the best out of 5
> > results, but I was wondering if this was correct or simply a waste
> > of time.
> >
> > Thanks,
> >
> > Louis
> >
> > 2008/4/16, Thomas Ludwig <Thomas.Ludwig@xxxxxx>:
> > > > Thank you very much Thomas.  So in fact the walk-forward
> > > > measures the data-mining bias in some way?  I will read what
> > > > you say I should read, and I will look at chapter 20 in
> > > > Howard's book...
> > >
> > > Yes, it's explained there in detail. It's great that Amibroker
> > > now automates this process (that wasn't the case when Howard's
> > > book was published).
> > >
> > > > But still, so far I get the impression that if I backtested
> > > > let's say cross (ma,ma...) for 2000 to 2008 and I take this
> > > > best result and it is 100% CAR, then eben if OOS is 50% CAR I
> > > > guess there can still be place for data-ming bias (or
> > > > curve-fitting) because the optimization was done with the best
> > > > result.
> > >
> > > If your IS period is 2000-2008 I'm afraid that there is no time
> > > left for the OOS period  ;-) Both periods must not overlap! In
> > > the book (and you can configure Amibroker accordingly) the IS
> > > period is chosen to be 2 years and OOS is one year. If you start
> > > the process in 2000, the first IS period would be 2000-2001 and
> > > the first OOS period would be 2002. The second IS period is
> > > 2001-2002 and OOS is 2003 - and so forth. In my understanding
> > > this approach simulates the fact that you normally don't trade a
> > > sytem with the same parameters unchanged for many years (unless
> > > it's a really long-term system). Rather, you would re-optimize
> > > the system every 1-2 years to adjust it to changing market
> > > conditions. That's what the walk-forward test is actually doing.
> > > Every new optimization is compared with the results of an OOS
> > > period. If the OOS results are considerably worse than the IS
> > > results this is a strong hint that the sytem will not work in
> > > real trading.
> > >
> > > > Anyway, if I understand correctly what you say, the OOS will
> > > > ALWAYS be less than IS because the IS is optimized - that is,
> > > > it will take the best-of-100 (or 200, etc.) result, and compare
> > > > it with a more random result that would occure in real life.
> > > > Am I wrong?
> > >
> > > The OOS results are not necessarily worse - but most often they
> > > are. I've analyzed a couple of systems (which I don't trade) that
> > > had beautiful IS equity curves - they all failed the walk-forward
> > > test spectacularly. So it seems they were all overfitted and were
> > > not able to generalize.
> > >
> > > Greetings,
> > >
> > >
> > > Thomas
> > >
> > > > Thanks,
> > > >
> > > > Louis
> > > >
> > > > 2008/4/16, Thomas Ludwig <Thomas.Ludwig@xxxxxx>:
> > > > > Louis,
> > > > >
> > > > > in the IS period your system is optimized, then the best
> > > > > values from the optimization are used to perform a test over
> > > > > the IS and OOS periods.
> > > > >
> > > > > If the OOS results are worse than the IS results, this means
> > > > > that the system doesn't generalize well enough. BTW: This
> > > > > topic is very well explained in chapter 20 of Howard's book.
> > > > > I also suggest to look at
> > > > >
> > > > > http://www.amibroker.com/kb/2008/02/12/getting-started-with-a

> > > > >utomat ic-walk-forward-optimization/.

> > > > >
> > > > > I must say, that walk forward testing has completely changed
> > > > > my way of thinking. It's much easier to see now if a trading
> > > > > system is worth a second look.
> > > > >
> > > > > Greetings,
> > > > >
> > > > > Thomas
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I've been experimenting with walking-forward, and I have
> > > > > > some questions regarding how it works.
> > > > > >
> > > > > > I ran a complete random optimization or buying/selling
> > > > > > using the variables I set (a MCS in fact), and
> > > > > > systematically OOS results were worst than IS.  I don't
> > > > > > understand how it works, because whatever if the sampling
> > > > > > is IS or OOS it is always the same variables that are in
> > > > > > place.
> > > > > >
> > > > > > Anyone could explain how this work?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Louis
> > > > >
> > > > > ------------------------------------
> > > > >
> > > > > Please note that this group is for discussion between users
> > > > > only.
> > > > >
> > > > > To get support from AmiBroker please send an e-mail directly
> > > > > to SUPPORT {at} amibroker.com
> > > > >
> > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check
> > > > > DEVLOG: http://www.amibroker.com/devlog/
> > > > >
> > > > > For other support material please check also:
> > > > > http://www.amibroker.com/support.html
> > > > > Yahoo! Groups Links
> > >
> > > ------------------------------------
> > >
> > > Please note that this group is for discussion between users only.
> > >
> > > To get support from AmiBroker please send an e-mail directly to
> > > SUPPORT {at} amibroker.com
> > >
> > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> > > http://www.amibroker.com/devlog/
> > >
> > > For other support material please check also:
> > > http://www.amibroker.com/support.html
> > > Yahoo! Groups Links

------------------------------------

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:amibroker-digest@xxxxxxxxxxxxxxx
    mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/