Re: [amibroker] Difference betwee OOS and IS, AmiBroker Email List Archive

On 16/04/2008, Thomas Ludwig <Thomas.Ludwig@xxxxxx> wrote:

> Thank you very much Thomas.  So in fact the walk-forward measures the
> data-mining bias in some way?  I will read what you say I should
> read, and I will look at chapter 20 in Howard's book...

Yes, it's explained there in detail. It's great that Amibroker now
automates this process (that wasn't the case when Howard's book was
published).

>
> But still, so far I get the impression that if I backtested let's say
> cross (ma,ma...) for 2000 to 2008 and I take this best result and it
> is 100% CAR, then eben if OOS is 50% CAR I guess there can still be
> place for data-ming bias (or curve-fitting) because the optimization
> was done with the best result.

If your IS period is 2000-2008 I'm afraid that there is no time left for
the OOS period  ;-) Both periods must not overlap! In the book (and you
can configure Amibroker accordingly) the IS period is chosen to be 2
years and OOS is one year. If you start the process in 2000, the first
IS period would be 2000-2001 and the first OOS period would be 2002.
The second IS period is 2001-2002 and OOS is 2003 - and so forth. In my
understanding this approach simulates the fact that you normally don't
trade a sytem with the same parameters unchanged for many years (unless
it's a really long-term system). Rather, you would re-optimize the
system every 1-2 years to adjust it to changing market conditions.
That's what the walk-forward test is actually doing. Every new
optimization is compared with the results of an OOS period. If the OOS
results are considerably worse than the IS results this is a strong
hint that the sytem will not work in real trading.

>
> Anyway, if I understand correctly what you say, the OOS will ALWAYS
> be less than IS because the IS is optimized - that is, it will take
> the best-of-100 (or 200, etc.) result, and compare it with a more
> random result that would occure in real life.  Am I wrong?

The OOS results are not necessarily worse - but most often they are.
I've analyzed a couple of systems (which I don't trade) that had
beautiful IS equity curves - they all failed the walk-forward test
spectacularly. So it seems they were all overfitted and were not able
to generalize.

Greetings,

Thomas

>
> Thanks,
>
> Louis
>
> 2008/4/16, Thomas Ludwig <Thomas.Ludwig@xxxxxx>:
> > Louis,
> >
> > in the IS period your system is optimized, then the best values
> > from the optimization are used to perform a test over the IS and
> > OOS periods.
> >
> > If the OOS results are worse than the IS results, this means that
> > the system doesn't generalize well enough. BTW: This topic is very
> > well explained in chapter 20 of Howard's book. I also suggest to
> > look at
> >
> > http://www.amibroker.com/kb/2008/02/12/getting-started-with-automat
> >ic-walk-forward-optimization/.
> >
> > I must say, that walk forward testing has completely changed my way
> > of thinking. It's much easier to see now if a trading system is
> > worth a second look.
> >
> > Greetings,
> >
> > Thomas
> >
> > > Hi,
> > >
> > > I've been experimenting with walking-forward, and I have some
> > > questions regarding how it works.
> > >
> > > I ran a complete random optimization or buying/selling using the
> > > variables I set (a MCS in fact), and systematically OOS results
> > > were worst than IS.  I don't understand how it works, because
> > > whatever if the sampling is IS or OOS it is always the same
> > > variables that are in place.
> > >
> > > Anyone could explain how this work?
> > >
> > > Thanks,
> > >
> > > Louis
> >
> > ------------------------------------
> >
> > Please note that this group is for discussion between users only.
> >
> > To get support from AmiBroker please send an e-mail directly to
> > SUPPORT {at} amibroker.com
> >
> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> > http://www.amibroker.com/devlog/
> >
> > For other support material please check also:
> > http://www.amibroker.com/support.html
> > Yahoo! Groups Links

------------------------------------

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:amibroker-digest@xxxxxxxxxxxxxxx
    mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/