Re: [amibroker] Re: Difference betwee OOS and IS, AmiBroker Email List Archive

2008/4/18, Mike <sfclimbers@xxxxxxxxx>:

Louis,

Comments added in line...

--- In amibroker@xxxxxxxxxxxxxxx, "Louis Préfontaine"
<rockprog80@xxx> wrote:
>
> Hi Mike,
>
> Thanks for your reply. I understand what you say, but personnaly I
would
> appreciate to see other results than best results, because it makes
it
> possible to adjust a particular system.

Design your system based on optimization of one period. Make all the
adjustments necessary until you feel you have a workable system. Then
try walk forward.

If you have to adjust your system, then your system is not ready to
trade. Walk forward will make that very clear for you because your
OOS results will be marginal or poor. If you think that one set of
values should have been selected, but weren't, then you need to
reconsider your objective function.

> As an example, let's say that my
> 20,50 cross was first and now is second, hence the walk-forward
chooses
> something else for the next OOS. But maybe over the long run -
that is,
> many years - 20,50 would be in the top 5 and I only have to adjust
to what
> changed in the market (e.g. volatility, market going down, etc.) to
make it
> better.
> You see, in this particular case, I don't have to reject the 20,50
crossover
> as the walk-forward would propose, but only to make my 20,50 better
because
> in the long run it is more stable (I would see this if I could see
the 5
> best instead of only the best one).

It is highly unlikely that you will design a system that has fixed
parameter values (e.g. 20/50) and is optimal for all markets. The
best you can hope for with that approach is a system that is able to
get steady performance in most markets and still not get killed in a
bad market. It is far more likely that you will have to periodically
adjust your values to get optimal results as time goes on.

> Maybe I am not making sense, but knowing that 13,31 is better than
20,50
> which is now better than 14,18 is not helping me a lot;

It is helping your trading account a lot ;) If 13/31 is optimal then
that is the one that is going to give the the most money. Once 13/31
is no longer best, then 20/50 is going to give you the most money,
and so on.

> I'd prefer to see
> which of those systems performed well in what kind of market and
then adjust
> the parameters to the market.

That is *exactly* what walk forward is already doing for you. The IS
period tells you "what kind of market" and then gives you
the "adjusted parameter values" to use. The whole approach is based
on the assumption that there are no fixed values that will work for
all maket conditions. Therefore, what are the best values right *now*.

> Do you understand what I mean?

Yes, but you have to decide what your philosophy is. Do you believe
that you can design a strategy that never changes values once you
start trading. Or, do you believe that a strategy must periodically
adjust to changes in the market *as they happen*. If you believe that
you must adjust to changes, then it only makes sense to use the
*best* values when those changes happen.

Mike

> Thanks!
>
> Louis
>
> 2008/4/18, Mike <sfclimbers@xxx>:
> >
> > Louis,
> >
> > I don't think that you are understanding how walk forward works.
You
> > always want the *best* values for use in the next OOS. What you
have
> > to understand is that after the last IS period, you will have a
new
> > set of parameter values that you start trading *live* tomorrow.
> >
> > You don't want 2nd best. You don't care if last month you were
> > trading 20/50. What you care about is that starting tomorrow you
want
> > to trade 10/25 (for example). 20/50 is *no longer best*, so why do
> > you care if it's 2nd, 3rd or 100th best? It's no longer meaningful
> > going forward. This is why it is critical that you use an
objective
> > function that represents your definition of "best".
> >
> > As for your comments regarding random tests. It sounds like you
are
> > confusing two issues. Monte Carlo is not Walk Forward.
> >
> > If you want to apply Monte Carlo when first developing your
strategy,
> > that's fine. Monte Carlo is used to see how much your performance
> > differs when adding noise to the *same set of data* as the
original
> > optimization. Was it pure chance that your strategy perfomed well
> > over that data, or did your strategy still do well even when noise
> > was added to the data (i.e. strategy recognized the signal and was
> > able to ignore the noise)?
> >
> > Walk forward is used to validate that your strategy will adapt to
the
> > market and continue to give positive returns on data that it has
> > *never seen before*.
> >
> > If you want to add sensitivity analysis to your walk forward
process
> > such that you end up with parameter values that are surrounded by
> > other well performing values (as opposed to values isolated at the
> > top of a peak surrounded by poor performing values), then you can
use
> > the walk forward capabilities of IO (See Intelligent Optimizer in
> > Files section of this group).
> >
> > http://www.amibroker.org/userkb/?s=intelligent+optimizer
> >
> > Mike
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com>, "Louis
> > Préfontaine"
> > <rockprog80@> wrote:
> > >
> > > Hi Mike (and everyone),
> > >
> > > The problem I have with actual walk-forward is this: each
> > parameters are
> > > tested only once. As an example, if I choose a Cross ma (c,20),
ma
> > (c,50);
> > > and optimize the two variables, each time it will only be tested
> > once. But
> > > there could be many results with the same variables; I would
prefer
> > to be
> > > able to do a random test of the same variables let's say 1000
times
> > to get
> > > an average of the results. But with walk-forward that would take
> > WEEKS to
> > > get it done!
> > >
> > > The other problem is this one. Let's say that an IS and OOS
> > results show
> > > that 20 and 50 (in the previous example) are the two best
results
> > (based on
> > > only one test, as I talked previously, but still...). Then, the
> > next IS and
> > > OOS shows something different (let's say 10 and 25). But maybe
20
> > and 50 is
> > > still good, been the second best of the list. But the problem is
> > that I
> > > only see the ultimate best results in the walk-forward! I'd like
> > to see,
> > > let's say the 5 best results so if a parameter is in the 5 best
> > every time I
> > > have something stable even if it's not the best parameter at the
> > particular
> > > IS-OOS time!
> > >
> > > Sorry again if I gace someone a headache here. I wish I could
> > explain
> > > better what I mean, but maybe you understand my problem.
> > >
> > > Thanks,
> > >
> > > Louis
> > >
> > > 2008/4/18, Mike <sfclimbers@>:
> > > >
> > > > Louis,
> > > >
> > > > That's the whole point of walk forward analysis! The goal is
to
> > enter
> > > > each new OOS period with parameter values *most relevant* to
what
> > the
> > > > market is about to offer. When parameter values change, it is
not
> > a
> > > > sign of failure. It is a sign of adaptation to changing market
> > > > conditions.
> > > >
> > > > If your parameter values stay the same, that suggests that the
> > market
> > > > is continuing as it had been. If your parameter values change,
> > that
> > > > suggests that the market has changed and that your strategy
has
> > > > *adapted* to the change.
> > > >
> > > > The frequency at which your parameter values change will
largely
> > be
> > > > dependant upon the length of your IS period. If you have a 10
> > year IS
> > > > period and a 2 week OOS period, it will be a loooooooong time
> > before
> > > > your parameter values change from one OOS to the next OOS.
> > > >
> > > > That is why Howard advised you in your earlier thread to
> > experiment
> > > > with different IS period lengths to determine what the ideal
IS
> > > > period length is for your strategy.
> > > >
> > > > If you are unable to find a suitable IS:OOS period lengths
> > > > combination, that suggests that your strategy is poor and
needs
> > to be
> > > > reworked or abandoned.
> > > >
> > > > Mike
> > > >
> > > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%

40yahoogroups.com><amibroker%

> > 40yahoogroups.com>, "Louis
> > > > Préfontaine"
> > > > <rockprog80@> wrote:
> > > > >
> > > > > Hi Thomas,
> > > > >
> > > > > I understand what you mean and I agree with you on what you
say.
> > > > My concern
> > > > > is more about what to do when each IS and the following OOS
uses
> > > > some
> > > > > parameters and then each new IS-OOS uses another parameter.
> > > > Moreover, all
> > > > > those parameters are (as far as I know) the best-of-many and
> > they
> > > > are not
> > > > > resampled, so I see a potential problem and I have trouble
> > > > identifying where
> > > > > my system fails because if I use a walk-forward of 2 weeks
each
> > 2
> > > > weeks has
> > > > > new parameters...
> > > > >
> > > > > Do you understand what I mean? Sorry if I am confusing...
> > English
> > > > is not
> > > > > my first language.
> > > > >
> > > > > Louis
> > > > >
> > > > > 2008/4/18, Thomas Ludwig <Thomas.Ludwig@>:
> > > >
> > > > > >
> > > > > > > The problem with that is the following: let's say my
signal
> > is
> > > > a MA
> > > > > > > crossover, and I optimized each MA. I apply a walk-
forward
> > of 3
> > > > > > > months, and each time the MA Crossover is different.
So, in
> > > > the end,
> > > > > > > if the OOS is worse than IS, I don't know much more
because
> > > > each time
> > > > > > > the walk-forward was acting on different MA Crossover.
> > > > > >
> > > > > >
> > > > > > I disagree. Look at it this way: You perform an
optimization
> > ovet
> > > > your
> > > > > > IS period, and then you usually select the best parameter
> > > > combination
> > > > > > based on the metric you chose (like K-ratio). The basic
> > > > *assumtion* is
> > > > > > that this "best" parameter combination gives you a trading
> > system
> > > > that
> > > > > > can generalize - i.e., it will be able to be profitable
even
> > if
> > > > market
> > > > > > conditions change somewhat. But you can't be sure before
you
> > > > apply this
> > > > > > optimized system to NEW data different from the IS data -
> > that's
> > > > the
> > > > > > OOS period. That's exactly what walk-forward is doing. It
just
> > > > adds one
> > > > > > more assumption: That you normally wouldn't stick to the
same
> > > > > > parameters over several years. That's why it walks forward
> > though
> > > > the
> > > > > > time comparing the optimized result of the respective IS
> > period
> > > > with
> > > > > > the NEW data of the subsequent OOS period. Now, if you
compare
> > > > your IS
> > > > > > results with the OOS results and you find the latter ones
> > > > considerably
> > > > > > worse that shows that your system is not robust. Think
about
> > the
> > > > 3D
> > > > > > optimization graph available in Amibroker: Such a poor
system
> > > > would
> > > > > > probably produce a 3D surface plot with big changes
(spikes,
> > > > drops)
> > > > > > instead of stable plateaus - i.e. it is *very* dependent
on
> > the
> > > > > > parameters you chose. A small change in the parameters
for the
> > > > tested
> > > > > > period would lead to poor results as the system has
> > > > only "learned" the
> > > > > > market structure in that period and will fail if that
changes
> > over
> > > > > > time.
> > > > > >
> > > > > >
> > > > > > > Do you
> > > > > > > understand what I mean? And because the walk-forward
only
> > > > shows the
> > > > > > > BEST result of ONE optimized portfolio result (RAR, RRR,
> > CAR,
> > > > etc.) I
> > > > > > > just can't make it better because I can't see what is
result
> > > > number
> > > > > > > 2, or 3. In that way, optimization seems superior, but
> > maybe I
> > > > am
> > > > > > > not using walk-forward correctly. That is my main
concern!
> > > > > > >
> > > > > > > Louis
> > > > > > >
> > > > > > > 2008/4/17, Louis Préfontaine <rockprog80@>:
> > > > > > > > Hi Thomas,
> > > > > > > >
> > > > > > > > Do you use Walk-forward as a random optimizer like
Monte
> > Carlo
> > > > > > > > Simulation or do you use it as a parameter
optimization
> > > > (let's say
> > > > > > > > you want to know the best numbers for a MA
crossover). Or
> > > > maybe
> > > > > > > > both?
> > > > > > > >
> > > > > > > > I ask this because my feeling is that if I use it
only as
> > a
> > > > > > > > parameter optimization, each parameter would be tested
> > only
> > > > one
> > > > > > > > time in each IS or OOS, hence this could not be
> > > > significative. I
> > > > > > > > tried to add a random simulation 5 times to get the
best
> > out
> > > > of 5
> > > > > > > > results, but I was wondering if this was correct or
> > simply a
> > > > waste
> > > > > > > > of time.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Louis
> > > > > > > >
> > > > > > > > 2008/4/16, Thomas Ludwig <Thomas.Ludwig@>:
> > > > > > > > > > Thank you very much Thomas. So in fact the walk-
> > forward
> > > > > > > > > > measures the data-mining bias in some way? I will
read
> > > > what
> > > > > > > > > > you say I should read, and I will look at chapter
20
> > in
> > > > > > > > > > Howard's book...
> > > > > > > > >
> > > > > > > > > Yes, it's explained there in detail. It's great that
> > > > Amibroker
> > > > > > > > > now automates this process (that wasn't the case
when
> > > > Howard's
> > > > > > > > > book was published).
> > > > > > > > >
> > > > > > > > > > But still, so far I get the impression that if I
> > > > backtested
> > > > > > > > > > let's say cross (ma,ma...) for 2000 to 2008 and I
take
> > > > this
> > > > > > > > > > best result and it is 100% CAR, then eben if OOS
is
> > 50%
> > > > CAR I
> > > > > > > > > > guess there can still be place for data-ming bias
(or
> > > > > > > > > > curve-fitting) because the optimization was done
with
> > the
> > > > best
> > > > > > > > > > result.
> > > > > > > > >
> > > > > > > > > If your IS period is 2000-2008 I'm afraid that
there is
> > no
> > > > time
> > > > > > > > > left for the OOS period ;-) Both periods must not
> > overlap!
> > > > In
> > > > > > > > > the book (and you can configure Amibroker
accordingly)
> > the
> > > > IS
> > > > > > > > > period is chosen to be 2 years and OOS is one year.
If
> > you
> > > > start
> > > > > > > > > the process in 2000, the first IS period would be
2000-
> > 2001
> > > > and
> > > > > > > > > the first OOS period would be 2002. The second IS
> > period is
> > > > > > > > > 2001-2002 and OOS is 2003 - and so forth. In my
> > > > understanding
> > > > > > > > > this approach simulates the fact that you normally
don't
> > > > trade a
> > > > > > > > > sytem with the same parameters unchanged for many
years
> > > > (unless
> > > > > > > > > it's a really long-term system). Rather, you would
re-
> > > > optimize
> > > > > > > > > the system every 1-2 years to adjust it to changing
> > market
> > > > > > > > > conditions. That's what the walk-forward test is
> > actually
> > > > doing.
> > > > > > > > > Every new optimization is compared with the results
of
> > an
> > > > OOS
> > > > > > > > > period. If the OOS results are considerably worse
than
> > the
> > > > IS
> > > > > > > > > results this is a strong hint that the sytem will
not
> > work
> > > > in
> > > > > > > > > real trading.
> > > > > > > > >
> > > > > > > > > > Anyway, if I understand correctly what you say,
the
> > OOS
> > > > will
> > > > > > > > > > ALWAYS be less than IS because the IS is
optimized -
> > that
> > > > is,
> > > > > > > > > > it will take the best-of-100 (or 200, etc.)
result,
> > and
> > > > compare
> > > > > > > > > > it with a more random result that would occure in
real
> > > > life.
> > > > > > > > > > Am I wrong?
> > > > > > > > >
> > > > > > > > > The OOS results are not necessarily worse - but most
> > often
> > > > they
> > > > > > > > > are. I've analyzed a couple of systems (which I
don't
> > > > trade) that
> > > > > > > > > had beautiful IS equity curves - they all failed the
> > walk-
> > > > forward
> > > > > > > > > test spectacularly. So it seems they were all
overfitted
> > > > and were
> > > > > > > > > not able to generalize.
> > > > > > > > >
> > > > > > > > > Greetings,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thomas
> > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Louis
> > > > > > > > > >
> > > > > > > > > > 2008/4/16, Thomas Ludwig <Thomas.Ludwig@>:
> > > > > > > > > > > Louis,
> > > > > > > > > > >
> > > > > > > > > > > in the IS period your system is optimized, then
the
> > best
> > > > > > > > > > > values from the optimization are used to
perform a
> > test
> > > > over
> > > > > > > > > > > the IS and OOS periods.
> > > > > > > > > > >
> > > > > > > > > > > If the OOS results are worse than the IS
results,
> > this
> > > > means
> > > > > > > > > > > that the system doesn't generalize well enough.
BTW:
> > > > This
> > > > > > > > > > > topic is very well explained in chapter 20 of
> > Howard's
> > > > book.
> > > > > > > > > > > I also suggest to look at
> > > > > > > > > > >
> > > > > > > > > > > http://www.amibroker.com/kb/2008/02/12/getting-
> > started-
> > > > with-a
> > > > > >
> > > > > > > > > > >utomat ic-walk-forward-optimization/.
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I must say, that walk forward testing has
completely
> > > > changed
> > > > > > > > > > > my way of thinking. It's much easier to see now
if a
> > > > trading
> > > > > > > > > > > system is worth a second look.
> > > > > > > > > > >
> > > > > > > > > > > Greetings,
> > > > > > > > > > >
> > > > > > > > > > > Thomas
> > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > I've been experimenting with walking-forward,
and
> > I
> > > > have
> > > > > > > > > > > > some questions regarding how it works.
> > > > > > > > > > > >
> > > > > > > > > > > > I ran a complete random optimization or
> > buying/selling
> > > > > > > > > > > > using the variables I set (a MCS in fact), and
> > > > > > > > > > > > systematically OOS results were worst than
IS. I
> > > > don't
> > > > > > > > > > > > understand how it works, because whatever if
the
> > > > sampling
> > > > > > > > > > > > is IS or OOS it is always the same variables
that
> > are
> > > > in
> > > > > > > > > > > > place.
> > > > > > > > > > > >
> > > > > > > > > > > > Anyone could explain how this work?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Louis
> > > > > > > > > > >
> > > > > > > > > > > ------------------------------------
> > > > > > > > > > >
> > > > > > > > > > > Please note that this group is for discussion
> > between
> > > > users
> > > > > > > > > > > only.
> > > > > > > > > > >
> > > > > > > > > > > To get support from AmiBroker please send an e-
mail
> > > > directly
> > > > > > > > > > > to SUPPORT {at} amibroker.com
> > > > > > > > > > >
> > > > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news
always
> > > > check
> > > > > > > > > > > DEVLOG: http://www.amibroker.com/devlog/
> > > > > > > > > > >
> > > > > > > > > > > For other support material please check also:
> > > > > > > > > > > http://www.amibroker.com/support.html
> > > > > > > > > > > Yahoo! Groups Links
> > > > > > > > >
> > > > > > > > > ------------------------------------
> > > > > > > > >
> > > > > > > > > Please note that this group is for discussion
between
> > users
> > > > only.
> > > > > > > > >
> > > > > > > > > To get support from AmiBroker please send an e-mail
> > > > directly to
> > > > > > > > > SUPPORT {at} amibroker.com
> > > > > > > > >
> > > > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always
> > check
> > > > DEVLOG:
> > > > > > > > > http://www.amibroker.com/devlog/
> > > > > > > > >
> > > > > > > > > For other support material please check also:
> > > > > > > > > http://www.amibroker.com/support.html
> > > > > > > > > Yahoo! Groups Links
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------------
> > > > > >
> > > > > > Please note that this group is for discussion between
users
> > only.
> > > > > >
> > > > > > To get support from AmiBroker please send an e-mail
directly
> > to
> > > > > > SUPPORT {at} amibroker.com
> > > > > >
> > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check
> > DEVLOG:
> > > > > > http://www.amibroker.com/devlog/
> > > > > >
> > > > > > For other support material please check also:
> > > > > > http://www.amibroker.com/support.html
> > > > > > Yahoo! Groups Links
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>