Re: [amibroker] Re: Walk Forward Issue, AmiBroker Email List Archive

2009/7/15 brianw468 <wild21@xxxxxxxxxxxxxxx>

Hi Keith,
And many thanks for your excellent commonsense suggestions. Following this up has proved to be a useful learning experience (including a fair amount of re-learning!).
The first lesson is that one should always do some optimisation and backtesting before leaping into a full-blown WF analysis. The second point is that I should not expect max profit when optimising on a different parameter - RAR/MDD in this case. However, I would not have expected the profit to be too far down from the max - though this may be wishful thinking!.
In case anyone is interested this is a summary of what I found by optimising individual steps from the WF process:-
The detailed results and the 3-D graph for my IS1 period showed large plateau areas, with over 100 cases giving the same (max) RAR/MDD. Most of these had the same profit figure (533), but another group (still with the same RAR/MDD) had a lower profit figure. Lesson no.3 was that my optimisation steps were closer spaced than necessary.
Similarly, in IS2 a large number of cases had the same RAR/MDD (16.52) and profit(474). Another group had RAR/MDD at 14.75 and P of 360. Then a third set, and then a fourth with RAR/MDD at 13.14 and P at 1126. This was the profit figure that showed up in the OOS1 results in the WF. It remains a puzzle to me, because a Backtest in OOS1 using the optimum parameters from IS1 gave a profit of 474. It might remain an unsolved mystery.
Finally, I guess that with a system like this that produces only a small number of trades in a 6 month period, I should consider a more complex fitness function - perhaps (e.g.) combining RAR/MDD with Profit in some way.
Thanks again

Brian

--- In amibroker@xxxxxxxxxxxxxxx, Keith McCombs <kmccombs@xxx> wrote:
>
> Brain --
> I don't know what the problem or the solution is. However, this should
> be fairly easy for you to debug.
>
> All you should have to do is repeat the walk forward process, one step
> at a time:
> 1. Optimize for IS1. Then using the resulting parameters, backtest
> IS1, and record the trades.
> 2. Using same parameters, backtest OOS1, and record the trades.
> 3. Compare with your WF results. You should have enough information
> now to find the problem.
> 4. If after 3 above, you still can't figure it out, then optimize OOS1
> and IS2, which should produce identical results. If not, why not?
>
> The secret to debugging is "divide and conquer", just "one step at a time".
> -- Keith
>
> brianw468 wrote:
> >
> >
> > Oops,
> > This message became a mess as Yahoo apparently removed most of the
> > spaces, destroying the layout..
> > Results in the format (Sample, Profit, No. of Trades) are:
> > (IS1,533,1) (OOS1,1126,2) (IS2,474,1) (OOS2,-537,2) (IS3,43,1)
> > (OOS3,784,2).
> > I hope this comes out a bit clearer!
> >
> > Brian
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <mailto:amibroker%40yahoogroups.com>,

> > "brianw468" <wild21@> wrote:
> > >
> > > Hi
> > > I recently tested a trading system but got some strange results from
> > the WF test. The WF process used exhaustive optimization.
> > > An extract from the results (too large to reproduce in full) is:-
> > > Item Profit No. Trades
> > > IS1 533 1
> > > OOS1 1126 2
> > >
> > > IS2 474 1
> > > OOS2 -537 2
> > >
> > > IS3 43 1
> > > OOS 784 2
> > >
> > > In two cases the OOS results are better than the preceding IS
> > results. Though a little surprising,this is quite plausible, because
> > the time periods differ.
> > > However,the IS2 profit is much poorer than the OOS1 profit, and this
> > is a worry since exhaustive optimization is used, and the time periods
> > are the same. Can anyone offer an explanation? Would it lie in the way
> > profit is handled when a trade is not completed by the end of the
> > selected period?
> > >
> > > Thanks
> > > Brian
> > >
> >
> >
>