Re: [amibroker] Re: Backtest - WF Optimization, AmiBroker Email List Archive

Greetings all --

If the walk forward tests show poor out-of-sample results for the entire out-of-sample period, that means the model is learning noise, not signal. If they are good for a short period out-of-sample, then fall off, that means the characteristics of the market being modeled are changing and you need more frequent reoptimization -- a shorter out-of-sample time period.

If you have a model that has in the past been profitable out-of-sample, and it becomes unprofitable during a period of real trading, you can reoptimize ahead of schedule. In fact, if you have valid system, you can reoptimize as frequently as you want to. You might want to run some walk forward tests to verify that this works for your system. If you have not tested early reoptimizing, but do it to try to resync a system that is in a drawdown, I recommend paper trading the new parameter set for a while to be certain that the model just needed re-synchronizing, and was not terminally broken.

Thanks,
Howard

On Mon, Sep 8, 2008 at 2:05 PM, Mike <sfclimbers@xxxxxxxxx> wrote:

Sometimes it is not so much adaptation that is needed. But rather just
capturing the transition of cycles.

Consider a house located in a region that has warm summers and cold
winters. The variables would be climate control (e.g. air
conditioner/heater). The variable values will change as the seasons
unfold. To optimize comfort, we will be using increased air
conditioning during the summers and increased heating during the
winters.

By experimenting with OOS time periods we may be able to identify
market cycles. If you only reoptimized your values once a year (e.g.
set for average temperature for the year), your climate control would
likely be generally too warm in the summers and generally too cold in
the winters. But, if you reoptimized monthly, then you would gradually
use increasing air conditioning leading into the summer and conversely
gradually increase heating leading into the winter.

Obviously the markets are not so perfectly cyclical as the seasons.
But, the example illustrates the point.

Mike

--- In amibroker@xxxxxxxxxxxxxxx, "Ara Kaloustian" <ara1@xxx> wrote:
>
> WF testing seems like a logical step to take in creating a trading
system. While I am not suggesting that we don't do it, I'd like to
clear my understanding.
>
> It seems to me that we are looking to create a system with
parameters that are valid "for all time". If a test is succesful in
OOS period, then we can assume that we have found something that
remains applicable for the future - at least in this one instance.
>
> Given that the markets change all the time, the obvious conclusion
for me is that we need indicators that are adaptive... and that any
that do not adapt will simply not work in OOS.
>
> The issue of providing feedback from equity curve seems valid and it
can provide a warning about system starting to not perform as expected
... so regardless of type of indicators used, this kind of feedback is
good!.
>
> The point I am making is that there has been very little said about
adaptive indicators ...
>
> maybe that is the holy grail ... and therefore untennable ...
>