[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] Re: Walk-Forward Out of Sample (OOS) Testing



PureBytes Links

Trading Reference Links

Among other comments, Chuck Rademacher wrote:


> I WOULD NEVER DREAM OF USING DIFFERENT PARAMETERS FOR  STOCKS, EITHER
UNIQUE
> TO EACH STOCK OR UNIQUE TO DIFFERENT TIME PERIODS.

This is pretty much how I feel about futures trading, though some folks have
argued strongly that certain contracts have unique characters that must be
taken into account in trying to trade them.  For example, it's pretty easy
to devise systems that are moderately profitable on the S&P but lose money
on nearly any other contract.  One minor test I use to decide whether a
trading technique is really interesting is to run it past the S&P, T-bonds,
and yen.  If it makes decent money on all three, it's worth looking at.
Very few pass this test.

However, the articles that Meyers has published in Active Trader seem to
take a different view.  This is never clearly stated, but it seems implicit
in the author's method of testing.   In each of these articles, he optimizes
his parameters over one month of data and "proves" that his system works by
testing it on the next four weeks of data.  That suggests that he intends us
to step each window ahead and re-optimize, so that each month of
out-of-sample testing uses the best parameters from the previous month.  One
of his articles states, "To use this system in real-time trading, at least
20 more test and out-of-sample windows should be examined to confirm the
viability of the results shown here."  This injunction to use both "test and
out-of-sample windows" seems to verify that he intends to re-optimize each
time, though not as clearly as one might have liked.

Incidentally, credit where it is due:  His suggestion of using at least 20
more months of data to test a system using 5-minute bars is getting up into
the range where the results would begin to be interesting.

In any case, the idea of continuously re-optimizing one's parameters so
offends me that I finally decided to put it to a test, if it could be done
without wasting too much of my time by manually running the optimization and
testing with the range options on the Automatic Analysis screen.  However, I
don't see any way to accomplish this.  AFL does not provide any way to set
the range over which the optimization runs, and the optimization doesn't
happen until you click the button on the Automatic Analysis screen.  Under
the circumstances, it seems that the entire Automatic Analysis system would
have to go inside a loop, and that assumes one could simulate the button
click.

There is another issue in his systems, which I should probably put in a
message of its own, but this unpromising technique has already taken up more
time than it seems to justify, so:  Meyers often goes long (or short) when
an indicator reverses by a certain optimized amount from its lowest (or
highest) value during the previous short (or long) trade.  It is not clear
to me how one would accomplish this within AFL, either.  Though I haven't
put a lot of thought into it yet.

Owen Davies


------------------------ Yahoo! Groups Sponsor ---------------------~-->
Get A Free Psychic Reading! Your Online Answer To Life's Important Questions.
http://us.click.yahoo.com/Lj3uPC/Me7FAA/ySSFAA/GHeqlB/TM
---------------------------------------------------------------------~->

Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx 
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/