[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] What is a valid number of Back test results to Optimize?



PureBytes Links

Trading Reference Links

Hi Chris --

You can do anything you want to in your search for a good trading system.  The data period you work with during that search is the in-sample period.  The results you achieve over the in-sample period have no value in predicting what the future performance will be.  In order to estimate the future performance, you need to test the program on a set of data that follows the in-sample period and has not been used at all in the development of the system.  That data is called the out-of-sample data.  You can perform statistical tests on the out-of-sample results, but the quickest way to evaluate it is to look at the out-of-sample equity curve.

Be careful to avoid the following procedure.  Optimize in-sample, evaluate out-of-sample, modify the system based on the the out-of-sample results, retest out-of-sample.  The previously out-of-sample data period has become part of an expanded in-sample data set and a new out-of-sample test is required in order to estimate future performance.

There is a lot more to system development, testing, and validation than those two paragraphs.  I am presenting a two-day workshop in Las Vegas February 21 and 22 devoted to that subject. 
http://www.ftmonitor.com/lv08/lv08intro.html

And I have written a book devoted to that subject.
http://www.quantitativetradingsystems.com/

Thanks,
Howard

On Feb 5, 2008 4:34 AM, ChrisB <kris45mar@xxxxxxxxxxxx> wrote:

What is a valid or reasonable number of backtest results to subject to
Optimization?

For general statistics a minimum of 30 or so is needed to start getting
valid StdDevs etc.

If I run a backtest on hourly currency data over three months I get
around 16 -20 tradeable signals per currency.
This give a nice smooth plateau on 3D optimization.

If I test over two months of data I get around 10 - 12 trades

If I test over only 1 month I get only 5 or 6 trades.

These shorter time periods still give visually acceptable 3D plateaus
but I am wondering if there is enough data to be statistically significant.

I am trying to get a handle on how close I can get to current
fluctuations in the market without hitting noise. The idea being to redo
the Optimization every x time frame and shift the entry and exit
parameters to stay in the middle of the plateau.

Of course I can backtest over longer time frames, say 6 months of data,
shifting the starting date forward by one month at a time, but this
would seem to introduce more "lag" into my selection of best parameters
to trade.

Does anyone have any thoughts/references on this?

--
Regards

ChrisB


__._,_.___

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html




Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___