[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] What is a valid number of Back test results to Optimize?



PureBytes Links

Trading Reference Links

I would add one thing - the out of sample test should be the same type of market as that used for the in sample test.  I have seen, for example, some good neural net models that take this into account, resulting in models for up, down, and flat markets, as opposed to one model fits all.  Of course, defining up, down, and flat can be a bit subjective but performance definitely improves when one takes a shot at it.  In any case, the equity curve (bottom line) will keep one generally on the right track without serious derailments.  There was much discussion of this topic on this board several years ago with reference to the many books and articles on the subject which all tend to come down in the same place.
 
Bill
 
----- Original Message -----
From: Howard B
Sent: Tuesday, February 05, 2008 10:38 AM
Subject: Re: [amibroker] What is a valid number of Back test results to Optimize?

Hi Chris --

You can do anything you want to in your search for a good trading system.  The data period you work with during that search is the in-sample period.  The results you achieve over the in-sample period have no value in predicting what the future performance will be.  In order to estimate the future performance, you need to test the program on a set of data that follows the in-sample period and has not been used at all in the development of the system.  That data is called the out-of-sample data.  You can perform statistical tests on the out-of-sample results, but the quickest way to evaluate it is to look at the out-of-sample equity curve.

Be careful to avoid the following procedure.  Optimize in-sample, evaluate out-of-sample, modify the system based on the the out-of-sample results, retest out-of-sample.  The previously out-of-sample data period has become part of an expanded in-sample data set and a new out-of-sample test is required in order to estimate future performance.

There is a lot more to system development, testing, and validation than those two paragraphs.  I am presenting a two-day workshop in Las Vegas February 21 and 22 devoted to that subject. 
http://www.ftmonitor.com/lv08/lv08intro.html

And I have written a book devoted to that subject.
http://www.quantitativetradingsystems.com/

Thanks,
Howard

On Feb 5, 2008 4:34 AM, ChrisB <kris45mar@xxxxxxxxxxxx> wrote:

What is a valid or reasonable number of backtest results to subject to
Optimization?

For general statistics a minimum of 30 or so is needed to start getting
valid StdDevs etc.

If I run a backtest on hourly currency data over three months I get
around 16 -20 tradeable signals per currency.
This give a nice smooth plateau on 3D optimization.

If I test over two months of data I get around 10 - 12 trades

If I test over only 1 month I get only 5 or 6 trades.

These shorter time periods still give visually acceptable 3D plateaus
but I am wondering if there is enough data to be statistically significant.

I am trying to get a handle on how close I can get to current
fluctuations in the market without hitting noise. The idea being to redo
the Optimization every x time frame and shift the entry and exit
parameters to stay in the middle of the plateau.

Of course I can backtest over longer time frames, say 6 months of data,
shifting the starting date forward by one month at a time, but this
would seem to introduce more "lag" into my selection of best parameters
to trade.

Does anyone have any thoughts/references on this?

--
Regards

ChrisB



No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.5.516 / Virus Database: 269.19.19/1258 - Release Date: 2/4/2008 10:10 AM
__._,_.___

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html




Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___