|
I would add one thing - the out of sample test should
be the same type of market as that used for the in sample test. I have
seen, for example, some good neural net models that take this into account,
resulting in models for up, down, and flat markets, as opposed to one model fits
all. Of course, defining up, down, and flat can be a bit subjective
but performance definitely improves when one takes a shot at it. In any
case, the equity curve (bottom line) will keep one generally on the right track
without serious derailments. There was much discussion of this topic on
this board several years ago with reference to the many books and articles on
the subject which all tend to come down in the same place.
Bill
----- Original Message -----
Sent: Tuesday, February 05, 2008 10:38
AM
Subject: Re: [amibroker] What is a valid
number of Back test results to Optimize?
Hi Chris --
You can do anything you want to in your
search for a good trading system. The data period you work with during
that search is the in-sample period. The results you achieve over the
in-sample period have no value in predicting what the future performance will
be. In order to estimate the future performance, you need to test the
program on a set of data that follows the in-sample period and has not been
used at all in the development of the system. That data is called the
out-of-sample data. You can perform statistical tests on the
out-of-sample results, but the quickest way to evaluate it is to look at the
out-of-sample equity curve.
Be careful to avoid the following
procedure. Optimize in-sample, evaluate out-of-sample, modify the system
based on the the out-of-sample results, retest out-of-sample. The
previously out-of-sample data period has become part of an expanded in-sample
data set and a new out-of-sample test is required in order to estimate future
performance.
There is a lot more to system development, testing, and
validation than those two paragraphs. I am presenting a two-day workshop
in Las Vegas February 21 and 22 devoted to that subject.
http://www.ftmonitor.com/lv08/lv08intro.html
And
I have written a book devoted to that subject.
http://www.quantitativetradingsystems.com/
Thanks,
Howard
On Feb 5, 2008 4:34 AM, ChrisB < kris45mar@xxxxxxxxxxxx> wrote:
What is a valid or reasonable number of backtest results to subject to
Optimization?
For general statistics a minimum of 30 or so is
needed to start getting
valid StdDevs etc.
If I run a backtest on
hourly currency data over three months I get
around 16 -20 tradeable
signals per currency.
This give a nice smooth plateau on 3D
optimization.
If I test over two months of data I get around 10 - 12
trades
If I test over only 1 month I get only 5 or 6
trades.
These shorter time periods still give visually acceptable 3D
plateaus
but I am wondering if there is enough data to be statistically
significant.
I am trying to get a handle on how close I can get to
current
fluctuations in the market without hitting noise. The idea being
to redo
the Optimization every x time frame and shift the entry and exit
parameters to stay in the middle of the plateau.
Of course I can
backtest over longer time frames, say 6 months of data,
shifting the
starting date forward by one month at a time, but this
would seem to
introduce more "lag" into my selection of best parameters
to
trade.
Does anyone have any thoughts/references on this?
--
Regards
ChrisB
No virus found in this incoming message.
Checked by AVG Free
Edition.
Version: 7.5.516 / Virus Database: 269.19.19/1258 - Release Date:
2/4/2008 10:10 AM
__._,_.___
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
![]()
__,_._,___
|