|
PureBytes Links
Trading Reference Links
|
Appreciate your input Howard.
Halfway through my first cursory read of QTS and at this stage I guess I
have more Q's than A's.
To be sure if I was in the area would love to attend the workshop (bit
far from Oz at the moment)
I guess there is no straightforward answer to my initial question: it
needs to be seen in context of what I am doing.
With a maximum hold period in any trade (win or lose) of 35 (hourly)
bars this is just over one day in Fx.
The thought occurred to try and follow the flow of the market more
closely by re-backtesting and optimising over _shorter_ periods , say
every 2x periods, where x is my maximum hold time.
Of interest is that as I get to Sub-prime meltdown time in Aug 2007 the
backtest data for Long GBPJPY starts to degrade more slowly than I would
have thought. Not that one would be long during this period, but looking
at the BT and Optimize results is interesting: There are less Long
trades after Aug and the results are still poorer, but still OK. Only
lately do I get a "do not trade this" message from the BT results. This
implies to me that fundamentals aside, waiting for three months before
retesting and re-setting my entry and exit criteria would be a bit long.
Totally get the concept of In and Out of sample testing but the
practical application still confuses me and that led to my initial question.
Let's say I test Jan to end March 2007 as In Sample. Over this time
frame the market changes and so do the BT results.
I then do OOS out of sample, say April to Jun, without real trading. The
market morphs more during this time.
Over each 3 month period I can get acceptable results to take forward
into the next 3 month period, but each time I wait for the next walk
forward test period to end, the market conditions change.
My paradox at the moment is: the more OOS walk forwards I do, let's say
hypothetically for one whole year, the further away I get from actual
trading, and the further away I get from the initial data set (tradeable
market conditions).
To re-phrase my initial question (and to try and make this clearer for
myself at least) if I start trading with the Jan to March data at the
beginning of April, should I wait until the end of my next three month
time frame, (i.e.until the end of June) to try and stay in the middle of
the expectancy curve?
In theory I could
a. simply start trading at the beginning of April and then at the end of
April do another Optimization, but looking back three months into Feb
March April.
b. or only look back for one month, based on my average hold time of
less than one day.
The problem with (b) seems to be that I get less backtest samples to
Optimize, and my initial question applies: Is there a gut feel way, or
a way to quantify the validity or not of only 10 or 20 backtest results?
The problem with (a) is that as you imply I am curve fitting and not
testing OOS adequately.
This then begs the next question: at what point do you start trading
your system?
Would it be a fair hypothesis to make that given a positive expectancy
and a broad plateau of tradeability based on the 3D Optimization output,
one should just start trading? My job then as a trader is to pick when
the results change sufficiently, to alter my entries and exits? The
sooner I can detect this and make the necessary adjustments the better?
Hate it when what should be a numbers game becomes a philosophical
musing, but guess these questions need an answer, to allow goal
orientated system design.
Do appreciate the time you made for your response.
Regards
ChrisB
Howard B wrote:
>
> Hi Chris --
>
> You can do anything you want to in your search for a good trading
> system. The data period you work with during that search is the
> in-sample period. The results you achieve over the in-sample period
> have no value in predicting what the future performance will be. In
> order to estimate the future performance, you need to test the program
> on a set of data that follows the in-sample period and has not been
> used at all in the development of the system. That data is called the
> out-of-sample data. You can perform statistical tests on the
> out-of-sample results, but the quickest way to evaluate it is to look
> at the out-of-sample equity curve.
>
> Be careful to avoid the following procedure. Optimize in-sample,
> evaluate out-of-sample, modify the system based on the the
> out-of-sample results, retest out-of-sample. The previously
> out-of-sample data period has become part of an expanded in-sample
> data set and a new out-of-sample test is required in order to estimate
> future performance.
>
> There is a lot more to system development, testing, and validation
> than those two paragraphs. I am presenting a two-day workshop in Las
> Vegas February 21 and 22 devoted to that subject.
> http://www.ftmonito r.com/lv08/ lv08intro. html
> <http://www.ftmonitor.com/lv08/lv08intro.html>
>
> And I have written a book devoted to that subject.
> http://www.quantita tivetradingsyste ms.com/
> <http://www.quantitativetradingsystems.com/>
>
> Thanks,
> Howard
>
> On Feb 5, 2008 4:34 AM, ChrisB <kris45mar@xxxxxx net.au
> <mailto:kris45mar@xxxxxxxxxxxx>> wrote:
>
> What is a valid or reasonable number of backtest results to
> subject to
> Optimization?
>
> For general statistics a minimum of 30 or so is needed to start
> getting
> valid StdDevs etc.
>
> If I run a backtest on hourly currency data over three months I get
> around 16 -20 tradeable signals per currency.
> This give a nice smooth plateau on 3D optimization.
>
> If I test over two months of data I get around 10 - 12 trades
>
> If I test over only 1 month I get only 5 or 6 trades.
>
> These shorter time periods still give visually acceptable 3D plateaus
> but I am wondering if there is enough data to be statistically
> significant.
>
> I am trying to get a handle on how close I can get to current
> fluctuations in the market without hitting noise. The idea being
> to redo
> the Optimization every x time frame and shift the entry and exit
> parameters to stay in the middle of the plateau.
>
> Of course I can backtest over longer time frames, say 6 months of
> data,
> shifting the starting date forward by one month at a time, but this
> would seem to introduce more "lag" into my selection of best
> parameters
> to trade.
>
> Does anyone have any thoughts/references on this?
>
> --
> Regards
>
> ChrisB
>
>
>
--
Regards
ChrisB
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|