|
PureBytes Links
Trading Reference Links
|
Howard,
Re your statement:
>the quickest way to evaluate it is to look at the out-of-sample
>equity curve
Sorry but I don't find any value in looking at the equity curve,
since it is only one of an infinite number of journeys (assuming that
the markets will continue to trade indefinitely).
Perhaps if you had nominated some particular metrics, like slope and
standard error (K Ratio?) I might have conceded that as a reasonable
point.
The analogy is that we can't gauge much about the performance of a
vehicle from one road trip, especially if it is a short one.
However if the performance, in one road trip, isn't all over the shop
then it is more likely to be a true representation of future
performance.
Still, the majority of single equity curve metrics are pretty much
useless.
On the other hand, if we get under the bonnet and 'look' at the key
performance metrics then we can predict the outcome of any journey
with reasonable precision (barring an act of God).
IMO trading is a series of binomial events.
If we standardise the quantity and quality of the events as
ProfitFactor and combine that with variance (as the secondary
measure) then we have a measure of what powers equity curves.
If we control variance via stops (and it is quite easy to do that)
etc then trading performance is proportional to binomial PF.
A lot of the other measures are nice to have but they are only
effects and not causes.
BTW Kestner says that he went off PF as a measure of perfermance
because he found that wild outliers often biased results.
However there is more to a designing a system than metrics and IMO
Kestner overlooked some other key parameters e.g. control of the
variance of our trade time series easily overcomes the problem he
mentioned.
Using the same 'road trip' analogy; there is nothing particularly
special about future trips (OutOfSample) compared to past trips (In
Sample). The exception is optimization which isn't so much a road
trip as a heck of a lot of miles over a single training circuit,
trying out a whole range of tunings and modifications etc).
Of themselves, trip 1, 2 or 3 provide exactly the same amount of
information as each other.
The real value in successive trips lies in their relative performance.
Successive trips, with similar performance characteristics, do give
us a lot of confidence if future performance.
Also, IMO, people pay far too much attention to the minuatae of
comparitive tuning and not enough to the relative performance of
different engines, and even cars, over the same training track.
brian_z111
--- In amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@xxx> wrote:
>
> Hi Chris --
>
> You can do anything you want to in your search for a good trading
system.
> The data period you work with during that search is the in-sample
period.
> The results you achieve over the in-sample period have no value in
> predicting what the future performance will be. In order to
estimate the
> future performance, you need to test the program on a set of data
that
> follows the in-sample period and has not been used at all in the
development
> of the system. That data is called the out-of-sample data. You
can perform
> statistical tests on the out-of-sample results, but the quickest
way to
> evaluate it is to look at the out-of-sample equity curve.
>
> Be careful to avoid the following procedure. Optimize in-sample,
evaluate
> out-of-sample, modify the system based on the the out-of-sample
results,
> retest out-of-sample. The previously out-of-sample data period has
become
> part of an expanded in-sample data set and a new out-of-sample test
is
> required in order to estimate future performance.
>
> There is a lot more to system development, testing, and validation
than
> those two paragraphs. I am presenting a two-day workshop in Las
Vegas
> February 21 and 22 devoted to that subject.
> http://www.ftmonitor.com/lv08/lv08intro.html
>
> And I have written a book devoted to that subject.
> http://www.quantitativetradingsystems.com/
>
> Thanks,
> Howard
>
> On Feb 5, 2008 4:34 AM, ChrisB <kris45mar@xxx> wrote:
>
> > What is a valid or reasonable number of backtest results to
subject to
> > Optimization?
> >
> > For general statistics a minimum of 30 or so is needed to start
getting
> > valid StdDevs etc.
> >
> > If I run a backtest on hourly currency data over three months I
get
> > around 16 -20 tradeable signals per currency.
> > This give a nice smooth plateau on 3D optimization.
> >
> > If I test over two months of data I get around 10 - 12 trades
> >
> > If I test over only 1 month I get only 5 or 6 trades.
> >
> > These shorter time periods still give visually acceptable 3D
plateaus
> > but I am wondering if there is enough data to be statistically
> > significant.
> >
> > I am trying to get a handle on how close I can get to current
> > fluctuations in the market without hitting noise. The idea being
to redo
> > the Optimization every x time frame and shift the entry and exit
> > parameters to stay in the middle of the plateau.
> >
> > Of course I can backtest over longer time frames, say 6 months of
data,
> > shifting the starting date forward by one month at a time, but
this
> > would seem to introduce more "lag" into my selection of best
parameters
> > to trade.
> >
> > Does anyone have any thoughts/references on this?
> >
> > --
> > Regards
> >
> > ChrisB
> >
> >
> >
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|