|
PureBytes Links
Trading Reference Links
|
Good example, let's use it: "*random* trades on the S&P500 from 1997
to 1999 will show an average expected return >zero." I agree. Let's
assume this period is OOS for your system and apply the test I posted.
To see if your entry method provides value added *when trading the
S&P500 from 1997 to 1999*, it must do better than 95% (or whatever
significance level you choose) of the backtests with random entries
*on the S&P500 from 1997 to 1999*. Using simulated or any other data
would be comparing apples and oranges. Also, this example makes it
easier to see why the number of trades should be around the same as in
the OOS backtest.
Don't want to get into semantics either but no, I didn't mean the
bootstrap which is what I'd call resampling with replacement. You
asked what I'd simulate if not market data. Well, in addition to
random entries and exits, one could bootstrap a series of OOS trades
and get a distribution of drawdows... or estimate the probability of
profit in n trades... or the probability of a string or n losing
trades in a row... or...
--- In amibroker@xxxxxxxxxxxxxxx, "vlanschot" <ecbu@xxx> wrote:
>
> --- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
> <quanttrader714@> wrote:
> >
> > I stressed OOS only because if you have enough trades, this test
> will
> > work even with deliberately curve-fit systems.
> >
> > If you don't keep the number of trades approximately the same, the
> > comparisons won't be valid because some metrics are more affected by
> > the # trades than others. So you need to replicate that aspect of
> the
> > OOS test but you're *randomly* doing it (drawing similarly sized
> > samples for an apples-to-apples comparison). So no, I don't see any
> > bias.
>
> The underlying series on which this is *randomly* done is crucial, in
> this case the same, e.g. 10,000 simulations of your series of
> *random* trades on the S&P500 from 1997 to 1999 will show an average
> expected return >zero.
>
> > Monte Carlo simulations use that kind of input all the time.
> > Which, BTW, this test is a form of.
> You mean bootstrap.
> >
> > I wouldn't recommend simulating market data unless you can somehow
> > capture all the nuances, characteristics and interrelationships that
> > result from fear and greed and everything else that goes into the
> mix.
>
> Without getting too much into the semantics of things, but that is
> already captured in the market data? (Forget private info). If not,
> you simulate what?
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "vlanschot" <ecbu@> wrote:
> > >
> > > quanttrader714,
> > >
> > > Q for you:
> > >
> > > Not knowing the other settings, let's assume the system shows
> > > promising results over the IS-period (otherwise why bother
> testing
> > > further). Let's further assume that the risk/return profile(s) of
> the
> > > underlying series is fairly stable over time. Is there not
> already a
> > > natural bias in the fact that the number of trades, regardless of
> IS
> > > or OOS, is inticately linked to the aforementioned profile, i.e.
> the
> > > expected return, simply because we assume "1 history"? Therefore,
> > > having buy-signals drawn "randomly" but benchmarked to the number
> of
> > > trades in the OOS-period doesn't give you an unbiased view of the
> > > system versus chance?
> > >
> > > FAC, I'm not criticising you. I realize your suggestion is meant
> as a
> > > quick test, but I would suggest to extend it via MCS: generate
> > > simulated price-series (stress-tested or not), thus generating
> > > hundreds of "alternative histories" and apply one's system to
> these.
> > > All this can already be achieved in AB now, although TJ is
> planning a
> > > native MCS-functionality.
> > >
> > > PS
> > >
> > > (For Brian: unfortunately Capra hates the markets [see his
> > > book "Hidden Connections"]. Tried to explain things to him. He
> didn't
> > > want to listen. Suggest private e-mail if you want to know more).
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
> > > <quanttrader714@> wrote:
> > > >
> > > > This is OT on psychology but a while back I believe you were
> asking
> > > > about statistics and trading? Here's a very simple statistical
> test
> > > > that can be run using AB alone. This simplified example will
> > > estimate
> > > > the strength of a "long only" system's entries. Long and short
> > > > systems and exits are a bit trickier but the principle is the
> same.
> > > >
> > > > Run an *out of sample* (OOS) system backtest. Save the
> results.
> > > Note:
> > > > OOS only!
> > > >
> > > > Add the following line of code to specify the number of
> iterations.
> > > > I'd run 1000 or more but as few as 100 will still give a crude
> > > > estimate.
> > > >
> > > > Iterations = Optimize("Iteration",1,1,1000,1);
> > > >
> > > > Replace the system's buy condition with the following code but
> leave
> > > > the original settings, sell condition and stops in place.
> Tweak the
> > > > value in the Buy line (0.975 in this case) so the number of
> trades
> > > is
> > > > approx. the same number as in the original OOS backtest. BTW, I
> > > > personally wouldn't be comfortable with this procedure unless
> the
> > > OOS
> > > > backtest has at least several hundred trades.
> > > >
> > > > Buy= Random()>0.975;
> > > >
> > > > Optimize over the OOS period. Sort results by the metric you
> want to
> > > > analyze. The fraction of optimized results that is greater
> than or
> > > > equal to the OOS backtest metric is an estimate of the
> probability
> > > > that one can do as well as or better than the original system
> entry
> > > by
> > > > chance alone. Of course no matter how good the results,
> there's no
> > > > guarantee of future profitability. But this is an easy way to
> get a
> > > > decent estimate of how much better than chance your OOS metrics
> are.
> >
>
Content-Description: "AVG certification"
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.13.27/517 - Release Date: 11/3/2006
|