|
PureBytes Links
Trading Reference Links
|
I stressed OOS only because if you have enough trades, this test will
work even with deliberately curve-fit systems.
If you don't keep the number of trades approximately the same, the
comparisons won't be valid because some metrics are more affected by
the # trades than others. So you need to replicate that aspect of the
OOS test but you're *randomly* doing it (drawing similarly sized
samples for an apples-to-apples comparison). So no, I don't see any
bias. Monte Carlo simulations use that kind of input all the time.
Which, BTW, this test is a form of.
I wouldn't recommend simulating market data unless you can somehow
capture all the nuances, characteristics and interrelationships that
result from fear and greed and everything else that goes into the mix.
--- In amibroker@xxxxxxxxxxxxxxx, "vlanschot" <ecbu@xxx> wrote:
>
> quanttrader714,
>
> Q for you:
>
> Not knowing the other settings, let's assume the system shows
> promising results over the IS-period (otherwise why bother testing
> further). Let's further assume that the risk/return profile(s) of the
> underlying series is fairly stable over time. Is there not already a
> natural bias in the fact that the number of trades, regardless of IS
> or OOS, is inticately linked to the aforementioned profile, i.e. the
> expected return, simply because we assume "1 history"? Therefore,
> having buy-signals drawn "randomly" but benchmarked to the number of
> trades in the OOS-period doesn't give you an unbiased view of the
> system versus chance?
>
> FAC, I'm not criticising you. I realize your suggestion is meant as a
> quick test, but I would suggest to extend it via MCS: generate
> simulated price-series (stress-tested or not), thus generating
> hundreds of "alternative histories" and apply one's system to these.
> All this can already be achieved in AB now, although TJ is planning a
> native MCS-functionality.
>
> PS
>
> (For Brian: unfortunately Capra hates the markets [see his
> book "Hidden Connections"]. Tried to explain things to him. He didn't
> want to listen. Suggest private e-mail if you want to know more).
>
> --- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
> <quanttrader714@> wrote:
> >
> > This is OT on psychology but a while back I believe you were asking
> > about statistics and trading? Here's a very simple statistical test
> > that can be run using AB alone. This simplified example will
> estimate
> > the strength of a "long only" system's entries. Long and short
> > systems and exits are a bit trickier but the principle is the same.
> >
> > Run an *out of sample* (OOS) system backtest. Save the results.
> Note:
> > OOS only!
> >
> > Add the following line of code to specify the number of iterations.
> > I'd run 1000 or more but as few as 100 will still give a crude
> > estimate.
> >
> > Iterations = Optimize("Iteration",1,1,1000,1);
> >
> > Replace the system's buy condition with the following code but leave
> > the original settings, sell condition and stops in place. Tweak the
> > value in the Buy line (0.975 in this case) so the number of trades
> is
> > approx. the same number as in the original OOS backtest. BTW, I
> > personally wouldn't be comfortable with this procedure unless the
> OOS
> > backtest has at least several hundred trades.
> >
> > Buy= Random()>0.975;
> >
> > Optimize over the OOS period. Sort results by the metric you want to
> > analyze. The fraction of optimized results that is greater than or
> > equal to the OOS backtest metric is an estimate of the probability
> > that one can do as well as or better than the original system entry
> by
> > chance alone. Of course no matter how good the results, there's no
> > guarantee of future profitability. But this is an easy way to get a
> > decent estimate of how much better than chance your OOS metrics are.
Content-Description: "AVG certification"
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.13.27/517 - Release Date: 11/3/2006
|