|
PureBytes Links
Trading Reference Links
|
--- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
<quanttrader714@xxx> wrote:
>
> I stressed OOS only because if you have enough trades, this test
will
> work even with deliberately curve-fit systems.
>
> If you don't keep the number of trades approximately the same, the
> comparisons won't be valid because some metrics are more affected by
> the # trades than others. So you need to replicate that aspect of
the
> OOS test but you're *randomly* doing it (drawing similarly sized
> samples for an apples-to-apples comparison). So no, I don't see any
> bias.
The underlying series on which this is *randomly* done is crucial, in
this case the same, e.g. 10,000 simulations of your series of
*random* trades on the S&P500 from 1997 to 1999 will show an average
expected return >zero.
> Monte Carlo simulations use that kind of input all the time.
> Which, BTW, this test is a form of.
You mean bootstrap.
>
> I wouldn't recommend simulating market data unless you can somehow
> capture all the nuances, characteristics and interrelationships that
> result from fear and greed and everything else that goes into the
mix.
Without getting too much into the semantics of things, but that is
already captured in the market data? (Forget private info). If not,
you simulate what?
>
> --- In amibroker@xxxxxxxxxxxxxxx, "vlanschot" <ecbu@> wrote:
> >
> > quanttrader714,
> >
> > Q for you:
> >
> > Not knowing the other settings, let's assume the system shows
> > promising results over the IS-period (otherwise why bother
testing
> > further). Let's further assume that the risk/return profile(s) of
the
> > underlying series is fairly stable over time. Is there not
already a
> > natural bias in the fact that the number of trades, regardless of
IS
> > or OOS, is inticately linked to the aforementioned profile, i.e.
the
> > expected return, simply because we assume "1 history"? Therefore,
> > having buy-signals drawn "randomly" but benchmarked to the number
of
> > trades in the OOS-period doesn't give you an unbiased view of the
> > system versus chance?
> >
> > FAC, I'm not criticising you. I realize your suggestion is meant
as a
> > quick test, but I would suggest to extend it via MCS: generate
> > simulated price-series (stress-tested or not), thus generating
> > hundreds of "alternative histories" and apply one's system to
these.
> > All this can already be achieved in AB now, although TJ is
planning a
> > native MCS-functionality.
> >
> > PS
> >
> > (For Brian: unfortunately Capra hates the markets [see his
> > book "Hidden Connections"]. Tried to explain things to him. He
didn't
> > want to listen. Suggest private e-mail if you want to know more).
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
> > <quanttrader714@> wrote:
> > >
> > > This is OT on psychology but a while back I believe you were
asking
> > > about statistics and trading? Here's a very simple statistical
test
> > > that can be run using AB alone. This simplified example will
> > estimate
> > > the strength of a "long only" system's entries. Long and short
> > > systems and exits are a bit trickier but the principle is the
same.
> > >
> > > Run an *out of sample* (OOS) system backtest. Save the
results.
> > Note:
> > > OOS only!
> > >
> > > Add the following line of code to specify the number of
iterations.
> > > I'd run 1000 or more but as few as 100 will still give a crude
> > > estimate.
> > >
> > > Iterations = Optimize("Iteration",1,1,1000,1);
> > >
> > > Replace the system's buy condition with the following code but
leave
> > > the original settings, sell condition and stops in place.
Tweak the
> > > value in the Buy line (0.975 in this case) so the number of
trades
> > is
> > > approx. the same number as in the original OOS backtest. BTW, I
> > > personally wouldn't be comfortable with this procedure unless
the
> > OOS
> > > backtest has at least several hundred trades.
> > >
> > > Buy= Random()>0.975;
> > >
> > > Optimize over the OOS period. Sort results by the metric you
want to
> > > analyze. The fraction of optimized results that is greater
than or
> > > equal to the OOS backtest metric is an estimate of the
probability
> > > that one can do as well as or better than the original system
entry
> > by
> > > chance alone. Of course no matter how good the results,
there's no
> > > guarantee of future profitability. But this is an easy way to
get a
> > > decent estimate of how much better than chance your OOS metrics
are.
>
Content-Description: "AVG certification"
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.13.27/517 - Release Date: 11/3/2006
|