|
PureBytes Links
Trading Reference Links
|
My final comments. It has been illuminating.
--- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
<quanttrader714@xxx> wrote:
>
> Good example, let's use it: "*random* trades on the S&P500 from
1997
> to 1999 will show an average expected return >zero." I agree.
Let's
> assume this period is OOS for your system and apply the test I
posted.
> To see if your entry method provides value added *when trading the
> S&P500 from 1997 to 1999*, it must do better than 95% (or whatever
> significance level you choose) of the backtests with random entries
> *on the S&P500 from 1997 to 1999*. Using simulated or any other
data
> would be comparing apples and oranges. Also, this example makes it
> easier to see why the number of trades should be around the same
as in
> the OOS backtest.
I understand all that. Basically you compare a system's performance
with a multitude of alternative random systems. The expected
(average) outcome however of those random systems does not change
due to the same underlying series (i.e. history). This is different
from comparing a system's performance within a multitude of
alternative histories. Whatever you feel comfortable with.
>
> Don't want to get into semantics either but no,
I didn't mean the bootstrap
> which is what I'd call resampling with replacement. You
> asked what I'd simulate if not market data. Well, in addition to
> random entries and exits, one could
bootstrap (???)
> a series of OOS trades
> and get a distribution of drawdows... or estimate the probability
of
> profit in n trades... or the probability of a string or n losing
> trades in a row... or...
These are all valid and useful, particularly if they
are "bootstrapped" from multiple series of trades generated by a
system that was backtested against multiple histories.
Thanks for the discussion.
PS
>
> --- In amibroker@xxxxxxxxxxxxxxx, "vlanschot" <ecbu@> wrote:
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
> > <quanttrader714@> wrote:
> > >
> > > I stressed OOS only because if you have enough trades, this
test
> > will
> > > work even with deliberately curve-fit systems.
> > >
> > > If you don't keep the number of trades approximately the same,
the
> > > comparisons won't be valid because some metrics are more
affected by
> > > the # trades than others. So you need to replicate that
aspect of
> > the
> > > OOS test but you're *randomly* doing it (drawing similarly
sized
> > > samples for an apples-to-apples comparison). So no, I don't
see any
> > > bias.
> >
> > The underlying series on which this is *randomly* done is
crucial, in
> > this case the same, e.g. 10,000 simulations of your series of
> > *random* trades on the S&P500 from 1997 to 1999 will show an
average
> > expected return >zero.
> >
> > > Monte Carlo simulations use that kind of input all the time.
> > > Which, BTW, this test is a form of.
> > You mean bootstrap.
> > >
> > > I wouldn't recommend simulating market data unless you can
somehow
> > > capture all the nuances, characteristics and
interrelationships that
> > > result from fear and greed and everything else that goes into
the
> > mix.
> >
> > Without getting too much into the semantics of things, but that
is
> > already captured in the market data? (Forget private info). If
not,
> > you simulate what?
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx, "vlanschot" <ecbu@> wrote:
> > > >
> > > > quanttrader714,
> > > >
> > > > Q for you:
> > > >
> > > > Not knowing the other settings, let's assume the system
shows
> > > > promising results over the IS-period (otherwise why bother
> > testing
> > > > further). Let's further assume that the risk/return profile
(s) of
> > the
> > > > underlying series is fairly stable over time. Is there not
> > already a
> > > > natural bias in the fact that the number of trades,
regardless of
> > IS
> > > > or OOS, is inticately linked to the aforementioned profile,
i.e.
> > the
> > > > expected return, simply because we assume "1 history"?
Therefore,
> > > > having buy-signals drawn "randomly" but benchmarked to the
number
> > of
> > > > trades in the OOS-period doesn't give you an unbiased view
of the
> > > > system versus chance?
> > > >
> > > > FAC, I'm not criticising you. I realize your suggestion is
meant
> > as a
> > > > quick test, but I would suggest to extend it via MCS:
generate
> > > > simulated price-series (stress-tested or not), thus
generating
> > > > hundreds of "alternative histories" and apply one's system
to
> > these.
> > > > All this can already be achieved in AB now, although TJ is
> > planning a
> > > > native MCS-functionality.
> > > >
> > > > PS
> > > >
> > > > (For Brian: unfortunately Capra hates the markets [see his
> > > > book "Hidden Connections"]. Tried to explain things to him.
He
> > didn't
> > > > want to listen. Suggest private e-mail if you want to know
more).
> > > >
> > > > --- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
> > > > <quanttrader714@> wrote:
> > > > >
> > > > > This is OT on psychology but a while back I believe you
were
> > asking
> > > > > about statistics and trading? Here's a very simple
statistical
> > test
> > > > > that can be run using AB alone. This simplified example
will
> > > > estimate
> > > > > the strength of a "long only" system's entries. Long and
short
> > > > > systems and exits are a bit trickier but the principle is
the
> > same.
> > > > >
> > > > > Run an *out of sample* (OOS) system backtest. Save the
> > results.
> > > > Note:
> > > > > OOS only!
> > > > >
> > > > > Add the following line of code to specify the number of
> > iterations.
> > > > > I'd run 1000 or more but as few as 100 will still give a
crude
> > > > > estimate.
> > > > >
> > > > > Iterations = Optimize("Iteration",1,1,1000,1);
> > > > >
> > > > > Replace the system's buy condition with the following code
but
> > leave
> > > > > the original settings, sell condition and stops in place.
> > Tweak the
> > > > > value in the Buy line (0.975 in this case) so the number
of
> > trades
> > > > is
> > > > > approx. the same number as in the original OOS backtest.
BTW, I
> > > > > personally wouldn't be comfortable with this procedure
unless
> > the
> > > > OOS
> > > > > backtest has at least several hundred trades.
> > > > >
> > > > > Buy= Random()>0.975;
> > > > >
> > > > > Optimize over the OOS period. Sort results by the metric
you
> > want to
> > > > > analyze. The fraction of optimized results that is
greater
> > than or
> > > > > equal to the OOS backtest metric is an estimate of the
> > probability
> > > > > that one can do as well as or better than the original
system
> > entry
> > > > by
> > > > > chance alone. Of course no matter how good the results,
> > there's no
> > > > > guarantee of future profitability. But this is an easy
way to
> > get a
> > > > > decent estimate of how much better than chance your OOS
metrics
> > are.
> > >
> >
>
Content-Description: "AVG certification"
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.13.27/517 - Release Date: 11/3/2006
|