|
PureBytes Links
Trading Reference Links
|
I'd prefer to handle this by doing the comparisons on a real portfolio
of tradeables that has been driven only by market forces. But then
again, that's what makes a market. Appreciate your thoughts.
--- In amibroker@xxxxxxxxxxxxxxx, "vlanschot" <ecbu@xxx> wrote:
>
> My final comments. It has been illuminating.
>
> --- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
> <quanttrader714@> wrote:
> >
> > Good example, let's use it: "*random* trades on the S&P500 from
> 1997
> > to 1999 will show an average expected return >zero." I agree.
> Let's
> > assume this period is OOS for your system and apply the test I
> posted.
> > To see if your entry method provides value added *when trading the
> > S&P500 from 1997 to 1999*, it must do better than 95% (or whatever
> > significance level you choose) of the backtests with random entries
> > *on the S&P500 from 1997 to 1999*. Using simulated or any other
> data
> > would be comparing apples and oranges. Also, this example makes it
> > easier to see why the number of trades should be around the same
> as in
> > the OOS backtest.
>
> I understand all that. Basically you compare a system's performance
> with a multitude of alternative random systems. The expected
> (average) outcome however of those random systems does not change
> due to the same underlying series (i.e. history). This is different
> from comparing a system's performance within a multitude of
> alternative histories. Whatever you feel comfortable with.
>
> >
> > Don't want to get into semantics either but no,
>
> I didn't mean the bootstrap
>
> > which is what I'd call resampling with replacement. You
> > asked what I'd simulate if not market data. Well, in addition to
> > random entries and exits, one could
>
> bootstrap (???)
>
> > a series of OOS trades
> > and get a distribution of drawdows... or estimate the probability
> of
> > profit in n trades... or the probability of a string or n losing
> > trades in a row... or...
>
> These are all valid and useful, particularly if they
> are "bootstrapped" from multiple series of trades generated by a
> system that was backtested against multiple histories.
>
> Thanks for the discussion.
>
> PS
>
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "vlanschot" <ecbu@> wrote:
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
> > > <quanttrader714@> wrote:
> > > >
> > > > I stressed OOS only because if you have enough trades, this
> test
> > > will
> > > > work even with deliberately curve-fit systems.
> > > >
> > > > If you don't keep the number of trades approximately the same,
> the
> > > > comparisons won't be valid because some metrics are more
> affected by
> > > > the # trades than others. So you need to replicate that
> aspect of
> > > the
> > > > OOS test but you're *randomly* doing it (drawing similarly
> sized
> > > > samples for an apples-to-apples comparison). So no, I don't
> see any
> > > > bias.
> > >
> > > The underlying series on which this is *randomly* done is
> crucial, in
> > > this case the same, e.g. 10,000 simulations of your series of
> > > *random* trades on the S&P500 from 1997 to 1999 will show an
> average
> > > expected return >zero.
> > >
> > > > Monte Carlo simulations use that kind of input all the time.
> > > > Which, BTW, this test is a form of.
> > > You mean bootstrap.
> > > >
> > > > I wouldn't recommend simulating market data unless you can
> somehow
> > > > capture all the nuances, characteristics and
> interrelationships that
> > > > result from fear and greed and everything else that goes into
> the
> > > mix.
> > >
> > > Without getting too much into the semantics of things, but that
> is
> > > already captured in the market data? (Forget private info). If
> not,
> > > you simulate what?
> > > >
> > > > --- In amibroker@xxxxxxxxxxxxxxx, "vlanschot" <ecbu@> wrote:
> > > > >
> > > > > quanttrader714,
> > > > >
> > > > > Q for you:
> > > > >
> > > > > Not knowing the other settings, let's assume the system
> shows
> > > > > promising results over the IS-period (otherwise why bother
> > > testing
> > > > > further). Let's further assume that the risk/return profile
> (s) of
> > > the
> > > > > underlying series is fairly stable over time. Is there not
> > > already a
> > > > > natural bias in the fact that the number of trades,
> regardless of
> > > IS
> > > > > or OOS, is inticately linked to the aforementioned profile,
> i.e.
> > > the
> > > > > expected return, simply because we assume "1 history"?
> Therefore,
> > > > > having buy-signals drawn "randomly" but benchmarked to the
> number
> > > of
> > > > > trades in the OOS-period doesn't give you an unbiased view
> of the
> > > > > system versus chance?
> > > > >
> > > > > FAC, I'm not criticising you. I realize your suggestion is
> meant
> > > as a
> > > > > quick test, but I would suggest to extend it via MCS:
> generate
> > > > > simulated price-series (stress-tested or not), thus
> generating
> > > > > hundreds of "alternative histories" and apply one's system
> to
> > > these.
> > > > > All this can already be achieved in AB now, although TJ is
> > > planning a
> > > > > native MCS-functionality.
> > > > >
> > > > > PS
> > > > >
> > > > > (For Brian: unfortunately Capra hates the markets [see his
> > > > > book "Hidden Connections"]. Tried to explain things to him.
> He
> > > didn't
> > > > > want to listen. Suggest private e-mail if you want to know
> more).
> > > > >
> > > > > --- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
> > > > > <quanttrader714@> wrote:
> > > > > >
> > > > > > This is OT on psychology but a while back I believe you
> were
> > > asking
> > > > > > about statistics and trading? Here's a very simple
> statistical
> > > test
> > > > > > that can be run using AB alone. This simplified example
> will
> > > > > estimate
> > > > > > the strength of a "long only" system's entries. Long and
> short
> > > > > > systems and exits are a bit trickier but the principle is
> the
> > > same.
> > > > > >
> > > > > > Run an *out of sample* (OOS) system backtest. Save the
> > > results.
> > > > > Note:
> > > > > > OOS only!
> > > > > >
> > > > > > Add the following line of code to specify the number of
> > > iterations.
> > > > > > I'd run 1000 or more but as few as 100 will still give a
> crude
> > > > > > estimate.
> > > > > >
> > > > > > Iterations = Optimize("Iteration",1,1,1000,1);
> > > > > >
> > > > > > Replace the system's buy condition with the following code
> but
> > > leave
> > > > > > the original settings, sell condition and stops in place.
> > > Tweak the
> > > > > > value in the Buy line (0.975 in this case) so the number
> of
> > > trades
> > > > > is
> > > > > > approx. the same number as in the original OOS backtest.
> BTW, I
> > > > > > personally wouldn't be comfortable with this procedure
> unless
> > > the
> > > > > OOS
> > > > > > backtest has at least several hundred trades.
> > > > > >
> > > > > > Buy= Random()>0.975;
> > > > > >
> > > > > > Optimize over the OOS period. Sort results by the metric
> you
> > > want to
> > > > > > analyze. The fraction of optimized results that is
> greater
> > > than or
> > > > > > equal to the OOS backtest metric is an estimate of the
> > > probability
> > > > > > that one can do as well as or better than the original
> system
> > > entry
> > > > > by
> > > > > > chance alone. Of course no matter how good the results,
> > > there's no
> > > > > > guarantee of future profitability. But this is an easy
> way to
> > > get a
> > > > > > decent estimate of how much better than chance your OOS
> metrics
> > > are.
> > > >
> > >
> >
>
Content-Description: "AVG certification"
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.13.27/517 - Release Date: 11/3/2006
|