|
PureBytes Links
Trading Reference Links
|
Hi,
Here is an interesting observation on system testing:
Say you run a system test over 10,000 bars of data, then print out a
chart of the system's equity line. Then repeat the test, but start
100 bars later. Let's say two trades were included in those 100 bars,
so they've been dropped. Now print the second equity line and compare
it to the first. You'd get exactly the same equity line, but 100 bars
shorter. Right? Wrong!
When I do this I get a radically different equity line on the second
test, i.e., they are not near-mirror images of each other. My hunch
is that a form of the chaotician's "butterfly-effect" has arisen:
changing any given trade's market position (long, short, flat) will
effect in a chain reaction all the subsequent trades in complex and
unexpected ways. Here dropping the first two trades could very well
change the system's market position when the third trade is
calculated, and so on.
I believe this observation has profound and unfortunate implications
for the robustness of system testing. It's a second and more subtle
problem that lies behind the mere curve-fitting/optimization problem.
If dropping a couple of early trades will always effect later trades,
then there's no truly "neutral" starting point with any test data.
Where your test data starts determines the final test results just as
much as your system does.
The success or failure of many different mechanical systems is
predicated to a surprising and varying degree on the sequence of
events just prior to the first actual trade generated by the system.
The trade setup and timing of the first trade can have a profound
effect on the subsequent trading results. The circumstances and
timing of entry into the first trade can sometimes make a huge
difference in the overall trading performance.
Regards,
Pal
--- In amibroker@xxxxxxxxxxxxxxx, "Fred" <fctonetti@xxxx> wrote:
> That IS what I was trying to say. I suspect because equity feed
back
> is like looking in a rear view mirror, great for letting us know
> where we were and how we could have adjusted the past to make it
> better, but that's about it.
>
> --- In amibroker@xxxxxxxxxxxxxxx, "Dave Merrill" <dmerrill@xxxx>
> wrote:
> > don't think I get what you mean here fred.
> >
> > you can't be saying that metrics on the equity curve of a trading
> strategy
> > or its parameters aren't useful, right? that's the only thing we
> have to
> > judge the effectiveness of our methods and settings.
> >
> > so you must be saying that equity feedback isn't a useful concept,
> > regardless of how you measure "good" equity. do I have that right?
> >
> > if so, as I've said, my experience agrees -- none of the
indicators
> I've
> > tried are wonderfully profitable when auto-optimized this way. I
> just cannot
> > for the life of me understand why that's the case, if backtests
> tell us
> > anything useful about future performance.
> >
> > if I've misunderstood completely, my apoligies (:-)
> >
> > dave
> > Like a lot of other things that sound like they SHOULD work, I
> have
> > never found metrics related to equity curve feedback to be of
much
> > value in the determination of system parameter values.
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "Dave Merrill" <dmerrill@xxxx>
> > wrote:
> > > interesting as usual howard (:-). one piece I wanted to drill
> into
> > a bit.
> > >
> > > I wonder what the effect of using performance measures that
> > concentrate on
> > > certain things at the expense of others actually is.
> > >
> > > for example, my auto-optimization stuff currently uses simple
> > profit per bar
> > > to choose parameter values. my gut-level assumption was that
> since
> > it was
> > > ignoring drawdown (among other things), the resulting systems
> might
> > have
> > > higher drawdown than I was comfortable with, but that profit
per
> > bar should
> > > be as good as the trading method could produce.
> > >
> > > maybe that's not the case. maybe by choosing a more balanced
> > success metric,
> > > not only would the other factors not considered by my
simplistic
> > first pass
> > > metric be improved, but profitability might be improved as
well.
> > >
> > > is this something you've investigated or thought about?
anyone
> else?
> > >
> > > dave
> > > Note ? it is perfectly valid to have different objective
> > functions for
> > > different purposes. For example, I might be modeling the
> behavior
> > of a
> > > sector, say oil services, with the intent of trading
individual
> > stocks based
> > > on what I learn. In this case, I want to identify periods of
> > rising prices
> > > with careful attention to turning points, but without much
> interest
> > in
> > > overall profit. On the other hand, I might be modeling
> individual
> > high beta
> > > tech stocks, in which case my model includes several stop loss
> > techniques
> > > and I care most about avoiding drawdowns.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Howard
> >
> >
> > Yahoo! Groups Sponsor
> >
> >
> >
> > Send BUG REPORTS to bugs@xxxx
> > Send SUGGESTIONS to suggest@xxxx
> > -----------------------------------------
> > Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
> > (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> > --------------------------------------------
> > Check group FAQ at:
> > http://groups.yahoo.com/group/amibroker/files/groupfaq.html
> >
> > Your use of Yahoo! Groups is subject to the Yahoo! Terms of
> Service.
------------------------ Yahoo! Groups Sponsor ---------------------~-->
Buy Ink Cartridges or Refill Kits for your HP, Epson, Canon or Lexmark
Printer at MyInks.com. Free s/h on orders $50 or more to the US & Canada.
http://www.c1tracking.com/l.asp?cid=5511
http://us.click.yahoo.com/mOAaAA/3exGAA/qnsNAA/GHeqlB/TM
---------------------------------------------------------------------~->
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
|