|
PureBytes Links
Trading Reference Links
|
"Where your test data starts determines the final test results just as
much as your system does."
It does ? Maybe for what you trade, how you trade it, the size of the
sample and/or number of trades and what you used as a yardstick to
measure how good it is, but I can't say as I ever observed this
except of course when the sample was too short, like it typically is
with walk forward optimization.
--- In amibroker@xxxxxxxxxxxxxxx, "palsanand" <palsanand@xxxx> wrote:
> Hi,
>
> Here is an interesting observation on system testing:
>
> Say you run a system test over 10,000 bars of data, then print out
a
> chart of the system's equity line. Then repeat the test, but start
> 100 bars later. Let's say two trades were included in those 100
bars,
> so they've been dropped. Now print the second equity line and
compare
> it to the first. You'd get exactly the same equity line, but 100
bars
> shorter. Right? Wrong!
>
> When I do this I get a radically different equity line on the
second
> test, i.e., they are not near-mirror images of each other. My hunch
> is that a form of the chaotician's "butterfly-effect" has arisen:
> changing any given trade's market position (long, short, flat) will
> effect in a chain reaction all the subsequent trades in complex and
> unexpected ways. Here dropping the first two trades could very well
> change the system's market position when the third trade is
> calculated, and so on.
>
> I believe this observation has profound and unfortunate
implications
> for the robustness of system testing. It's a second and more subtle
> problem that lies behind the mere curve-fitting/optimization
problem.
>
> If dropping a couple of early trades will always effect later
trades,
> then there's no truly "neutral" starting point with any test data.
> Where your test data starts determines the final test results just
as
> much as your system does.
>
> The success or failure of many different mechanical systems is
> predicated to a surprising and varying degree on the sequence of
> events just prior to the first actual trade generated by the
system.
>
> The trade setup and timing of the first trade can have a profound
> effect on the subsequent trading results. The circumstances and
> timing of entry into the first trade can sometimes make a huge
> difference in the overall trading performance.
>
> Regards,
>
> Pal
>
> --- In amibroker@xxxxxxxxxxxxxxx, "Fred" <fctonetti@xxxx> wrote:
> > That IS what I was trying to say. I suspect because equity feed
> back
> > is like looking in a rear view mirror, great for letting us know
> > where we were and how we could have adjusted the past to make it
> > better, but that's about it.
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "Dave Merrill" <dmerrill@xxxx>
> > wrote:
> > > don't think I get what you mean here fred.
> > >
> > > you can't be saying that metrics on the equity curve of a
trading
> > strategy
> > > or its parameters aren't useful, right? that's the only thing
we
> > have to
> > > judge the effectiveness of our methods and settings.
> > >
> > > so you must be saying that equity feedback isn't a useful
concept,
> > > regardless of how you measure "good" equity. do I have that
right?
> > >
> > > if so, as I've said, my experience agrees -- none of the
> indicators
> > I've
> > > tried are wonderfully profitable when auto-optimized this way.
I
> > just cannot
> > > for the life of me understand why that's the case, if backtests
> > tell us
> > > anything useful about future performance.
> > >
> > > if I've misunderstood completely, my apoligies (:-)
> > >
> > > dave
> > > Like a lot of other things that sound like they SHOULD work,
I
> > have
> > > never found metrics related to equity curve feedback to be of
> much
> > > value in the determination of system parameter values.
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx, "Dave Merrill"
<dmerrill@xxxx>
> > > wrote:
> > > > interesting as usual howard (:-). one piece I wanted to
drill
> > into
> > > a bit.
> > > >
> > > > I wonder what the effect of using performance measures that
> > > concentrate on
> > > > certain things at the expense of others actually is.
> > > >
> > > > for example, my auto-optimization stuff currently uses
simple
> > > profit per bar
> > > > to choose parameter values. my gut-level assumption was
that
> > since
> > > it was
> > > > ignoring drawdown (among other things), the resulting
systems
> > might
> > > have
> > > > higher drawdown than I was comfortable with, but that
profit
> per
> > > bar should
> > > > be as good as the trading method could produce.
> > > >
> > > > maybe that's not the case. maybe by choosing a more balanced
> > > success metric,
> > > > not only would the other factors not considered by my
> simplistic
> > > first pass
> > > > metric be improved, but profitability might be improved as
> well.
> > > >
> > > > is this something you've investigated or thought about?
> anyone
> > else?
> > > >
> > > > dave
> > > > Note ? it is perfectly valid to have different objective
> > > functions for
> > > > different purposes. For example, I might be modeling the
> > behavior
> > > of a
> > > > sector, say oil services, with the intent of trading
> individual
> > > stocks based
> > > > on what I learn. In this case, I want to identify periods
of
> > > rising prices
> > > > with careful attention to turning points, but without much
> > interest
> > > in
> > > > overall profit. On the other hand, I might be modeling
> > individual
> > > high beta
> > > > tech stocks, in which case my model includes several stop
loss
> > > techniques
> > > > and I care most about avoiding drawdowns.
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Howard
> > >
> > >
> > > Yahoo! Groups Sponsor
> > >
> > >
> > >
> > > Send BUG REPORTS to bugs@xxxx
> > > Send SUGGESTIONS to suggest@xxxx
> > > -----------------------------------------
> > > Post AmiQuote-related messages ONLY to:
amiquote@xxxxxxxxxxxxxxx
> > > (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> > > --------------------------------------------
> > > Check group FAQ at:
> > > http://groups.yahoo.com/group/amibroker/files/groupfaq.html
> > >
> > > Your use of Yahoo! Groups is subject to the Yahoo! Terms of
> > Service.
------------------------ Yahoo! Groups Sponsor ---------------------~-->
Rent DVDs Online - Over 14,500 titles.
No Late Fees & Free Shipping.
Try Netflix for FREE!
http://us.click.yahoo.com/Tq9otC/XP.FAA/3jkFAA/GHeqlB/TM
---------------------------------------------------------------------~->
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
|