|
PureBytes Links
Trading Reference Links
|
<SPAN
class=160272911-17102003>great post howard, interesting, thanks. couple
questions, of course (:-), sorry for the length.
<SPAN
class=160272911-17102003>
<SPAN
class=160272911-17102003>- what are your thoughts on what measures to use to
evaluate past trading performance while optimizing? I figured that percent
growth per bar should optimize for simple profit. it ignores risk, drawdown,
etc, so it might not be the wisest metric in a larger sense, but I'm wondering
if it's less effective, even on a purely profit level, than some other approach.
<SPAN
class=160272911-17102003>
<SPAN
class=160272911-17102003>- keeping the in-sample test period short doesn't mean
that conditions won't change during that time, unless you can somehow align
optimization periods to coincide with the points at which conditions do change.
besides, isn't coping with changing market conditions the main thing a trading
strategy needs to be able to do?
<SPAN
class=160272911-17102003>
<SPAN
class=160272911-17102003>- I wouldn't go so far as to say that as soon as we've
looked at system performance on a given piece of out of sample data once, it's
no longer useful. you just can't tweak your model with optimizations
on data that's supposedly out of sample. you're right that you need to be
very clear about what's in sample data and what's out, but that's not the
same as saying that data is "ruined" for further testing once we've seen it at
all.
<SPAN
class=160272911-17102003>
<SPAN
class=160272911-17102003>- I understand that if many people use a specific
trading system, each person's profits subtract from the others' gains. I
understand also the general idea that a profitable trading method represents
inefficiencies that over time the market will reduce. but let's say you just
trade your method without publicizing it, and your trades are a small enough
fraction of overall volume that their pattern doesn't draw much conscious
attention. by what mechanism does the market as a system "realize" what you're
doing and compensate, in a way that your method's profitability disappears? on
an intuitive level I see that that should happen, I just don't get how. for
instance, if my trades are based on some complex formula with many inputs, it
seems unlikely that it will get reverse engineered and intentionally copied,
especially if I'm a pretty much below-the-radar small trader. so how then does
my individual trading cause the market's behavior to shift?
<SPAN
class=160272911-17102003>
<SPAN
class=160272911-17102003>- do you have any thoughts on the idea of constant
automatic optimization, mechanically choosing system parameters every day based
on past performance? if past performance is any guide to future results, I can't
see any logical reason why this shouldn't be the best way to go. if doing
this doesn't produce profitable results, which so far in my experience it mostly
doesn't, why doesn't that mean that past performance *doesn't* predict future
results, invalidating the whole notion of backtesting and trading analysis? I
don't really buy that conclusion, I'm looking for a hole in the
logic.
<SPAN
class=160272911-17102003>
<SPAN
class=160272911-17102003>thanks again for stopping by here, I hope we'll see you
around,
<SPAN
class=160272911-17102003>
<SPAN
class=160272911-17102003>dave
<SPAN
class=160272911-17102003>
<BLOCKQUOTE
>In
my opinion, anything we do in development of trading systems involves
asearch for a pattern than precedes a profitable trading
opportunity. Anytime we examine the results of alternative systems,
we are involved insearching; and when we select the most promising of
those alternatives, weare optimizing. Only a system based on truly
random entries and exits wouldnot be the result some optimization.
So the question of "should weoptimize?" is moot -- we have no choice but
to optimize. Consequently, weshould be aware of our optimization
techniques.Chuck referred to an optimization technique recommendation
I made to thecompany we both worked for in Denver a few years ago.
This is a shortdescription of it.The company is a Commodity
Trading Advisor which traded futures, notindividual stocks, but the
procedures are equally valid for both.When I joined the company, they
were using very long data series whendeveloping their models. They
used a technique sometimes called folding orjackknifing, where the data
was divided into several periods -- say ten.The modeling process made ten
passes. During each pass, one period was heldback to be used as
out-of-sample data, the other nine were used to selectthe best parameter
values. After all ten passes, the results were gatheredtogether and
the parameter values that scored best overall were chosen.There are
several problems with this method. One is the difficulty with
the"ramp up" period at the start of each segment, another is that it is
notvalid to use older data for out-of-sample testing than was used
forin-sample development, and another is that the data series were too
long.Chuck and I and others had many interesting discussions about how
long thein-sample data should be. My background is strong in
both the theory and the practice of modeling andsimulation, and includes a
great deal of experience with analysis offinancial time series. I
proposed the following method, which I continue tobelieve is
valid.First, before any modeling begins. Using judgment of
management andcomparison of trading profiles of many trading runs (real,
simulated, orimagined), pick an objective function by which the "goodness"
of a tradingsystem will be measured. This is important, it is a
personal or corporatejudgment, and it should not be subject to
optimization. Divide each data series into a sequence of
in-sample and out-of-sampleperiods. The length of the out-of-sample
period is the "reoptimization"period. Say there are about ten years
of historical data available(1/1/1993 through 1/1/2003. Set the
in-sample period to two years and theout-of-sample period to one
year. Run the following sequence: Search /optimize using 1993
and 1994; pick the "best" model for 1993-1994; forwardtest this model for
1995 and save the results; step forward onereoptimization period and
repeat until all the full in-sample periods havebeen used. The final
optimization will have been 2001 and 2002, with noout-of-sample data to
test. Ignore all in-sample results!! Examine theconcatenated
out-of-sample equity curve. If it is acceptable, you have
someconfidence that the parameters select by the final optimization (2001
and2002) will be profitable for 2003. No guarantees -- only some
confidence.How did I pick two years for in-sample and one year for
out-of-sample? Thatwas just an example. The method is to set
up an automated search where thelength of the in-sample period and the
length of the out-of-sample period --the reoptimization period -- are
variables, and then search through thatspace. Trading
systems work because they identify inefficiencies in markets.
Everyprofitable trade reduces the inefficiency until, finally, the trading
systemcannot overcome the frictional forces of commission and
slippage. This isthe same phenomenon that physicists talk about as
entropy.My feeling -- and it may be different than Chuck's -- is that
the market isnot only non-stationary, but that the probability that it
will return to aprevious state is near zero. Being
non-stationary means that market conditions change with respect to
ourtrading systems. If I am modeling a physical process, such as a
chemicalreaction, I can count on a predictable modelable output for a
given set ofinputs. If I am modeling a financial time series, the
output following agiven set of inputs changes over time. If a market
were stationary withrespect to an RSI oscillator system, I could always
buy a rise of the RSIthrough the 20 percent line, to use a very simplistic
example. I feel that the introduction of microcomputers, trading
system developmentsoftware, inexpensive individual brokerage accounts, and
discussion groupssuch as this one have permanently changed the realm of
trading. One,everyone who is interested can afford to buy a
computer, run AmiBroker, anddesign and test trading systems. Two, if
someone develops a profitablesystem and trades it, the profits it takes
reduce the potential profitsavailable to anyone else who trades it.
Consequently, the characteristicsof the market change in a way that moves
the market away from that modeluntil that trading system is no longer
profitable enough to overcomecommission and slippage. Three, a new
person beginning to study tradingsystem development typically tests a lot
of old systems. If one is found tobe profitable and they start
trading it, the market moves back to beingefficient. Consequently,
trading systems that used to work, but no longerwork, are very unlikely to
ever work again.So, I feel that the in-sample period should be short
so that the marketconditions do not change much over that period.
That is, I am looking for adata series that is stationary relative to my
model. The stationaryrelationship must extend beyond the in-sample
period far enough that themodel will be profitable when used for trading
in the out-of-sample data.The length of the extension determines the
reoptimization period. It couldbe years, months, or even one
day. Note that the holding period of atypical trade is very much
related to the length of both the in-sample andout-of-sample
periods. The typical trade should be much shorter than thein-sample
period and somewhat shorter than the out-of-sample period.The
important point in all this is that the only results being analyzed arethe
concatenated out-of-sample trades.As with all model development, every
time I look at the out-of-sampleresults in any way, I reduce the
probability that future trading resultswill be profitable. That
means that I should not perform thousands of testsof model parameters,
in-sample periods, and out-of-sample periods, on thesame data series and
then pick the best model base on my examination ofthousands of
out-of-sample results. In effect, I will have just convertedall
those out-of-sample results into in-sample data for another step in
thedevelopment. That is legitimate, just be aware of what is
happening.Thanks for listening,HowardSend
BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to
suggest@xxxxxxxxxxxxx-----------------------------------------Post
AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: <A
href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check
group FAQ at: <A
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the <A
href="">Yahoo! Terms of Service.
Yahoo! Groups Sponsor
ADVERTISEMENT
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
|