|
PureBytes Links
Trading Reference Links
|
Very provocative post, Howard. Thank you for sharing with the group. I
have two questions:
1. You stated "...it
is not valid to use older data for out-of-sample testing than was used for
in-sample development". Why is that?
Why is it invalid to use, say, an OOS period of 1998-1999 when the IS period was
2000-2002?
2. In your last paragraph, you wrote that you should not conduct numerous
tests of systems on a host of in-sample and out-of-sample periods and then pick
the best model based on your examination of the OOS periods. Why would you be
converting all those OOS data into IS results? You are only using the OOS
periods to confirm that your system behaves as its optimized parameters say it
should. You have never touched the OOS data for purposes of parameter
optimization. So, why do you say that using the OOS periods to confirm your
system performance nullifies your system?
Thanks again for stimulating thoughts.
Al Venosa
<BLOCKQUOTE
>
----- Original Message -----
<DIV
>From:
Howard
Bandy
To: <A title=amibroker@xxxxxxxxxxxxxxx
href="">amibroker@xxxxxxxxxxxxxxx
Sent: Thursday, October 16, 2003 2:19
PM
Subject: [amibroker] Optimization --
again
Greetings --In my opinion, anything we do in
development of trading systems involves asearch for a pattern than
precedes a profitable trading opportunity. Anytime we examine the
results of alternative systems, we are involved insearching; and when we
select the most promising of those alternatives, weare optimizing.
Only a system based on truly random entries and exits wouldnot be the
result some optimization. So the question of "should weoptimize?" is
moot -- we have no choice but to optimize. Consequently, weshould be
aware of our optimization techniques.Chuck referred to an optimization
technique recommendation I made to thecompany we both worked for in Denver
a few years ago. This is a shortdescription of it.The
company is a Commodity Trading Advisor which traded futures, notindividual
stocks, but the procedures are equally valid for both.When I joined
the company, they were using very long data series whendeveloping their
models. They used a technique sometimes called folding
orjackknifing, where the data was divided into several periods -- say
ten.The modeling process made ten passes. During each pass, one
period was heldback to be used as out-of-sample data, the other nine were
used to selectthe best parameter values. After all ten passes, the
results were gatheredtogether and the parameter values that scored best
overall were chosen.There are several problems with this method. One
is the difficulty with the"ramp up" period at the start of each segment,
another is that it is notvalid to use older data for out-of-sample testing
than was used forin-sample development, and another is that the data
series were too long.Chuck and I and others had many interesting
discussions about how long thein-sample data should be. My
background is strong in both the theory and the practice of modeling
andsimulation, and includes a great deal of experience with analysis
offinancial time series. I proposed the following method, which I
continue tobelieve is valid.First, before any modeling
begins. Using judgment of management andcomparison of trading
profiles of many trading runs (real, simulated, orimagined), pick an
objective function by which the "goodness" of a tradingsystem will be
measured. This is important, it is a personal or corporatejudgment,
and it should not be subject to optimization. Divide each data
series into a sequence of in-sample and out-of-sampleperiods. The
length of the out-of-sample period is the "reoptimization"period.
Say there are about ten years of historical data available(1/1/1993
through 1/1/2003. Set the in-sample period to two years and
theout-of-sample period to one year. Run the following
sequence: Search /optimize using 1993 and 1994; pick the "best"
model for 1993-1994; forwardtest this model for 1995 and save the results;
step forward onereoptimization period and repeat until all the full
in-sample periods havebeen used. The final optimization will have
been 2001 and 2002, with noout-of-sample data to test. Ignore all
in-sample results!! Examine theconcatenated out-of-sample equity
curve. If it is acceptable, you have someconfidence that the
parameters select by the final optimization (2001 and2002) will be
profitable for 2003. No guarantees -- only some confidence.How
did I pick two years for in-sample and one year for out-of-sample?
Thatwas just an example. The method is to set up an automated search
where thelength of the in-sample period and the length of the
out-of-sample period --the reoptimization period -- are variables, and
then search through thatspace. Trading systems work because
they identify inefficiencies in markets. Everyprofitable trade
reduces the inefficiency until, finally, the trading systemcannot overcome
the frictional forces of commission and slippage. This isthe same
phenomenon that physicists talk about as entropy.My feeling -- and it
may be different than Chuck's -- is that the market isnot only
non-stationary, but that the probability that it will return to aprevious
state is near zero. Being non-stationary means that market
conditions change with respect to ourtrading systems. If I am
modeling a physical process, such as a chemicalreaction, I can count on a
predictable modelable output for a given set ofinputs. If I am
modeling a financial time series, the output following agiven set of
inputs changes over time. If a market were stationary withrespect to
an RSI oscillator system, I could always buy a rise of the RSIthrough the
20 percent line, to use a very simplistic example. I feel that the
introduction of microcomputers, trading system developmentsoftware,
inexpensive individual brokerage accounts, and discussion groupssuch as
this one have permanently changed the realm of trading. One,everyone
who is interested can afford to buy a computer, run AmiBroker, anddesign
and test trading systems. Two, if someone develops a
profitablesystem and trades it, the profits it takes reduce the potential
profitsavailable to anyone else who trades it. Consequently, the
characteristicsof the market change in a way that moves the market away
from that modeluntil that trading system is no longer profitable enough to
overcomecommission and slippage. Three, a new person beginning to
study tradingsystem development typically tests a lot of old
systems. If one is found tobe profitable and they start trading it,
the market moves back to beingefficient. Consequently, trading
systems that used to work, but no longerwork, are very unlikely to ever
work again.So, I feel that the in-sample period should be short so
that the marketconditions do not change much over that period. That
is, I am looking for adata series that is stationary relative to my
model. The stationaryrelationship must extend beyond the in-sample
period far enough that themodel will be profitable when used for trading
in the out-of-sample data.The length of the extension determines the
reoptimization period. It couldbe years, months, or even one
day. Note that the holding period of atypical trade is very much
related to the length of both the in-sample andout-of-sample
periods. The typical trade should be much shorter than thein-sample
period and somewhat shorter than the out-of-sample period.The
important point in all this is that the only results being analyzed arethe
concatenated out-of-sample trades.As with all model development, every
time I look at the out-of-sampleresults in any way, I reduce the
probability that future trading resultswill be profitable. That
means that I should not perform thousands of testsof model parameters,
in-sample periods, and out-of-sample periods, on thesame data series and
then pick the best model base on my examination ofthousands of
out-of-sample results. In effect, I will have just convertedall
those out-of-sample results into in-sample data for another step in
thedevelopment. That is legitimate, just be aware of what is
happening.Thanks for listening,Howard
Send BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to
suggest@xxxxxxxxxxxxx-----------------------------------------Post
AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: <A
href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check
group FAQ at: <A
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the <A
href="">Yahoo! Terms of Service.
<BLOCKQUOTE
><FONT
face="Courier New">---Outgoing mail is certified Virus
Free.Checked by AVG anti-virus system (<A
href="">http://www.grisoft.com).Version: 6.0.525
/ Virus Database: 322 - Release Date:
10/9/2003
Yahoo! Groups Sponsor
ADVERTISEMENT
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
|