|
PureBytes Links
Trading Reference Links
|
Hi Dingo –
It’s not quite that bad.
My last paragraph is a warning for the
common technique that many people try – looking at out-of-sample results
so much that they become in-sample data. That is OK, if there is another
set of data that will be used for further testing. Sometimes model
development is done using three sets of data – learning, testing, and
validation. The learning data is extensively searched and used to select
parameter values. The testing data is used to determine when to stop
searching and use the results found to be best so far. For each set of
parameters, a run is made on the learning data, and the value of the objective
function is noted. Periodically, the model using the best parameters is
run against the test data. When the performance over the test data falls
off, that tells the optimizer to stop and return the parameters that worked
best over the test data. One (or a Very few) additional run is made
against the validation data. If results are acceptable, the model is
used. If the validation data produces poor results, do not use the
model. Rather, go back to the model design stage and come up with
something new to try.
If the model is able to recognize
profitable situations, and the parameters are stable, a wide range of
parameters will be profitable.
One of my points in the posting is that
the walk-forward testing with automatic selection of parameter values should be
taken all together as a process. If the results are good, then the
process has validity and there is a reasonable expectation that trading the
model and parameter set resulting from the last optimization will be
profitable.
Thanks,
Howard
-----Original Message-----
From: dingo [mailto:dingo@xxxxxxxxxx]
Sent: Thursday,
October 16, 2003 <span
>12:02 PM<font
size=2 face=Tahoma>
To: amibroker@xxxxxxxxxxxxxxx
Subject: RE: [amibroker]
Optimization -- again
So according to your last paragraph it is
impossbile to develop a system that is consistently successful since just
looking for it voids it. Right?
d
<span
>-----Original Message-----
From: Howard Bandy
[mailto:howardbandy@xxxxxxxxx]
Sent: Thursday,
October 16, 2003 <span
>2:20 PM<font size=2
face=Tahoma>
To: amibroker@xxxxxxxxxxxxxxx
Subject: [amibroker] Optimization
-- again
Greetings --<span
>
In my opinion, anything we do in development of
trading systems involves a
search for a pattern than precedes a profitable
trading opportunity. Any
time we examine the results of alternative
systems, we are involved in
searching; and when we select the most promising
of those alternatives, we
are optimizing. Only a system based on truly
random entries and exits would
not be the result some optimization. So the
question of "should we
optimize?" is moot -- we have no choice but
to optimize. Consequently, we
should be aware of our optimization techniques.
Chuck referred to an optimization technique
recommendation I made to the
company we both worked for in <font
size=2 face="Courier New">Denver<font
size=2 face="Courier New"> a few years
ago. This is a short<span
>
description of it.
The company is a Commodity Trading Advisor which
traded futures, not
individual stocks, but the procedures are equally
valid for both.
When I joined the company, they were using very
long data series when
developing their models. They used a
technique sometimes called folding or
jackknifing, where the data was divided into
several periods -- say ten.
The modeling process made ten passes. During
each pass, one period was held
back to be used as out-of-sample data, the other
nine were used to select
the best parameter values. After all ten
passes, the results were gathered
together and the parameter values that scored best
overall were chosen.
There are several problems with this method.
One is the difficulty with the
"ramp up" period at the start of each
segment, another is that it is not
valid to use older data for out-of-sample testing
than was used for
in-sample development, and another is that the
data series were too long.
Chuck and I and others had many interesting
discussions about how long the
in-sample data should be.
My background is strong in both the theory and the
practice of modeling and
simulation, and includes a great deal of
experience with analysis of
financial time series. I proposed the
following method, which I continue to
believe is valid.
First, before any modeling begins. Using
judgment of management and
comparison of trading profiles of many trading
runs (real, simulated, or
imagined), pick an objective function by which the
"goodness" of a trading
system will be measured. This is important,
it is a personal or corporate
judgment, and it should not be subject to
optimization.
Divide each data series into a sequence of
in-sample and out-of-sample
periods. The length of the out-of-sample
period is the "reoptimization"
period. Say there are about ten years of historical
data available
(<font
size=2 face="Courier New">1/1/1993<font
size=2 face="Courier New"> through 1/1/2003<font
size=2 face="Courier New">. Set the
in-sample period to two years and the<font size=2
face="Courier New">
out-of-sample period to one year. Run the
following sequence: Search /
optimize using 1993 and 1994; pick the
"best" model for 1993-1994; forward
test this model for 1995 and save the results;
step forward one
reoptimization period and repeat until all the
full in-sample periods have
been used. The final optimization will have
been 2001 and 2002, with no
out-of-sample data to test. Ignore all
in-sample results!! Examine the
concatenated out-of-sample equity curve. If
it is acceptable, you have some
confidence that the parameters select by the final
optimization (2001 and
2002) will be profitable for 2003. No
guarantees -- only some confidence.
How did I pick two years for in-sample and one
year for out-of-sample? That
was just an example. The method is to set up
an automated search where the
length of the in-sample period and the length of
the out-of-sample period --
the reoptimization period -- are variables, and
then search through that
space.
Trading systems work because they identify
inefficiencies in markets. Every
profitable trade reduces the inefficiency until,
finally, the trading system
cannot overcome the frictional forces of
commission and slippage. This is
the same phenomenon that physicists talk about as
entropy.
My feeling -- and it may be different than Chuck's
-- is that the market is
not only non-stationary, but that the probability
that it will return to a
previous state is near zero.
Being non-stationary means that market conditions
change with respect to our
trading systems. If I am modeling a physical
process, such as a chemical
reaction, I can count on a predictable modelable
output for a given set of
inputs. If I am modeling a financial time
series, the output following a
given set of inputs changes over time. If a
market were stationary with
respect to an RSI oscillator system, I could
always buy a rise of the RSI
through the 20 percent line, to use a very
simplistic example.
I feel that the introduction of microcomputers,
trading system development
software, inexpensive individual brokerage
accounts, and discussion groups
such as this one have permanently changed the
realm of trading. One,
everyone who is interested can afford to buy a
computer, run AmiBroker, and
design and test trading systems. Two, if
someone develops a profitable
system and trades it, the profits it takes reduce
the potential profits
available to anyone else who trades it.
Consequently, the characteristics
of the market change in a way that moves the
market away from that model
until that trading system is no longer profitable
enough to overcome
commission and slippage. Three, a new person
beginning to study trading
system development typically tests a lot of old
systems. If one is found to
be profitable and they start trading it, the
market moves back to being
efficient. Consequently, trading systems
that used to work, but no longer
work, are very unlikely to ever work again.
So, I feel that the in-sample period should be
short so that the market
conditions do not change much over that
period. That is, I am looking for a
data series that is stationary relative to my
model. The stationary
relationship must extend beyond the in-sample
period far enough that the
model will be profitable when used for trading in
the out-of-sample data.
The length of the extension determines the
reoptimization period. It could
be years, months, or even one day. Note that
the holding period of a
typical trade is very much related to the length
of both the in-sample and
out-of-sample periods. The typical trade
should be much shorter than the
in-sample period and somewhat shorter than the
out-of-sample period.
The important point in all this is that the only
results being analyzed are
the concatenated out-of-sample trades.
As with all model development, every time I look
at the out-of-sample
results in any way, I reduce the probability that
future trading results
will be profitable. That means that I should
not perform thousands of tests
of model parameters, in-sample periods, and
out-of-sample periods, on the
same data series and then pick the best model base
on my examination of
thousands of out-of-sample results. In
effect, I will have just converted
all those out-of-sample results into in-sample
data for another step in the
development. That is legitimate, just be
aware of what is happening.
Thanks for listening,
Howard
Send BUG
REPORTS to bugs@xxxxxxxxxxxxx<span
>
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to:
amiquote@xxxxxxxxxxxxxxx
(Web page: <a
href="">http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: <a
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of
Yahoo! Groups is subject to the Yahoo!
Terms of Service.
<font size=3
face="Times New Roman">
<font size=2
color="#003399" face="Times New Roman">Send
BUG REPORTS to bugs@xxxxxxxxxxxxx<font size=2
face="Courier New">
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to:
amiquote@xxxxxxxxxxxxxxx
(Web page: <a
href="">http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: <a
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of
Yahoo! Groups is subject to the Yahoo!
Terms of Service.
Yahoo! Groups Sponsor
ADVERTISEMENT
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
|