|
PureBytes Links
Trading Reference Links
|
Hi Jitu –
Yes, a number.
An optimization is a search for a $B!H(Bbest$B!I(B
model. Given one afl trading system, say a simple moving average crossover
system, the $B!H(Bbest$B!I(B model is the combination of moving average lengths that
gives the buy and sell signals that gives $B!H(Bbest$B!I(B results. Best to you may
be quite different than best to some other trader. You might be trading
an account that cannot stand deep drawdowns, so you might select a model with a
smaller total return than the other person. You might look at K-Ratio,
while some other person looks at RAR. Whatever you look at, it is called
the $B!H(Bobjective function$B!I(B and it is used to rank the goodness of a group of
possible models. If the model development process is to be automatic, you
must be comfortable accepting the model that appears at the top of a single
column. If there are several criteria that are important to you, you can
develop your own composite objective function. It might be 40% times RAR
plus 60% times K-Ratio. Whatever it is, it must reduce to a single,
scalar value.
I have already asked Tomasz to expose the
metrics reported after an optimization, such as Ulcer Performance Index, for
our use within afl code. And also to allow us to add additional fields to
the optimization report, so that we can create and report our own custom
objective functions. Maybe in a future release of AmiBroker.
My point in the post is that everyone
should decide on his or her own measure of goodness before spending a lot of
time running the optimizer. We all need to recognize a good model when
one appears. If a truly automatic model development process is developed,
the person will not see the alternative models and will not make any judgment
among them. Let me say that another way. If I look at the list of
alternative models sorted according to my objective function and decide, for
some reason, that I prefer one other that the one at the top of the list, then
I have not included something important in the definition of my objective
function. I need to revise my objective function, and rerun earlier test
runs and model selections.
I recommend producing many sets of trade
results and equity curves, computing statistics for them, and plotting their
equity curves. Quantify, based on your judgment and trading requirements,
which models are best; and see what metrics they have. Determine your
own, personal objective function and paste it on the wall. These do not
have to come from real trading systems – you can just make them up in Excel,
import the signals into AmiBroker, and let AmiBroker compute its reports.
Now, whenever a series of tests is run,
you know exactly which is $B!H(Bbest$B!I(B.
Note – it is perfectly valid to have
different objective functions for different purposes. For example, I
might be modeling the behavior of a sector, say oil services, with the intent
of trading individual stocks based on what I learn. In this case, I want
to identify periods of rising prices with careful attention to turning points,
but without much interest in overall profit. On the other hand, I might
be modeling individual high beta tech stocks, in which case my model includes
several stop loss techniques and I care most about avoiding drawdowns.
It might be interesting to hear what
AmiBroker users feel is important to them, and why, and what the resulting
objective function is.
Thanks,
Howard
-----Original Message-----
From: jtelang
[mailto:jtelang@xxxxxxxxx]
Sent: Thursday,
October 16, 2003 <span
>12:31 PM<font
size=2 face=Tahoma>
To: amibroker@xxxxxxxxxxxxxxx
Subject: [amibroker] Re:
Optimization -- again
<font size=2
face="Courier New">Very informative, especially
the last paragraph, which is what I feel <font size=2
face="Courier New">
most people, especially new traders
such as myself, end up doing <span
>
unknowingly.<font
size=2 face="Courier New">
Could you please elaborate further
on following? Do you mean a number <span
>
here or something else?<font
size=2 face="Courier New">
> First, before any modeling
begins. Using judgment of management and<font size=2
face="Courier New">
> comparison of trading profiles
of many trading runs (real, <span
>
simulated, or<font
size=2 face="Courier New">
> imagined), pick an objective
function by which the "goodness" of a <font size=2
face="Courier New">
trading<font size=2
face="Courier New">
> system will be measured.
This is important, it is a personal or <font size=2
face="Courier New">
corporate<font size=2
face="Courier New">
> judgment, and it should not be
subject to optimization. <span
>
Thanks.<font size=2
face="Courier New">
Jitu<font size=2
face="Courier New">
--- In amibroker@xxxxxxxxxxxxxxx,
"Howard Bandy" <howardbandy@xxxx> <font size=2
face="Courier New">
wrote:<font size=2
face="Courier New">
> Greetings --<font
size=2 face="Courier New">
> <font size=2
face="Courier New">
> In my opinion, anything we do
in development of trading systems <span
>
involves a<font size=2
face="Courier New">
> search for a pattern than
precedes a profitable trading <span
>
opportunity. Any<font
size=2 face="Courier New">
> time we examine the results of
alternative systems, we are involved <span
>
in<font size=2
face="Courier New">
> searching; and when we select
the most promising of those <span
>
alternatives, we<font
size=2 face="Courier New">
> are optimizing. Only a
system based on truly random entries and <font size=2
face="Courier New">
exits would<font size=2
face="Courier New">
> not be the result some
optimization. So the question of "should we<font size=2
face="Courier New">
> optimize?" is moot -- we
have no choice but to optimize. <font size=2
face="Courier New">
Consequently, we<font
size=2 face="Courier New">
> should be aware of our
optimization techniques.<span
>
> <font size=2
face="Courier New">
> Chuck referred to an
optimization technique recommendation I made <font size=2
face="Courier New">
to the<font size=2
face="Courier New">
> company we both worked for in
Denver a few years ago. This is a <font size=2
face="Courier New">
short<font size=2
face="Courier New">
> description of it.<font
size=2 face="Courier New">
> <font size=2
face="Courier New">
> The company is a Commodity
Trading Advisor which traded futures, not<font size=2
face="Courier New">
> individual stocks, but the
procedures are equally valid for both.<font size=2
face="Courier New">
> <font size=2
face="Courier New">
> When I joined the company, they
were using very long data series <span
>
when<font size=2
face="Courier New">
> developing their models.
They used a technique sometimes called <font size=2
face="Courier New">
folding or<font size=2
face="Courier New">
> jackknifing, where the data was
divided into several periods -- say <span
>
ten.<font size=2
face="Courier New">
> The modeling process made ten
passes. During each pass, one period <font size=2
face="Courier New">
was held<font size=2
face="Courier New">
> back to be used as out-of-sample
data, the other nine were used to <span
>
select<font size=2
face="Courier New">
> the best parameter
values. After all ten passes, the results were <font size=2
face="Courier New">
gathered<font size=2
face="Courier New">
> together and the parameter
values that scored best overall were <span
>
chosen.<font size=2
face="Courier New">
> There are several problems with
this method. One is the difficulty <font size=2
face="Courier New">
with the<font size=2
face="Courier New">
> "ramp up" period at
the start of each segment, another is that it <font size=2
face="Courier New">
is not<font size=2
face="Courier New">
> valid to use older data for
out-of-sample testing than was used for<font size=2
face="Courier New">
> in-sample development, and
another is that the data series were too <font size=2
face="Courier New">
long.<font size=2
face="Courier New">
> Chuck and I and others had many
interesting discussions about how <span
>
long the<font size=2
face="Courier New">
> in-sample data should be.
> <font size=2
face="Courier New">
> My background is strong in both
the theory and the practice of <span
>
modeling and<font
size=2 face="Courier New">
> simulation, and includes a
great deal of experience with analysis of<font size=2
face="Courier New">
> financial time series. I
proposed the following method, which I <font size=2
face="Courier New">
continue to<font size=2
face="Courier New">
> believe is valid.<font
size=2 face="Courier New">
> <font size=2
face="Courier New">
> First, before any modeling
begins. Using judgment of management and<font size=2
face="Courier New">
> comparison of trading profiles
of many trading runs (real, <span
>
simulated, or<font
size=2 face="Courier New">
> imagined), pick an objective
function by which the "goodness" of a <font size=2
face="Courier New">
trading<font size=2
face="Courier New">
> system will be measured.
This is important, it is a personal or <font size=2
face="Courier New">
corporate<font size=2
face="Courier New">
> judgment, and it should not be
subject to optimization. <span
>
> <font size=2
face="Courier New">
> Divide each data series into a
sequence of in-sample and out-of-<span
>
sample<font size=2
face="Courier New">
> periods. The length of
the out-of-sample period is <span
>
the "reoptimization"<font
size=2 face="Courier New">
> period. Say there are
about ten years of historical data available<font size=2
face="Courier New">
> (1/1/1993 through
1/1/2003. Set the in-sample period to two years <font size=2
face="Courier New">
and the<font size=2
face="Courier New">
> out-of-sample period to one
year. Run the following sequence: <font size=2
face="Courier New">
Search /<font size=2
face="Courier New">
> optimize using 1993 and 1994;
pick the "best" model for 1993-1994; <font size=2
face="Courier New">
forward<font size=2
face="Courier New">
> test this model for 1995 and
save the results; step forward one<span
>
> reoptimization period and repeat
until all the full in-sample <span
>
periods have<font
size=2 face="Courier New">
> been used. The final
optimization will have been 2001 and 2002, <font size=2
face="Courier New">
with no<font size=2
face="Courier New">
> out-of-sample data to
test. Ignore all in-sample results!! <font size=2
face="Courier New">
Examine the<font size=2
face="Courier New">
> concatenated out-of-sample
equity curve. If it is acceptable, you <font size=2
face="Courier New">
have some<font size=2
face="Courier New">
> confidence that the parameters
select by the final optimization <span
>
(2001 and<font size=2
face="Courier New">
> 2002) will be profitable for
2003. No guarantees -- only some <font size=2
face="Courier New">
confidence.<font size=2
face="Courier New">
> <font size=2
face="Courier New">
> How did I pick two years for
in-sample and one year for out-of-<span
>
sample? That<font
size=2 face="Courier New">
> was just an example. The
method is to set up an automated search <font size=2
face="Courier New">
where the<font size=2
face="Courier New">
> length of the in-sample period
and the length of the out-of-sample <span
>
period --<font size=2
face="Courier New">
> the reoptimization period --
are variables, and then search through <font size=2
face="Courier New">
that<font size=2
face="Courier New">
> space. <font
size=2 face="Courier New">
> <font size=2
face="Courier New">
> Trading systems work because
they identify inefficiencies in <span
>
markets. Every<font
size=2 face="Courier New">
> profitable trade reduces the
inefficiency until, finally, the <span
>
trading system<font
size=2 face="Courier New">
> cannot overcome the frictional
forces of commission and slippage. <font size=2
face="Courier New">
This is<font size=2
face="Courier New">
> the same phenomenon that
physicists talk about as entropy.<span
>
> <font size=2
face="Courier New">
> My feeling -- and it may be
different than Chuck's -- is that the <font size=2
face="Courier New">
market is<font size=2
face="Courier New">
> not only non-stationary, but
that the probability that it will <span
>
return to a<font size=2
face="Courier New">
> previous state is near
zero.
> <font size=2
face="Courier New">
> Being non-stationary means that
market conditions change with <span
>
respect to our<font
size=2 face="Courier New">
> trading systems. If I am
modeling a physical process, such as a <font size=2
face="Courier New">
chemical<font size=2
face="Courier New">
> reaction, I can count on a
predictable modelable output for a given <font size=2
face="Courier New">
set of<font size=2
face="Courier New">
> inputs. If I am modeling
a financial time series, the output <span
>
following a<font size=2
face="Courier New">
> given set of inputs changes
over time. If a market were stationary <font size=2
face="Courier New">
with<font size=2
face="Courier New">
> respect to an RSI oscillator
system, I could always buy a rise of <span
>
the RSI<font size=2
face="Courier New">
> through the 20 percent line, to
use a very simplistic example. <span
>
> <font size=2
face="Courier New">
> I feel that the introduction of
microcomputers, trading system <span
>
development<font size=2
face="Courier New">
> software, inexpensive
individual brokerage accounts, and discussion <font size=2
face="Courier New">
groups<font size=2
face="Courier New">
> such as this one have
permanently changed the realm of trading. <font size=2
face="Courier New">
One,<font size=2
face="Courier New">
> everyone who is interested can
afford to buy a computer, run <span
>
AmiBroker, and<font
size=2 face="Courier New">
> design and test trading
systems. Two, if someone develops a <font size=2
face="Courier New">
profitable<font size=2
face="Courier New">
> system and trades it, the
profits it takes reduce the potential <font size=2
face="Courier New">
profits<font size=2
face="Courier New">
> available to anyone else who trades
it. Consequently, the <span
>
characteristics<font
size=2 face="Courier New">
> of the market change in a way
that moves the market away from that <span
>
model<font size=2
face="Courier New">
> until that trading system is no
longer profitable enough to overcome<span
>
> commission and slippage.
Three, a new person beginning to study <font size=2
face="Courier New">
trading<font size=2
face="Courier New">
> system development typically
tests a lot of old systems. If one is <font size=2
face="Courier New">
found to<font size=2
face="Courier New">
> be profitable and they start
trading it, the market moves back to <span
>
being<font size=2
face="Courier New">
> efficient. Consequently,
trading systems that used to work, but no <font size=2
face="Courier New">
longer<font size=2
face="Courier New">
> work, are very unlikely to ever
work again.
> <font size=2
face="Courier New">
> So, I feel that the in-sample
period should be short so that the <span
>
market<font size=2
face="Courier New">
> conditions do not change much
over that period. That is, I am <font size=2
face="Courier New">
looking for a<font
size=2 face="Courier New">
> data series that is stationary
relative to my model. The stationary<font size=2
face="Courier New">
> relationship must extend beyond
the in-sample period far enough <span
>
that the<font size=2
face="Courier New">
> model will be profitable when
used for trading in the out-of-sample <font size=2
face="Courier New">
data.<font size=2
face="Courier New">
> The length of the extension
determines the reoptimization period. <font size=2
face="Courier New">
It could<font size=2
face="Courier New">
> be years, months, or even one
day. Note that the holding period of <font size=2
face="Courier New">
a<font size=2
face="Courier New">
> typical trade is very much
related to the length of both the in-<span
>
sample and<font size=2
face="Courier New">
> out-of-sample periods.
The typical trade should be much shorter <font size=2
face="Courier New">
than the<font size=2
face="Courier New">
> in-sample period and somewhat
shorter than the out-of-sample period.<font size=2
face="Courier New">
> <font size=2
face="Courier New">
> The important point in all this
is that the only results being <span
>
analyzed are<font
size=2 face="Courier New">
> the concatenated out-of-sample
trades.
> <font size=2
face="Courier New">
> As with all model development,
every time I look at the out-of-<span
>
sample<font size=2
face="Courier New">
> results in any way, I reduce
the probability that future trading <span
>
results<font size=2
face="Courier New">
> will be profitable. That
means that I should not perform thousands <font size=2
face="Courier New">
of tests<font size=2
face="Courier New">
> of model parameters, in-sample
periods, and out-of-sample periods, <span
>
on the<font size=2
face="Courier New">
> same data series and then pick
the best model base on my <span
>
examination of<font
size=2 face="Courier New">
> thousands of out-of-sample
results. In effect, I will have just <font size=2
face="Courier New">
converted<font size=2
face="Courier New">
> all those out-of-sample results
into in-sample data for another <span
>
step in the<font size=2
face="Courier New">
> development. That is
legitimate, just be aware of what is <span
>
happening.<font size=2
face="Courier New">
> <font size=2
face="Courier New">
> Thanks for listening,<font
size=2 face="Courier New">
> Howard<font size=2
face="Courier New">
<font size=2
color="#003399" face="Times New Roman">Send
BUG REPORTS to bugs@xxxxxxxxxxxxx<font size=2
face="Courier New">
Send SUGGESTIONS to
suggest@xxxxxxxxxxxxx<span
>
-----------------------------------------<font
size=2 face="Courier New">
Post AmiQuote-related messages ONLY
to: amiquote@xxxxxxxxxxxxxxx <span
>
(Web page: <a
href="">http://groups.yahoo.com/group/amiquote/messages/)<font
size=2 face="Courier New">
--------------------------------------------<font
size=2 face="Courier New">
Check group FAQ at: <a
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of
Yahoo! Groups is subject to the Yahoo!
Terms of Service.
Yahoo! Groups Sponsor
ADVERTISEMENT
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
|