|
PureBytes Links
Trading Reference Links
|
Gary<font size=2
color=navy face=Arial> –
Pardon my jumping in, but look at my post
today titled “objective functions”.
Howard
-----Original Message-----
From: Gary A. Serkhoshian
[mailto:serkhoshian777@xxxxxxxxx]
Sent: Friday, October 17, 2003
3:13 PM
To: amibroker@xxxxxxxxxxxxxxx
Subject: RE: [amibroker] Re:
Optimization -- again
Chuck,
Related to your comments to Fred:
The software has (if I remember correctly)
over 100 objective functions by which out-of-sample results can be measured.
<span
>Could you elaborate a bit on what you mean
by objective functions. I think of MAR, CAR, K-Ratio, UPI, etc. when I
read your comments above, and wanted to make sure I'm on track.
Unfortunately, a humanoid has to decide
which objective function to use and which results look
"best". This is where (IMO) the problem arises. While
the fund is still making nice profits for its clients, it could be doing a lot
better with the right person driving the research software.
"Best" in terms of looking for
the smoothest curve in sensitivity analysis? Clearly you are alluding to
the importance of experience in making this decision. However, you've
also alluded in past posts to the fact that less experience (i.e. myself) can
get a good solution by looking for the smooth curves, and not getting
overly complex.
I believe that parameters are chosen based
on too short of learning period. More importantly, too few trades are
used when determing parameters going forward.
What do
you think is a reasonable number of trades for a good sample size.
Thanks in
advance,
Gary
Chuck Rademacher
<chuck_rademacher@xxxxxxxxxx> wrote:
Fred,
I couldn't help but reply to your question
below.
IMO, the software that Howard and I used
when we worked together was "capable" of doing the automated task of
optimizing and selecting parameters to trade going forward. It was
(and is) being used by a fairly successful futures fund in Denver.
This particular fund has been ranked in the top-ten based on several different
criteria over its 15 years of operation. I still work for the
company, but Howard has moved on to do other things.
Before addressing the issue you raised, I
have to tell you a bit about this software. It was written by a
fella' by the name of Jim Yonan, with a lot of input from Howard and others
working for this particular company. To me, the most interesting
thing about it was the way it could distribute processing over an entire
network of PC's. We have about 20 PC's in the office and some in
the homes of staff members. I have 13 PC's in my home here in New
Zealand. The software is capable of sending out small batches of
optimization runs to any number of PC's and integrating the results as each
batch finishes. It's truly amazing to watch!
The software has (if I remember correctly)
over 100 objective functions by which out-of-sample results can be
measured. Howard actually created most of these functions and they are
excellent. Unfortunately, a humanoid has to decide which objective
function to use and which results look "best". This is
where (IMO) the problem arises. While the fund is still making nice
profits for its clients, it could be doing a lot better with the right person
driving the research software. I believe that parameters are chosen
based on too short of learning period. More importantly, too few trades
are used when determing parameters going forward.
So, I believe the answer to your question
is that such software exists. It has been sold to other fund
managers, but the price tag is over $1 million so it isn't going to help the
average punter. But the concept can be fairly easily
re-created by a couple of good programmers in a month or two.
BTW, this particular futures fund is 100%
mechancal. The trading software monitors every tick in 47 different
futures markets and automatically generates orders. Orders are either
printed out for a dealer to call into the floor or, in some cases, go straight
to the floor electronically.
The manager of the fund is about to launch
a U.S. based stock hedge fund using similar concepts.
<font size=2
face="Times New Roman">-----Original
Message-----
From: Fred
[mailto:fctonetti@xxxxxxxxx]
Sent: Friday, October 17, 2003
11:01 AM
To: amibroker@xxxxxxxxxxxxxxx
Subject: [amibroker] Re:
Optimization -- again
Walk forward testing and optimization is a great concept in theory <font
size=2 face="Courier New">
and although I've seen lots of ideas for how to
set it up and be used
over the years, unfortunately I've yet to see
anyone demonstrate that
it actually works over a variety of market
conditions. There are I'm
sure lots of reasons for this which I won't delve
into here but the
question remains, has anyone actually seen this
put into practice
where the result has been a viable system to use
in mechanical
trading ? If so can you please point at
something that could be
looked at objectively that doesn't reside in a
black box ?
--- In amibroker@xxxxxxxxxxxxxxx, "Howard
Bandy" <howardbandy@xxxx>
wrote:
> Greetings --
>
> In my opinion, anything we do in development
of trading systems
involves a
> search for a pattern than precedes a
profitable trading
opportunity. Any
> time we examine the results of alternative
systems, we are involved
in
> searching; and when we select the most
promising of those
alternatives, we
> are optimizing. Only a system based on
truly random entries and
exits would
> not be the result some optimization. So
the question of "should we
> optimize?" is moot -- we have no choice
but to optimize.
Consequently, we
> should be aware of our optimization
techniques.
>
> Chuck referred to an optimization technique
recommendation I made
to the
> company we both worked for in Denver a few
years ago. This is a
short
> description of it.
>
> The company is a Commodity Trading Advisor
which traded futures, not
> individual stocks, but the procedures are
equally valid for both.
>
> When I joined the company, they were using
very long data series
when
> developing their models. They used a
technique sometimes called
folding or
> jackknifing, where the data was divided into
several periods -- say
ten.
> The modeling process made ten passes.
During each pass, one period
was held
> back to be used as out-of-sample data, the
other nine were used to
select
> the best parameter values. After all
ten passes, the results were
gathered
> together and the parameter values that scored
best overall were
chosen.
> There are several problems with this
method. One is the difficulty
with the
> "ramp up" period at the start of
each segment, another is that it
is not
> valid to use older data for out-of-sample
testing than was used for
> in-sample development, and another is that
the data series were too
long.
> Chuck and I and others had many interesting
discussions about how
long the
> in-sample data should be.
>
> My background is strong in both the theory
and the practice of
modeling and
> simulation, and includes a great deal of
experience with analysis of
> financial time series. I proposed the
following method, which I
continue to
> believe is valid.
>
> First, before any modeling begins.
Using judgment of management and
> comparison of trading profiles of many
trading runs (real,
simulated, or
> imagined), pick an objective function by
which the "goodness" of a
trading
> system will be measured. This is
important, it is a personal or
corporate
> judgment, and it should not be subject to
optimization.
>
> Divide each data series into a sequence of
in-sample and out-of-
sample
> periods. The length of the
out-of-sample period is
the "reoptimization"
> period. Say there are about ten years
of historical data available
> (1/1/1993 through 1/1/2003. Set the
in-sample period to two years
and the
> out-of-sample period to one year. Run
the following sequence:
Search /
> optimize using 1993 and 1994; pick the
"best" model for 1993-1994;
forward
> test this model for 1995 and save the
results; step forward one
> reoptimization period and repeat until all
the full in-sample
periods have
> been used. The final optimization will
have been 2001 and 2002,
with no
> out-of-sample data to test. Ignore all
in-sample results!!
Examine the
> concatenated out-of-sample equity
curve. If it is acceptable, you
have some
> confidence that the parameters select by the
final optimization
(2001 and
> 2002) will be profitable for 2003. No
guarantees -- only some
confidence.
>
> How did I pick two years for in-sample and
one year for out-of-
sample? That
> was just an example. The method is to
set up an automated search
where the
> length of the in-sample period and the length
of the out-of-sample
period --
> the reoptimization period -- are variables,
and then search through
that
> space.
>
> Trading systems work because they identify
inefficiencies in
markets. Every
> profitable trade reduces the inefficiency
until, finally, the
trading system
> cannot overcome the frictional forces of
commission and slippage.
This is
> the same phenomenon that physicists talk
about as entropy.
>
> My feeling -- and it may be different than
Chuck's -- is that the
market is
> not only non-stationary, but that the
probability that it will
return to a
> previous state is near zero.
>
> Being non-stationary means that market
conditions change with
respect to our
> trading systems. If I am modeling a
physical process, such as a
chemical
> reaction, I can count on a predictable
modelable output for a given
set of
> inputs. If I am modeling a financial
time series, the output
following a
> given set of inputs changes over time.
If a market were stationary
with
> respect to an RSI oscillator system, I could
always buy a rise of
the RSI
> through the 20 percent line, to use a very
simplistic example.
>
> I feel that the introduction of
microcomputers, trading system
development
> software, inexpensive individual brokerage
accounts, and discussion
groups
> such as this one have permanently changed the
realm of trading.
One,
> everyone who is interested can afford to buy
a computer, run
AmiBroker, and
> design and test trading systems. Two,
if someone develops a
profitable
> system and trades it, the profits it takes
reduce the potential
profits
> available to anyone else who trades it.
Consequently, the
characteristics
> of the market change in a way that moves the
market away from that
model
> until that trading system is no longer
profitable enough to overcome
> commission and slippage. Three, a new
person beginning to study
trading
> system development typically tests a lot of
old systems. If one is
found to
> be profitable and they start trading it, the
market moves back to
being
> efficient. Consequently, trading
systems that used to work, but no
longer
> work, are very unlikely to ever work again.
>
> So, I feel that the in-sample period should
be short so that the
market
> conditions do not change much over that
period. That is, I am
looking for a
> data series that is stationary relative to my
model. The stationary
> relationship must extend beyond the in-sample
period far enough
that the
> model will be profitable when used for
trading in the out-of-sample
data.
> The length of the extension determines the
reoptimization period.
It could
> be years, months, or even one day. Note
that the holding period of
a
> typical trade is very much related to the
length of both the in-
sample and
> out-of-sample periods. The typical
trade should be much shorter
than the
> in-sample period and somewhat shorter than
the out-of-sample period.
>
> The important point in all this is that the
only results being
analyzed are
> the concatenated out-of-sample trades.
>
> As with all model development, every time I
look at the out-of-
sample
> results in any way, I reduce the probability
that future trading
results
> will be profitable. That means that I
should not perform thousands
of tests
> of model parameters, in-sample periods, and
out-of-sample periods,
on the
> same data series and then pick the best model
base on my
examination of
> thousands of out-of-sample results. In
effect, I will have just
converted
> all those out-of-sample results into
in-sample data for another
step in the
> development. That is legitimate, just
be aware of what is
happening.
>
> Thanks for listening,
> Howard
Send BUG
REPORTS to bugs@xxxxxxxxxxxxx<span
>
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to:
amiquote@xxxxxxxxxxxxxxx
(Web page: <a
href="">http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: <a
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of
Yahoo! Groups is subject to the Yahoo!
Terms of Service.
Send
BUG REPORTS to bugs@xxxxxxxxxxxxx<font size=2
face="Courier New">
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to:
amiquote@xxxxxxxxxxxxxxx
(Web page: <a
href="">http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: <a
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of
Yahoo! Groups is subject to the Yahoo!
Terms of Service.
<font size=3
face="Times New Roman">
Do you Yahoo!?
<a
href="">The
New Yahoo! Shopping - with improved product search
<font size=2
face="Courier New">Send BUG REPORTS to
bugs@xxxxxxxxxxxxx<span
>
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to:
amiquote@xxxxxxxxxxxxxxx
(Web page: <a
href="">http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: <a
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of
Yahoo! Groups is subject to the Yahoo!
Terms of Service.
Yahoo! Groups Sponsor
ADVERTISEMENT
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
|