|
PureBytes Links
Trading Reference Links
|
<FONT face=Arial color=#0000ff
size=2>Fred,
<FONT face=Arial color=#0000ff
size=2>
I
couldn't help but reply to your question below.
<FONT face=Arial color=#0000ff
size=2>
<SPAN
class=682104718-17102003>IMO, the software that Howard and I used when we worked
together was "capable" of doing the automated task of optimizing and selecting
parameters to trade going forward. It was (and is) being used by a
fairly successful futures fund in Denver. This particular fund has
been ranked in the top-ten based on several different criteria over its 15 years
of operation. I still work for
the company, but Howard has moved on to do other
things.
<SPAN
class=682104718-17102003>
Before
addressing the issue you raised, I have to tell you a bit about this
software. It was written by a fella' by the name of Jim Yonan, with
a lot of input from Howard and others working for this particular
company. To me, the most interesting thing about it was the way it
could distribute processing over an entire network of PC's. We have
about 20 PC's in the office and some in the homes of staff members.
I have 13 PC's in my home here in New Zealand. The software is
capable of sending out small batches of optimization runs to any number of PC's
and integrating the results as each batch finishes. It's truly
amazing to watch!
<FONT face=Arial color=#0000ff
size=2>
The
software has (if I remember correctly) over 100 objective functions by which
out-of-sample results can be measured. Howard actually created most of
these functions and they are excellent. Unfortunately, a humanoid
has to decide which objective function to use and which results look
"best". This is where (IMO) the problem arises. While
the fund is still making nice profits for its clients, it could be doing a lot
better with the right person driving the research software. I
believe that parameters are chosen based on too short of learning period.
More importantly, too few trades are used when determing parameters going
forward.
<FONT face=Arial color=#0000ff
size=2>
So, I
believe the answer to your question is that such software exists. It
has been sold to other fund managers, but the price tag is over $1 million so it
isn't going to help the average punter. But the concept can be
fairly easily re-created by a couple of good programmers in a month or
two.
<FONT face=Arial color=#0000ff
size=2>
BTW,
this particular futures fund is 100% mechancal. The trading software
monitors every tick in 47 different futures markets and automatically generates
orders. Orders are either printed out for a dealer to call into the floor
or, in some cases, go straight to the floor electronically.
<FONT face=Arial color=#0000ff
size=2>
The
manager of the fund is about to launch a U.S. based stock hedge fund using
similar concepts.
<BLOCKQUOTE
>
<FONT face="Times New Roman"
size=2>-----Original Message-----From: Fred
[mailto:fctonetti@xxxxxxxxx]Sent: Friday, October 17, 2003 11:01
AMTo: amibroker@xxxxxxxxxxxxxxxSubject: [amibroker] Re:
Optimization -- againWalk forward testing and
optimization is a great concept in theory and although I've seen lots of
ideas for how to set it up and be used over the years, unfortunately I've
yet to see anyone demonstrate that it actually works over a variety of
market conditions. There are I'm sure lots of reasons for this which
I won't delve into here but the question remains, has anyone actually seen
this put into practice where the result has been a viable system to use in
mechanical trading ? If so can you please point at something that
could be looked at objectively that doesn't reside in a black box
?--- In amibroker@xxxxxxxxxxxxxxx, "Howard Bandy"
<howardbandy@xxxx> wrote:> Greetings --> > In
my opinion, anything we do in development of trading systems involves
a> search for a pattern than precedes a profitable trading
opportunity. Any> time we examine the results of alternative
systems, we are involved in> searching; and when we select the most
promising of those alternatives, we> are optimizing. Only a
system based on truly random entries and exits would> not be the
result some optimization. So the question of "should we>
optimize?" is moot -- we have no choice but to optimize.
Consequently, we> should be aware of our optimization
techniques.> > Chuck referred to an optimization technique
recommendation I made to the> company we both worked for in Denver
a few years ago. This is a short> description of it.>
> The company is a Commodity Trading Advisor which traded futures,
not> individual stocks, but the procedures are equally valid for
both.> > When I joined the company, they were using very long
data series when> developing their models. They used a
technique sometimes called folding or> jackknifing, where the data
was divided into several periods -- say ten.> The modeling process
made ten passes. During each pass, one period was held> back
to be used as out-of-sample data, the other nine were used to
select> the best parameter values. After all ten passes, the
results were gathered> together and the parameter values that
scored best overall were chosen.> There are several problems with
this method. One is the difficulty with the> "ramp up" period
at the start of each segment, another is that it is not> valid to
use older data for out-of-sample testing than was used for> in-sample
development, and another is that the data series were too long.>
Chuck and I and others had many interesting discussions about how long
the> in-sample data should be. > > My background is
strong in both the theory and the practice of modeling and>
simulation, and includes a great deal of experience with analysis of>
financial time series. I proposed the following method, which I
continue to> believe is valid.> > First, before any
modeling begins. Using judgment of management and> comparison of
trading profiles of many trading runs (real, simulated, or>
imagined), pick an objective function by which the "goodness" of a
trading> system will be measured. This is important, it is a
personal or corporate> judgment, and it should not be subject to
optimization. > > Divide each data series into a sequence
of in-sample and out-of-sample> periods. The length of the
out-of-sample period is the "reoptimization"> period. Say
there are about ten years of historical data available> (1/1/1993
through 1/1/2003. Set the in-sample period to two years and
the> out-of-sample period to one year. Run the following
sequence: Search /> optimize using 1993 and 1994; pick the
"best" model for 1993-1994; forward> test this model for 1995 and
save the results; step forward one> reoptimization period and repeat
until all the full in-sample periods have> been used. The
final optimization will have been 2001 and 2002, with no>
out-of-sample data to test. Ignore all in-sample results!!
Examine the> concatenated out-of-sample equity curve. If it
is acceptable, you have some> confidence that the parameters select
by the final optimization (2001 and> 2002) will be profitable for
2003. No guarantees -- only some confidence.> > How
did I pick two years for in-sample and one year for out-of-sample?
That> was just an example. The method is to set up an automated
search where the> length of the in-sample period and the length of
the out-of-sample period --> the reoptimization period -- are
variables, and then search through that> space. >
> Trading systems work because they identify inefficiencies in
markets. Every> profitable trade reduces the inefficiency
until, finally, the trading system> cannot overcome the frictional
forces of commission and slippage. This is> the same
phenomenon that physicists talk about as entropy.> > My feeling
-- and it may be different than Chuck's -- is that the market is>
not only non-stationary, but that the probability that it will return to
a> previous state is near zero. > > Being
non-stationary means that market conditions change with respect to
our> trading systems. If I am modeling a physical process, such
as a chemical> reaction, I can count on a predictable modelable
output for a given set of> inputs. If I am modeling a
financial time series, the output following a> given set of inputs
changes over time. If a market were stationary with> respect
to an RSI oscillator system, I could always buy a rise of the RSI>
through the 20 percent line, to use a very simplistic example. >
> I feel that the introduction of microcomputers, trading system
development> software, inexpensive individual brokerage accounts,
and discussion groups> such as this one have permanently changed
the realm of trading. One,> everyone who is interested can
afford to buy a computer, run AmiBroker, and> design and test
trading systems. Two, if someone develops a profitable>
system and trades it, the profits it takes reduce the potential
profits> available to anyone else who trades it.
Consequently, the characteristics> of the market change in a way
that moves the market away from that model> until that trading
system is no longer profitable enough to overcome> commission and
slippage. Three, a new person beginning to study trading>
system development typically tests a lot of old systems. If one is
found to> be profitable and they start trading it, the market moves
back to being> efficient. Consequently, trading systems that
used to work, but no longer> work, are very unlikely to ever work
again.> > So, I feel that the in-sample period should be short
so that the market> conditions do not change much over that
period. That is, I am looking for a> data series that is
stationary relative to my model. The stationary> relationship
must extend beyond the in-sample period far enough that the> model
will be profitable when used for trading in the out-of-sample
data.> The length of the extension determines the reoptimization
period. It could> be years, months, or even one day.
Note that the holding period of a> typical trade is very much
related to the length of both the in-sample and> out-of-sample
periods. The typical trade should be much shorter than the>
in-sample period and somewhat shorter than the out-of-sample period.>
> The important point in all this is that the only results being
analyzed are> the concatenated out-of-sample trades.>
> As with all model development, every time I look at the
out-of-sample> results in any way, I reduce the probability that
future trading results> will be profitable. That means that I
should not perform thousands of tests> of model parameters,
in-sample periods, and out-of-sample periods, on the> same data
series and then pick the best model base on my examination of>
thousands of out-of-sample results. In effect, I will have just
converted> all those out-of-sample results into in-sample data for
another step in the> development. That is legitimate, just be
aware of what is happening.> > Thanks for listening,>
HowardSend
BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to
suggest@xxxxxxxxxxxxx-----------------------------------------Post
AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: <A
href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check
group FAQ at: <A
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the <A
href="">Yahoo! Terms of Service.
Yahoo! Groups Sponsor
ADVERTISEMENT
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
|