|
PureBytes Links
Trading Reference Links
|
<FONT face=Arial color=#0000ff
size=2>Probably the much more dangerous naivet'e rather than the benign sense of
humor.
<FONT face=Arial color=#0000ff
size=2>
<FONT face=Arial color=#0000ff
size=2>IMHO.
<FONT face=Arial color=#0000ff
size=2>
<FONT face=Arial color=#0000ff
size=2>d
<FONT
face=Tahoma size=2>-----Original Message-----From:
CedarCreekTrading [mailto:kernish@xxxxxxxxxxx] Sent: Sunday,
October 19, 2003 3:01 PMTo:
amibroker@xxxxxxxxxxxxxxxSubject: Re: [amibroker] Re: Optimization
-- again
Whatever Larry says has great validity, as it's
coming from probably the most famous and most knowledgeable trader of
modern times.
Certainly, you jest.
Take care,
Steve
<BLOCKQUOTE
>
----- Original Message -----
<DIV
>From:
palsanand
To: <A title=amibroker@xxxxxxxxxxxxxxx
href="">amibroker@xxxxxxxxxxxxxxx
Sent: Sunday, October 19, 2003 10:45
AM
Subject: [amibroker] Re: Optimization
-- again
Hi All,About Larry Williams' contribution in
support of Walk Forward Testing: Whatever Larry says has great validity,
as it's coming from probably the most famous and most knowledgeable
trader of modern times.It's certainly correct that if done
properly, walk forward testing has great value. For those of you not
aware of walk forward testing, it's first setting your system parameters
and then testing the results in the future using those pre-set
parameters without benefit of additional or new optimization. Some
people refer to that as "hypothetical real-time trading."
However, walk forward testing can in fact be a trap if done
incorrectly. That's because there's a problem in deciding what
pre-set algorithm or parameters to use prior to the so-called walk
forward test. If we arrive at those parameters by an optimization
process, then we may be guilty of optimizing the walk forward test
without even realizing we have done that. Another pitfall, is the
great tendency to optimize the walk forward testing time period
itself. Possibly the only way to do it correctly, is to first
arrive at a set of parameters and algorithm based on logic, experience,
or sound trading principals that won't be subject to change. Then do a
walk forward with no attempt to improve results via
optimization.Regards,Pal--- In
amibroker@xxxxxxxxxxxxxxx, "palsanand" <palsanand@xxxx> wrote:>
Hi,> > I came across something interesting from a famous
trader:> > Walk Forward Tested System Is Better than the Best
> Optimized One - Larry Williams > > It looks like the
history of technical analysis has been largely > influenced by
optimization. That is, we studied the past, found > something that
looked significant, then optimized rules and > procedures to trade
the observation in the future.> > Sometimes that has worked.
Often it has not. That's our dilemma. What > are we to do? In the
past, we answered these questions by doing more > optimization,
more curve fitting. Indeed, we treated historical data > like
prisoners of war. Our thesis was, if you beat them often enough >
they would reveal anything. Which is true, but you want them to >
reveal everything, not anything.> > This brings me to one
point. I think we will all make much more > headway with system
development by spending less time on optimization > and more time
on walking systems and procedures forward.> > If on a walk
forward test, the system holds up, we probably have > something. And
for sure, what we have will be better than the very > best optimized
system when it comes to real time trading. Hence, > let's see what we
can learn from each other about conducting walk > forward tests. Any
ideas will be appreciated by all, I am certain.> >
Regards,> > Pal> --- In amibroker@xxxxxxxxxxxxxxx,
"Howard Bandy" <howardbandy@xxxx> > wrote:> > Hi
Dingo -> > > > > > > > It's
not quite that bad. > > > > > >
> > My last paragraph is a warning for the common technique that
many > people try> > - looking at out-of-sample results so
much that they become in-> sample data.> > That is OK, if
there is another set of data that will be used for > further>
> testing. Sometimes model development is done using three sets of
> data -> > learning, testing, and validation. The
learning data is extensively> > searched and used to select
parameter values. The testing data is > used to>
> determine when to stop searching and use the results found to be
> best so> > far. For each set of parameters, a run
is made on the learning > data, and> > the value of the
objective function is noted. Periodically, the > model
using> > the best parameters is run against the test data.
When the > performance over> > the test data falls off,
that tells the optimizer to stop and > return the> >
parameters that worked best over the test data. One (or a Very
few)> > additional run is made against the validation
data. If results are> > acceptable, the model is
used. If the validation data produces poor> > results,
do not use the model. Rather, go back to the model design >
stage> > and come up with something new to try.> >
> > > > > > If the model is able to
recognize profitable situations, and the > parameters> >
are stable, a wide range of parameters will be profitable.> >
> > > > > > One of my points in the
posting is that the walk-forward testing > with> >
automatic selection of parameter values should be taken all >
together as a> > process. If the results are good, then the
process has validity > and there> > is a reasonable
expectation that trading the model and parameter set> >
resulting from the last optimization will be profitable.> >
> > > > > > Thanks,> >
> > Howard> > > > > > >
> -----Original Message-----> > From: dingo [mailto:dingo@xxxx]
> > Sent: Thursday, October 16, 2003 12:02 PM> > To:
amibroker@xxxxxxxxxxxxxxx> > Subject: RE: [amibroker] Optimization
-- again> > > > > > > > So
according to your last paragraph it is impossbile to develop a >
system> > that is consistently successful since just looking for
it voids > it. Right?> > > > >
> > > > > d> > > >
-----Original Message-----> > From: Howard Bandy
[mailto:howardbandy@xxxx] > > Sent: Thursday, October 16, 2003
2:20 PM> > To: amibroker@xxxxxxxxxxxxxxx> > Subject:
[amibroker] Optimization -- again> > > > Greetings
--> > > > In my opinion, anything we do in development
of trading systems > involves a> > search for a pattern
than precedes a profitable trading > opportunity. Any>
> time we examine the results of alternative systems, we are involved
> in> > searching; and when we select the most promising of
those > alternatives, we> > are optimizing. Only a
system based on truly random entries and > exits would> >
not be the result some optimization. So the question of "should
we> > optimize?" is moot -- we have no choice but to
optimize. > Consequently, we> > should be aware of
our optimization techniques.> > > > Chuck referred to an
optimization technique recommendation I made > to the> >
company we both worked for in Denver a few years ago. This is a
> short> > description of it.> > > >
The company is a Commodity Trading Advisor which traded futures,
not> > individual stocks, but the procedures are equally valid
for both.> > > > When I joined the company, they were
using very long data series > when> > developing their
models. They used a technique sometimes called > folding
or> > jackknifing, where the data was divided into several periods
-- say > ten.> > The modeling process made ten
passes. During each pass, one period > was held>
> back to be used as out-of-sample data, the other nine were used to
> select> > the best parameter values. After all ten
passes, the results were > gathered> > together and the
parameter values that scored best overall were > chosen.> >
There are several problems with this method. One is the difficulty
> with the> > "ramp up" period at the start of each
segment, another is that it > is not> > valid to use older
data for out-of-sample testing than was used for> > in-sample
development, and another is that the data series were too >
long.> > Chuck and I and others had many interesting discussions
about how > long the> > in-sample data should be.
> > > > My background is strong in both the theory and
the practice of > modeling and> > simulation, and includes
a great deal of experience with analysis of> > financial time
series. I proposed the following method, which I > continue
to> > believe is valid.> > > > First, before
any modeling begins. Using judgment of management and> >
comparison of trading profiles of many trading runs (real, >
simulated, or> > imagined), pick an objective function by which
the "goodness" of a > trading> > system will be
measured. This is important, it is a personal or >
corporate> > judgment, and it should not be subject to
optimization. > > > > Divide each data series into
a sequence of in-sample and out-of-> sample> >
periods. The length of the out-of-sample period is > the
"reoptimization"> > period. Say there are about ten years of
historical data available> > (1/1/1993 through 1/1/2003.
Set the in-sample period to two years > and the> >
out-of-sample period to one year. Run the following sequence:
> Search /> > optimize using 1993 and 1994; pick the "best"
model for 1993-1994; > forward> > test this model for
1995 and save the results; step forward one> > reoptimization
period and repeat until all the full in-sample > periods have>
> been used. The final optimization will have been 2001 and 2002,
> with no> > out-of-sample data to test. Ignore all
in-sample results!! > Examine the> > concatenated
out-of-sample equity curve. If it is acceptable, you > have
some> > confidence that the parameters select by the final
optimization > (2001 and> > 2002) will be profitable for
2003. No guarantees -- only some > confidence.> >
> > How did I pick two years for in-sample and one year for
out-of-> sample? That> > was just an example.
The method is to set up an automated search > where the> >
length of the in-sample period and the length of the out-of-sample
> period --> > the reoptimization period -- are variables,
and then search through > that> > space. >
> > > Trading systems work because they identify inefficiencies
in > markets. Every> > profitable trade reduces the
inefficiency until, finally, the > trading system> > cannot
overcome the frictional forces of commission and slippage.
> This is> > the same phenomenon that physicists talk about
as entropy.> > > > My feeling -- and it may be different
than Chuck's -- is that the > market is> > not only
non-stationary, but that the probability that it will > return to
a> > previous state is near zero. > > >
> Being non-stationary means that market conditions change with >
respect to our> > trading systems. If I am modeling a
physical process, such as a > chemical> > reaction, I can
count on a predictable modelable output for a given > set
of> > inputs. If I am modeling a financial time series, the
output > following a> > given set of inputs changes over
time. If a market were stationary > with> >
respect to an RSI oscillator system, I could always buy a rise of >
the RSI> > through the 20 percent line, to use a very simplistic
example. > > > > I feel that the introduction of
microcomputers, trading system > development> > software,
inexpensive individual brokerage accounts, and discussion >
groups> > such as this one have permanently changed the realm of
trading. > One,> > everyone who is interested can
afford to buy a computer, run > AmiBroker, and> > design
and test trading systems. Two, if someone develops a >
profitable> > system and trades it, the profits it takes reduce
the potential > profits> > available to anyone else who
trades it. Consequently, the > characteristics> > of
the market change in a way that moves the market away from that >
model> > until that trading system is no longer profitable enough
to overcome> > commission and slippage. Three, a new
person beginning to study > trading> > system development
typically tests a lot of old systems. If one is > found
to> > be profitable and they start trading it, the market moves
back to > being> > efficient. Consequently, trading
systems that used to work, but no > longer> > work, are
very unlikely to ever work again.> > > > So, I feel that
the in-sample period should be short so that the > market>
> conditions do not change much over that period. That is, I am
> looking for a> > data series that is stationary relative
to my model. The stationary> > relationship must extend
beyond the in-sample period far enough > that the> > model
will be profitable when used for trading in the out-of-sample >
data.> > The length of the extension determines the reoptimization
period. > It could> > be years, months, or even
one day. Note that the holding period of > a> >
typical trade is very much related to the length of both the in->
sample and> > out-of-sample periods. The typical trade
should be much shorter > than the> > in-sample period and
somewhat shorter than the out-of-sample period.> > >
> The important point in all this is that the only results being >
analyzed are> > the concatenated out-of-sample trades.>
> > > As with all model development, every time I look at the
out-of-> sample> > results in any way, I reduce the
probability that future trading > results> > will be
profitable. That means that I should not perform thousands
> of tests> > of model parameters, in-sample periods, and
out-of-sample periods, > on the> > same data series and
then pick the best model base on my > examination of> >
thousands of out-of-sample results. In effect, I will have just
> converted> > all those out-of-sample results into
in-sample data for another > step in the> >
development. That is legitimate, just be aware of what is >
happening.> > > > Thanks for listening,> >
Howard> > > > > > > > > >
> > Send BUG REPORTS to bugs@xxxx> > Send SUGGESTIONS to
suggest@xxxx> > ----------------------------------------->
> Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
> > (Web page: <A
href="">http://groups.yahoo.com/group/amiquote/messages/)>
> --------------------------------------------> > Check group
FAQ at:> > <A
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
> > > > Your use of Yahoo! Groups is subject to the
Yahoo!> > <<A
href="">http://docs.yahoo.com/info/terms/>
Terms of Service. > > > > > > >
> > > Yahoo! Groups Sponsor> > > > >
> > > ADVERTISEMENT> > > > >
<<A
href="">http://rd.yahoo.com/M=244522.3707890.4968055.1261774/D=egroupweb/S=17>
056321> > >
98:HM/A=1595056/R=0/SIG=124p07ne0/*http:/ashnin.com/clk/muryutaitakena>
ttogyo> > ?YH=3707890&yhad=1595056> Click Here!>
> > > > > > > <<A
href="">http://us.adserver.yahoo.com/l?>
M=244522.3707890.4968055.1261774/D=egroupmai> >
l/S=:HM/A=1595056/rand=910277976> > > > > >
> Send BUG REPORTS to bugs@xxxx> > Send SUGGESTIONS to
suggest@xxxx> > ----------------------------------------->
> Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
> > (Web page: <A
href="">http://groups.yahoo.com/group/amiquote/messages/)>
> --------------------------------------------> > Check group
FAQ at:> > <A
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
> > > > Your use of Yahoo! Groups is subject to the
Yahoo!> > <<A
href="">http://docs.yahoo.com/info/terms/>
Terms of Service.Send BUG REPORTS to
bugs@xxxxxxxxxxxxxSend SUGGESTIONS to
suggest@xxxxxxxxxxxxx-----------------------------------------Post
AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page:
<A
href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check
group FAQ at: <A
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the <A
href="">Yahoo! Terms of Service.
Send
BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to
suggest@xxxxxxxxxxxxx-----------------------------------------Post
AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: <A
href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check
group FAQ at: <A
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the <A
href="">Yahoo! Terms of Service.
Yahoo! Groups Sponsor
ADVERTISEMENT
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
|