|
PureBytes Links
Trading Reference Links
|
LOL ... funny ...
What Larry knows may be interesting, but what he says and what he
sells is something else entirely.
--- In amibroker@xxxxxxxxxxxxxxx, "palsanand" <palsanand@xxxx> wrote:
> Hi All,
>
> About Larry Williams' contribution in support of Walk Forward
> Testing: Whatever Larry says has great validity, as it's coming
from
> probably the most famous and most knowledgeable trader of modern
> times.
>
> It's certainly correct that if done properly, walk forward testing
> has great value. For those of you not aware of walk forward
testing,
> it's first setting your system parameters and then testing the
> results in the future using those pre-set parameters without
benefit
> of additional or new optimization. Some people refer to that
> as "hypothetical real-time trading."
>
> However, walk forward testing can in fact be a trap if done
> incorrectly. That's because there's a problem in deciding what pre-
> set algorithm or parameters to use prior to the so-called walk
> forward test. If we arrive at those parameters by an optimization
> process, then we may be guilty of optimizing the walk forward test
> without even realizing we have done that. Another pitfall, is the
> great tendency to optimize the walk forward testing time period
> itself.
>
> Possibly the only way to do it correctly, is to first arrive at a
set
> of parameters and algorithm based on logic, experience, or sound
> trading principals that won't be subject to change. Then do a walk
> forward with no attempt to improve results via optimization.
>
> Regards,
>
> Pal
> --- In amibroker@xxxxxxxxxxxxxxx, "palsanand" <palsanand@xxxx>
wrote:
> > Hi,
> >
> > I came across something interesting from a famous trader:
> >
> > Walk Forward Tested System Is Better than the Best
> > Optimized One - Larry Williams
> >
> > It looks like the history of technical analysis has been largely
> > influenced by optimization. That is, we studied the past, found
> > something that looked significant, then optimized rules and
> > procedures to trade the observation in the future.
> >
> > Sometimes that has worked. Often it has not. That's our dilemma.
> What
> > are we to do? In the past, we answered these questions by doing
> more
> > optimization, more curve fitting. Indeed, we treated historical
> data
> > like prisoners of war. Our thesis was, if you beat them often
> enough
> > they would reveal anything. Which is true, but you want them to
> > reveal everything, not anything.
> >
> > This brings me to one point. I think we will all make much more
> > headway with system development by spending less time on
> optimization
> > and more time on walking systems and procedures forward.
> >
> > If on a walk forward test, the system holds up, we probably have
> > something. And for sure, what we have will be better than the
very
> > best optimized system when it comes to real time trading. Hence,
> > let's see what we can learn from each other about conducting walk
> > forward tests. Any ideas will be appreciated by all, I am certain.
> >
> > Regards,
> >
> > Pal
> > --- In amibroker@xxxxxxxxxxxxxxx, "Howard Bandy"
<howardbandy@xxxx>
> > wrote:
> > > Hi Dingo -
> > >
> > >
> > >
> > > It's not quite that bad.
> > >
> > >
> > >
> > > My last paragraph is a warning for the common technique that
many
> > people try
> > > - looking at out-of-sample results so much that they become in-
> > sample data.
> > > That is OK, if there is another set of data that will be used
for
> > further
> > > testing. Sometimes model development is done using three sets
of
> > data -
> > > learning, testing, and validation. The learning data is
> extensively
> > > searched and used to select parameter values. The testing data
> is
> > used to
> > > determine when to stop searching and use the results found to
be
> > best so
> > > far. For each set of parameters, a run is made on the learning
> > data, and
> > > the value of the objective function is noted. Periodically,
the
> > model using
> > > the best parameters is run against the test data. When the
> > performance over
> > > the test data falls off, that tells the optimizer to stop and
> > return the
> > > parameters that worked best over the test data. One (or a Very
> few)
> > > additional run is made against the validation data. If results
> are
> > > acceptable, the model is used. If the validation data produces
> poor
> > > results, do not use the model. Rather, go back to the model
> design
> > stage
> > > and come up with something new to try.
> > >
> > >
> > >
> > > If the model is able to recognize profitable situations, and
the
> > parameters
> > > are stable, a wide range of parameters will be profitable.
> > >
> > >
> > >
> > > One of my points in the posting is that the walk-forward
testing
> > with
> > > automatic selection of parameter values should be taken all
> > together as a
> > > process. If the results are good, then the process has
validity
> > and there
> > > is a reasonable expectation that trading the model and
parameter
> set
> > > resulting from the last optimization will be profitable.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Howard
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: dingo [mailto:dingo@x...]
> > > Sent: Thursday, October 16, 2003 12:02 PM
> > > To: amibroker@xxxxxxxxxxxxxxx
> > > Subject: RE: [amibroker] Optimization -- again
> > >
> > >
> > >
> > > So according to your last paragraph it is impossbile to develop
a
> > system
> > > that is consistently successful since just looking for it voids
> > it. Right?
> > >
> > >
> > >
> > >
> > > d
> > >
> > > -----Original Message-----
> > > From: Howard Bandy [mailto:howardbandy@x...]
> > > Sent: Thursday, October 16, 2003 2:20 PM
> > > To: amibroker@xxxxxxxxxxxxxxx
> > > Subject: [amibroker] Optimization -- again
> > >
> > > Greetings --
> > >
> > > In my opinion, anything we do in development of trading systems
> > involves a
> > > search for a pattern than precedes a profitable trading
> > opportunity. Any
> > > time we examine the results of alternative systems, we are
> involved
> > in
> > > searching; and when we select the most promising of those
> > alternatives, we
> > > are optimizing. Only a system based on truly random entries
and
> > exits would
> > > not be the result some optimization. So the question
of "should
> we
> > > optimize?" is moot -- we have no choice but to optimize.
> > Consequently, we
> > > should be aware of our optimization techniques.
> > >
> > > Chuck referred to an optimization technique recommendation I
made
> > to the
> > > company we both worked for in Denver a few years ago. This is
a
> > short
> > > description of it.
> > >
> > > The company is a Commodity Trading Advisor which traded
futures,
> not
> > > individual stocks, but the procedures are equally valid for
both.
> > >
> > > When I joined the company, they were using very long data
series
> > when
> > > developing their models. They used a technique sometimes
called
> > folding or
> > > jackknifing, where the data was divided into several periods --
> say
> > ten.
> > > The modeling process made ten passes. During each pass, one
> period
> > was held
> > > back to be used as out-of-sample data, the other nine were used
> to
> > select
> > > the best parameter values. After all ten passes, the results
> were
> > gathered
> > > together and the parameter values that scored best overall were
> > chosen.
> > > There are several problems with this method. One is the
> difficulty
> > with the
> > > "ramp up" period at the start of each segment, another is that
it
> > is not
> > > valid to use older data for out-of-sample testing than was used
> for
> > > in-sample development, and another is that the data series were
> too
> > long.
> > > Chuck and I and others had many interesting discussions about
how
> > long the
> > > in-sample data should be.
> > >
> > > My background is strong in both the theory and the practice of
> > modeling and
> > > simulation, and includes a great deal of experience with
analysis
> of
> > > financial time series. I proposed the following method, which
I
> > continue to
> > > believe is valid.
> > >
> > > First, before any modeling begins. Using judgment of
management
> and
> > > comparison of trading profiles of many trading runs (real,
> > simulated, or
> > > imagined), pick an objective function by which the "goodness"
of
> a
> > trading
> > > system will be measured. This is important, it is a personal
or
> > corporate
> > > judgment, and it should not be subject to optimization.
> > >
> > > Divide each data series into a sequence of in-sample and out-of-
> > sample
> > > periods. The length of the out-of-sample period is
> > the "reoptimization"
> > > period. Say there are about ten years of historical data
> available
> > > (1/1/1993 through 1/1/2003. Set the in-sample period to two
> years
> > and the
> > > out-of-sample period to one year. Run the following sequence:
> > Search /
> > > optimize using 1993 and 1994; pick the "best" model for 1993-
> 1994;
> > forward
> > > test this model for 1995 and save the results; step forward one
> > > reoptimization period and repeat until all the full in-sample
> > periods have
> > > been used. The final optimization will have been 2001 and
2002,
> > with no
> > > out-of-sample data to test. Ignore all in-sample results!!
> > Examine the
> > > concatenated out-of-sample equity curve. If it is acceptable,
> you
> > have some
> > > confidence that the parameters select by the final optimization
> > (2001 and
> > > 2002) will be profitable for 2003. No guarantees -- only some
> > confidence.
> > >
> > > How did I pick two years for in-sample and one year for out-of-
> > sample? That
> > > was just an example. The method is to set up an automated
search
> > where the
> > > length of the in-sample period and the length of the out-of-
> sample
> > period --
> > > the reoptimization period -- are variables, and then search
> through
> > that
> > > space.
> > >
> > > Trading systems work because they identify inefficiencies in
> > markets. Every
> > > profitable trade reduces the inefficiency until, finally, the
> > trading system
> > > cannot overcome the frictional forces of commission and
> slippage.
> > This is
> > > the same phenomenon that physicists talk about as entropy.
> > >
> > > My feeling -- and it may be different than Chuck's -- is that
the
> > market is
> > > not only non-stationary, but that the probability that it will
> > return to a
> > > previous state is near zero.
> > >
> > > Being non-stationary means that market conditions change with
> > respect to our
> > > trading systems. If I am modeling a physical process, such as
a
> > chemical
> > > reaction, I can count on a predictable modelable output for a
> given
> > set of
> > > inputs. If I am modeling a financial time series, the output
> > following a
> > > given set of inputs changes over time. If a market were
> stationary
> > with
> > > respect to an RSI oscillator system, I could always buy a rise
of
> > the RSI
> > > through the 20 percent line, to use a very simplistic example.
> > >
> > > I feel that the introduction of microcomputers, trading system
> > development
> > > software, inexpensive individual brokerage accounts, and
> discussion
> > groups
> > > such as this one have permanently changed the realm of
trading.
> > One,
> > > everyone who is interested can afford to buy a computer, run
> > AmiBroker, and
> > > design and test trading systems. Two, if someone develops a
> > profitable
> > > system and trades it, the profits it takes reduce the potential
> > profits
> > > available to anyone else who trades it. Consequently, the
> > characteristics
> > > of the market change in a way that moves the market away from
> that
> > model
> > > until that trading system is no longer profitable enough to
> overcome
> > > commission and slippage. Three, a new person beginning to
study
> > trading
> > > system development typically tests a lot of old systems. If
one
> is
> > found to
> > > be profitable and they start trading it, the market moves back
to
> > being
> > > efficient. Consequently, trading systems that used to work,
but
> no
> > longer
> > > work, are very unlikely to ever work again.
> > >
> > > So, I feel that the in-sample period should be short so that
the
> > market
> > > conditions do not change much over that period. That is, I am
> > looking for a
> > > data series that is stationary relative to my model. The
> stationary
> > > relationship must extend beyond the in-sample period far enough
> > that the
> > > model will be profitable when used for trading in the out-of-
> sample
> > data.
> > > The length of the extension determines the reoptimization
> period.
> > It could
> > > be years, months, or even one day. Note that the holding
period
> of
> > a
> > > typical trade is very much related to the length of both the in-
> > sample and
> > > out-of-sample periods. The typical trade should be much
shorter
> > than the
> > > in-sample period and somewhat shorter than the out-of-sample
> period.
> > >
> > > The important point in all this is that the only results being
> > analyzed are
> > > the concatenated out-of-sample trades.
> > >
> > > As with all model development, every time I look at the out-of-
> > sample
> > > results in any way, I reduce the probability that future
trading
> > results
> > > will be profitable. That means that I should not perform
> thousands
> > of tests
> > > of model parameters, in-sample periods, and out-of-sample
> periods,
> > on the
> > > same data series and then pick the best model base on my
> > examination of
> > > thousands of out-of-sample results. In effect, I will have
just
> > converted
> > > all those out-of-sample results into in-sample data for another
> > step in the
> > > development. That is legitimate, just be aware of what is
> > happening.
> > >
> > > Thanks for listening,
> > > Howard
> > >
> > >
> > >
> > >
> > >
> > > Send BUG REPORTS to bugs@xxxx
> > > Send SUGGESTIONS to suggest@xxxx
> > > -----------------------------------------
> > > Post AmiQuote-related messages ONLY to:
amiquote@xxxxxxxxxxxxxxx
> > > (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> > > --------------------------------------------
> > > Check group FAQ at:
> > > http://groups.yahoo.com/group/amibroker/files/groupfaq.html
> > >
> > > Your use of Yahoo! Groups is subject to the Yahoo!
> > > <http://docs.yahoo.com/info/terms/> Terms of Service.
> > >
> > >
> > >
> > >
> > > Yahoo! Groups Sponsor
> > >
> > >
> > >
> > > ADVERTISEMENT
> > >
> > >
> >
>
<http://rd.yahoo.com/M=244522.3707890.4968055.1261774/D=egroupweb/S=17
> > 056321
> > >
> >
>
98:HM/A=1595056/R=0/SIG=124p07ne0/*http:/ashnin.com/clk/muryutaitakena
> > ttogyo
> > > ?YH=3707890&yhad=1595056> Click Here!
> > >
> > >
> > >
> > > <http://us.adserver.yahoo.com/l?
> > M=244522.3707890.4968055.1261774/D=egroupmai
> > > l/S=:HM/A=1595056/rand=910277976>
> > >
> > >
> > > Send BUG REPORTS to bugs@xxxx
> > > Send SUGGESTIONS to suggest@xxxx
> > > -----------------------------------------
> > > Post AmiQuote-related messages ONLY to:
amiquote@xxxxxxxxxxxxxxx
> > > (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> > > --------------------------------------------
> > > Check group FAQ at:
> > > http://groups.yahoo.com/group/amibroker/files/groupfaq.html
> > >
> > > Your use of Yahoo! Groups is subject to the Yahoo!
> > > <http://docs.yahoo.com/info/terms/> Terms of Service.
------------------------ Yahoo! Groups Sponsor ---------------------~-->
Rent DVDs Online - Over 14,500 titles.
No Late Fees & Free Shipping.
Try Netflix for FREE!
http://us.click.yahoo.com/Tq9otC/XP.FAA/3jkFAA/GHeqlB/TM
---------------------------------------------------------------------~->
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
|