[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[amibroker] Re: Optimization -- again



PureBytes Links

Trading Reference Links

Hi,

I came across something interesting from a famous trader:

Walk Forward Tested System Is Better than the Best 
Optimized One - Larry Williams 

It looks like the history of technical analysis has been largely 
influenced by optimization. That is, we studied the past, found 
something that looked significant, then optimized rules and 
procedures to trade the observation in the future.

Sometimes that has worked. Often it has not. That's our dilemma. What 
are we to do? In the past, we answered these questions by doing more 
optimization, more curve fitting. Indeed, we treated historical data 
like prisoners of war. Our thesis was, if you beat them often enough 
they would reveal anything. Which is true, but you want them to 
reveal everything, not anything.

This brings me to one point. I think we will all make much more 
headway with system development by spending less time on optimization 
and more time on walking systems and procedures forward.

If on a walk forward test, the system holds up, we probably have 
something. And for sure, what we have will be better than the very 
best optimized system when it comes to real time trading. Hence, 
let's see what we can learn from each other about conducting walk 
forward tests. Any ideas will be appreciated by all, I am certain.

Regards,

Pal
--- In amibroker@xxxxxxxxxxxxxxx, "Howard Bandy" <howardbandy@xxxx> 
wrote:
> Hi Dingo -
> 
>  
> 
> It's not quite that bad.  
> 
>  
> 
> My last paragraph is a warning for the common technique that many 
people try
> - looking at out-of-sample results so much that they become in-
sample data.
> That is OK, if there is another set of data that will be used for 
further
> testing.  Sometimes model development is done using three sets of 
data -
> learning, testing, and validation.  The learning data is extensively
> searched and used to select parameter values.  The testing data is 
used to
> determine when to stop searching and use the results found to be 
best so
> far.  For each set of parameters, a run is made on the learning 
data, and
> the value of the objective function is noted.  Periodically, the 
model using
> the best parameters is run against the test data.  When the 
performance over
> the test data falls off, that tells the optimizer to stop and 
return the
> parameters that worked best over the test data.  One (or a Very few)
> additional run is made against the validation data.  If results are
> acceptable, the model is used.  If the validation data produces poor
> results, do not use the model.  Rather, go back to the model design 
stage
> and come up with something new to try.
> 
>  
> 
> If the model is able to recognize profitable situations, and the 
parameters
> are stable, a wide range of parameters will be profitable.
> 
>  
> 
> One of my points in the posting is that the walk-forward testing 
with
> automatic selection of parameter values should be taken all 
together as a
> process.  If the results are good, then the process has validity 
and there
> is a reasonable expectation that trading the model and parameter set
> resulting from the last optimization will be profitable.
> 
>  
> 
> Thanks,
> 
> Howard
> 
>  
> 
> -----Original Message-----
> From: dingo [mailto:dingo@x...] 
> Sent: Thursday, October 16, 2003 12:02 PM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: RE: [amibroker] Optimization -- again
> 
>  
> 
> So according to your last paragraph it is impossbile to develop a 
system
> that is consistently successful since just looking for it voids 
it.  Right?
> 
> 
>  
> 
> d
> 
> -----Original Message-----
> From: Howard Bandy [mailto:howardbandy@x...] 
> Sent: Thursday, October 16, 2003 2:20 PM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: [amibroker] Optimization -- again
> 
> Greetings --
> 
> In my opinion, anything we do in development of trading systems 
involves a
> search for a pattern than precedes a profitable trading 
opportunity.  Any
> time we examine the results of alternative systems, we are involved 
in
> searching; and when we select the most promising of those 
alternatives, we
> are optimizing.  Only a system based on truly random entries and 
exits would
> not be the result some optimization.  So the question of "should we
> optimize?" is moot -- we have no choice but to optimize.  
Consequently, we
> should be aware of our optimization techniques.
> 
> Chuck referred to an optimization technique recommendation I made 
to the
> company we both worked for in Denver a few years ago.  This is a 
short
> description of it.
> 
> The company is a Commodity Trading Advisor which traded futures, not
> individual stocks, but the procedures are equally valid for both.
> 
> When I joined the company, they were using very long data series 
when
> developing their models.  They used a technique sometimes called 
folding or
> jackknifing, where the data was divided into several periods -- say 
ten.
> The modeling process made ten passes.  During each pass, one period 
was held
> back to be used as out-of-sample data, the other nine were used to 
select
> the best parameter values.  After all ten passes, the results were 
gathered
> together and the parameter values that scored best overall were 
chosen.
> There are several problems with this method.  One is the difficulty 
with the
> "ramp up" period at the start of each segment, another is that it 
is not
> valid to use older data for out-of-sample testing than was used for
> in-sample development, and another is that the data series were too 
long.
> Chuck and I and others had many interesting discussions about how 
long the
> in-sample data should be.  
> 
> My background is strong in both the theory and the practice of 
modeling and
> simulation, and includes a great deal of experience with analysis of
> financial time series.  I proposed the following method, which I 
continue to
> believe is valid.
> 
> First, before any modeling begins.  Using judgment of management and
> comparison of trading profiles of many trading runs (real, 
simulated, or
> imagined), pick an objective function by which the "goodness" of a 
trading
> system will be measured.  This is important, it is a personal or 
corporate
> judgment, and it should not be subject to optimization.  
> 
> Divide each data series into a sequence of in-sample and out-of-
sample
> periods.  The length of the out-of-sample period is 
the "reoptimization"
> period.  Say there are about ten years of historical data available
> (1/1/1993 through 1/1/2003.  Set the in-sample period to two years 
and the
> out-of-sample period to one year.  Run the following sequence:  
Search /
> optimize using 1993 and 1994; pick the "best" model for 1993-1994; 
forward
> test this model for 1995 and save the results; step forward one
> reoptimization period and repeat until all the full in-sample 
periods have
> been used.  The final optimization will have been 2001 and 2002, 
with no
> out-of-sample data to test.  Ignore all in-sample results!!  
Examine the
> concatenated out-of-sample equity curve.  If it is acceptable, you 
have some
> confidence that the parameters select by the final optimization 
(2001 and
> 2002) will be profitable for 2003.  No guarantees -- only some 
confidence.
> 
> How did I pick two years for in-sample and one year for out-of-
sample?  That
> was just an example.  The method is to set up an automated search 
where the
> length of the in-sample period and the length of the out-of-sample 
period --
> the reoptimization period -- are variables, and then search through 
that
> space.  
> 
> Trading systems work because they identify inefficiencies in 
markets.  Every
> profitable trade reduces the inefficiency until, finally, the 
trading system
> cannot overcome the frictional forces of commission and slippage.  
This is
> the same phenomenon that physicists talk about as entropy.
> 
> My feeling -- and it may be different than Chuck's -- is that the 
market is
> not only non-stationary, but that the probability that it will 
return to a
> previous state is near zero.  
> 
> Being non-stationary means that market conditions change with 
respect to our
> trading systems.  If I am modeling a physical process, such as a 
chemical
> reaction, I can count on a predictable modelable output for a given 
set of
> inputs.  If I am modeling a financial time series, the output 
following a
> given set of inputs changes over time.  If a market were stationary 
with
> respect to an RSI oscillator system, I could always buy a rise of 
the RSI
> through the 20 percent line, to use a very simplistic example. 
> 
> I feel that the introduction of microcomputers, trading system 
development
> software, inexpensive individual brokerage accounts, and discussion 
groups
> such as this one have permanently changed the realm of trading.  
One,
> everyone who is interested can afford to buy a computer, run 
AmiBroker, and
> design and test trading systems.  Two, if someone develops a 
profitable
> system and trades it, the profits it takes reduce the potential 
profits
> available to anyone else who trades it.  Consequently, the 
characteristics
> of the market change in a way that moves the market away from that 
model
> until that trading system is no longer profitable enough to overcome
> commission and slippage.  Three, a new person beginning to study 
trading
> system development typically tests a lot of old systems.  If one is 
found to
> be profitable and they start trading it, the market moves back to 
being
> efficient.  Consequently, trading systems that used to work, but no 
longer
> work, are very unlikely to ever work again.
> 
> So, I feel that the in-sample period should be short so that the 
market
> conditions do not change much over that period.  That is, I am 
looking for a
> data series that is stationary relative to my model.  The stationary
> relationship must extend beyond the in-sample period far enough 
that the
> model will be profitable when used for trading in the out-of-sample 
data.
> The length of the extension determines the reoptimization period.  
It could
> be years, months, or even one day.  Note that the holding period of 
a
> typical trade is very much related to the length of both the in-
sample and
> out-of-sample periods.  The typical trade should be much shorter 
than the
> in-sample period and somewhat shorter than the out-of-sample period.
> 
> The important point in all this is that the only results being 
analyzed are
> the concatenated out-of-sample trades.
> 
> As with all model development, every time I look at the out-of-
sample
> results in any way, I reduce the probability that future trading 
results
> will be profitable.  That means that I should not perform thousands 
of tests
> of model parameters, in-sample periods, and out-of-sample periods, 
on the
> same data series and then pick the best model base on my 
examination of
> thousands of out-of-sample results.  In effect, I will have just 
converted
> all those out-of-sample results into in-sample data for another 
step in the
> development.  That is legitimate, just be aware of what is 
happening.
> 
> Thanks for listening,
> Howard
> 
> 
> 
> 
> 
> Send BUG REPORTS to bugs@xxxx
> Send SUGGESTIONS to suggest@xxxx
> -----------------------------------------
> Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx 
> (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> --------------------------------------------
> Check group FAQ at:
> http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
> 
> Your use of Yahoo! Groups is subject to the Yahoo!
> <http://docs.yahoo.com/info/terms/>  Terms of Service. 
> 
>  
> 
> 
> Yahoo! Groups Sponsor
> 
> 
> 
> ADVERTISEMENT
>  
> 
<http://rd.yahoo.com/M=244522.3707890.4968055.1261774/D=egroupweb/S=17
056321
> 
98:HM/A=1595056/R=0/SIG=124p07ne0/*http:/ashnin.com/clk/muryutaitakena
ttogyo
> ?YH=3707890&yhad=1595056> Click Here!
> 
> 
>  
> <http://us.adserver.yahoo.com/l?
M=244522.3707890.4968055.1261774/D=egroupmai
> l/S=:HM/A=1595056/rand=910277976> 
> 
> 
> Send BUG REPORTS to bugs@xxxx
> Send SUGGESTIONS to suggest@xxxx
> -----------------------------------------
> Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx 
> (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> --------------------------------------------
> Check group FAQ at:
> http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
> 
> Your use of Yahoo! Groups is subject to the Yahoo!
> <http://docs.yahoo.com/info/terms/>  Terms of Service.


------------------------ Yahoo! Groups Sponsor ---------------------~-->
Buy Ink Cartridges or Refill Kits for your HP, Epson, Canon or Lexmark
Printer at MyInks.com. Free s/h on orders $50 or more to the US & Canada.
http://www.c1tracking.com/l.asp?cid=5511
http://us.click.yahoo.com/mOAaAA/3exGAA/qnsNAA/GHeqlB/TM
---------------------------------------------------------------------~->

Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx 
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/