[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] Re: Optimization -- again



PureBytes Links

Trading Reference Links




Whatever Larry says has great validity, as it's 
coming from probably the most famous and most knowledgeable trader of modern 
times.
Certainly, you jest.
 
Take care,
 
Steve
<BLOCKQUOTE 
>
  ----- Original Message ----- 
  <DIV 
  >From: 
  palsanand 
  
  To: <A title=amibroker@xxxxxxxxxxxxxxx 
  href="">amibroker@xxxxxxxxxxxxxxx 
  Sent: Sunday, October 19, 2003 10:45 
  AM
  Subject: [amibroker] Re: Optimization -- 
  again
  Hi All,About Larry Williams' contribution in 
  support of Walk Forward Testing: Whatever Larry says has great validity, 
  as it's coming from probably the most famous and most knowledgeable trader 
  of modern times.It's certainly correct that if done properly, walk 
  forward testing has great value. For those of you not aware of walk 
  forward testing, it's first setting your system parameters and then 
  testing the results in the future using those pre-set parameters without 
  benefit of additional or new optimization. Some people refer to that 
  as "hypothetical real-time trading." However, walk forward testing 
  can in fact be a trap if done incorrectly. That's because there's a 
  problem in deciding what pre-set algorithm or parameters to use prior to 
  the so-called walk forward test. If we arrive at those parameters by an 
  optimization process, then we may be guilty of optimizing the walk forward 
  test without even realizing we have done that. Another pitfall, is the 
  great tendency to optimize the walk forward testing time period 
  itself. Possibly the only way to do it correctly, is to first 
  arrive at a set of parameters and algorithm based on logic, experience, or 
  sound trading principals that won't be subject to change. Then do a walk 
  forward with no attempt to improve results via 
  optimization.Regards,Pal--- In amibroker@xxxxxxxxxxxxxxx, 
  "palsanand" <palsanand@xxxx> wrote:> Hi,> > I came 
  across something interesting from a famous trader:> > Walk 
  Forward Tested System Is Better than the Best > Optimized One - Larry 
  Williams > > It looks like the history of technical analysis has 
  been largely > influenced by optimization. That is, we studied the 
  past, found > something that looked significant, then optimized rules 
  and > procedures to trade the observation in the future.> 
  > Sometimes that has worked. Often it has not. That's our dilemma. 
  What > are we to do? In the past, we answered these questions by 
  doing more > optimization, more curve fitting. Indeed, we treated 
  historical data > like prisoners of war. Our thesis was, if you 
  beat them often enough > they would reveal anything. Which is true, 
  but you want them to > reveal everything, not anything.> 
  > This brings me to one point. I think we will all make much more 
  > headway with system development by spending less time on 
  optimization > and more time on walking systems and procedures 
  forward.> > If on a walk forward test, the system holds up, we 
  probably have > something. And for sure, what we have will be better 
  than the very > best optimized system when it comes to real time 
  trading. Hence, > let's see what we can learn from each other about 
  conducting walk > forward tests. Any ideas will be appreciated by all, 
  I am certain.> > Regards,> > Pal> --- In 
  amibroker@xxxxxxxxxxxxxxx, "Howard Bandy" <howardbandy@xxxx> > 
  wrote:> > Hi Dingo -> > > >  > > 
  > > It's not quite that bad.  > > > >  
  > > > > My last paragraph is a warning for the common 
  technique that many > people try> > - looking at 
  out-of-sample results so much that they become in-> sample 
  data.> > That is OK, if there is another set of data that will be 
  used for > further> > testing.  Sometimes model 
  development is done using three sets of > data -> > learning, 
  testing, and validation.  The learning data is extensively> 
  > searched and used to select parameter values.  The testing data 
  is > used to> > determine when to stop searching and use 
  the results found to be > best so> > far.  For each set 
  of parameters, a run is made on the learning > data, and> > 
  the value of the objective function is noted.  Periodically, the > 
  model using> > the best parameters is run against the test 
  data.  When the > performance over> > the test data 
  falls off, that tells the optimizer to stop and > return the> 
  > parameters that worked best over the test data.  One (or a Very 
  few)> > additional run is made against the validation 
  data.  If results are> > acceptable, the model is 
  used.  If the validation data produces poor> > results, do 
  not use the model.  Rather, go back to the model design > 
  stage> > and come up with something new to try.> > 
  > >  > > > > If the model is able to 
  recognize profitable situations, and the > parameters> > are 
  stable, a wide range of parameters will be profitable.> > > 
  >  > > > > One of my points in the posting is that 
  the walk-forward testing > with> > automatic selection of 
  parameter values should be taken all > together as a> > 
  process.  If the results are good, then the process has validity > 
  and there> > is a reasonable expectation that trading the model and 
  parameter set> > resulting from the last optimization will be 
  profitable.> > > >  > > > > 
  Thanks,> > > > Howard> > > >  
  > > > > -----Original Message-----> > From: 
  dingo [mailto:dingo@xxxx] > > Sent: Thursday, October 16, 2003 12:02 
  PM> > To: amibroker@xxxxxxxxxxxxxxx> > Subject: RE: 
  [amibroker] Optimization -- again> > > >  > 
  > > > So according to your last paragraph it is impossbile to 
  develop a > system> > that is consistently successful since 
  just looking for it voids > it.  Right?> > > > 
  > >  > > > > d> > > > 
  -----Original Message-----> > From: Howard Bandy 
  [mailto:howardbandy@xxxx] > > Sent: Thursday, October 16, 2003 2:20 
  PM> > To: amibroker@xxxxxxxxxxxxxxx> > Subject: 
  [amibroker] Optimization -- again> > > > Greetings 
  --> > > > In my opinion, anything we do in development of 
  trading systems > involves a> > search for a pattern than 
  precedes a profitable trading > opportunity.  Any> > 
  time we examine the results of alternative systems, we are involved 
  > in> > searching; and when we select the most promising of 
  those > alternatives, we> > are optimizing.  Only a 
  system based on truly random entries and > exits would> > not 
  be the result some optimization.  So the question of "should 
  we> > optimize?" is moot -- we have no choice but to 
  optimize.  > Consequently, we> > should be aware of our 
  optimization techniques.> > > > Chuck referred to an 
  optimization technique recommendation I made > to the> > 
  company we both worked for in Denver a few years ago.  This is a > 
  short> > description of it.> > > > The company 
  is a Commodity Trading Advisor which traded futures, not> > 
  individual stocks, but the procedures are equally valid for both.> > 
  > > When I joined the company, they were using very long data series 
  > when> > developing their models.  They used a 
  technique sometimes called > folding or> > jackknifing, where 
  the data was divided into several periods -- say > ten.> 
  > The modeling process made ten passes.  During each pass, one 
  period > was held> > back to be used as out-of-sample 
  data, the other nine were used to > select> > the best 
  parameter values.  After all ten passes, the results were > 
  gathered> > together and the parameter values that scored best 
  overall were > chosen.> > There are several problems with 
  this method.  One is the difficulty > with the> > 
  "ramp up" period at the start of each segment, another is that it > is 
  not> > valid to use older data for out-of-sample testing than was 
  used for> > in-sample development, and another is that the data 
  series were too > long.> > Chuck and I and others had 
  many interesting discussions about how > long the> > 
  in-sample data should be.  > > > > My background is 
  strong in both the theory and the practice of > modeling and> 
  > simulation, and includes a great deal of experience with analysis 
  of> > financial time series.  I proposed the following 
  method, which I > continue to> > believe is valid.> 
  > > > First, before any modeling begins.  Using judgment of 
  management and> > comparison of trading profiles of many trading 
  runs (real, > simulated, or> > imagined), pick an objective 
  function by which the "goodness" of a > trading> > system 
  will be measured.  This is important, it is a personal or > 
  corporate> > judgment, and it should not be subject to 
  optimization.  > > > > Divide each data series into a 
  sequence of in-sample and out-of-> sample> > periods.  
  The length of the out-of-sample period is > the 
  "reoptimization"> > period.  Say there are about ten years of 
  historical data available> > (1/1/1993 through 1/1/2003.  
  Set the in-sample period to two years > and the> > 
  out-of-sample period to one year.  Run the following sequence:  
  > Search /> > optimize using 1993 and 1994; pick the "best" 
  model for 1993-1994; > forward> > test this model for 
  1995 and save the results; step forward one> > reoptimization period 
  and repeat until all the full in-sample > periods have> > 
  been used.  The final optimization will have been 2001 and 2002, > 
  with no> > out-of-sample data to test.  Ignore all in-sample 
  results!!  > Examine the> > concatenated out-of-sample 
  equity curve.  If it is acceptable, you > have some> 
  > confidence that the parameters select by the final optimization > 
  (2001 and> > 2002) will be profitable for 2003.  No guarantees 
  -- only some > confidence.> > > > How did I pick 
  two years for in-sample and one year for out-of-> sample?  
  That> > was just an example.  The method is to set up an 
  automated search > where the> > length of the in-sample 
  period and the length of the out-of-sample > period --> > 
  the reoptimization period -- are variables, and then search through 
  > that> > space.  > > > > Trading 
  systems work because they identify inefficiencies in > markets.  
  Every> > profitable trade reduces the inefficiency until, finally, 
  the > trading system> > cannot overcome the frictional forces 
  of commission and slippage.  > This is> > the same 
  phenomenon that physicists talk about as entropy.> > > > 
  My feeling -- and it may be different than Chuck's -- is that the > 
  market is> > not only non-stationary, but that the probability that 
  it will > return to a> > previous state is near zero.  
  > > > > Being non-stationary means that market conditions 
  change with > respect to our> > trading systems.  If I 
  am modeling a physical process, such as a > chemical> > 
  reaction, I can count on a predictable modelable output for a given 
  > set of> > inputs.  If I am modeling a financial time 
  series, the output > following a> > given set of inputs 
  changes over time.  If a market were stationary > with> 
  > respect to an RSI oscillator system, I could always buy a rise of 
  > the RSI> > through the 20 percent line, to use a very 
  simplistic example. > > > > I feel that the introduction 
  of microcomputers, trading system > development> > software, 
  inexpensive individual brokerage accounts, and discussion > 
  groups> > such as this one have permanently changed the realm of 
  trading.  > One,> > everyone who is interested can 
  afford to buy a computer, run > AmiBroker, and> > design and 
  test trading systems.  Two, if someone develops a > 
  profitable> > system and trades it, the profits it takes reduce the 
  potential > profits> > available to anyone else who trades 
  it.  Consequently, the > characteristics> > of the 
  market change in a way that moves the market away from that > 
  model> > until that trading system is no longer profitable enough to 
  overcome> > commission and slippage.  Three, a new person 
  beginning to study > trading> > system development typically 
  tests a lot of old systems.  If one is > found to> > 
  be profitable and they start trading it, the market moves back to > 
  being> > efficient.  Consequently, trading systems that used to 
  work, but no > longer> > work, are very unlikely to ever 
  work again.> > > > So, I feel that the in-sample period 
  should be short so that the > market> > conditions do not 
  change much over that period.  That is, I am > looking for 
  a> > data series that is stationary relative to my model.  The 
  stationary> > relationship must extend beyond the in-sample 
  period far enough > that the> > model will be profitable when 
  used for trading in the out-of-sample > data.> > The 
  length of the extension determines the reoptimization period.  
  > It could> > be years, months, or even one day.  Note 
  that the holding period of > a> > typical trade is very 
  much related to the length of both the in-> sample and> > 
  out-of-sample periods.  The typical trade should be much shorter > 
  than the> > in-sample period and somewhat shorter than the 
  out-of-sample period.> > > > The important point in 
  all this is that the only results being > analyzed are> > the 
  concatenated out-of-sample trades.> > > > As with all 
  model development, every time I look at the out-of-> sample> 
  > results in any way, I reduce the probability that future trading > 
  results> > will be profitable.  That means that I should not 
  perform thousands > of tests> > of model parameters, 
  in-sample periods, and out-of-sample periods, > on the> > 
  same data series and then pick the best model base on my > examination 
  of> > thousands of out-of-sample results.  In effect, I will 
  have just > converted> > all those out-of-sample results into 
  in-sample data for another > step in the> > 
  development.  That is legitimate, just be aware of what is > 
  happening.> > > > Thanks for listening,> > 
  Howard> > > > > > > > > > 
  > > Send BUG REPORTS to bugs@xxxx> > Send SUGGESTIONS to 
  suggest@xxxx> > -----------------------------------------> 
  > Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx > 
  > (Web page: <A 
  href="">http://groups.yahoo.com/group/amiquote/messages/)> 
  > --------------------------------------------> > Check group FAQ 
  at:> > <A 
  href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
  > > > > Your use of Yahoo! Groups is subject to the 
  Yahoo!> > <<A 
  href="">http://docs.yahoo.com/info/terms/>  
  Terms of Service. > > > >  > > > 
  > > > Yahoo! Groups Sponsor> > > > > 
  > > > ADVERTISEMENT> >  > > > 
  <<A 
  href="">http://rd.yahoo.com/M=244522.3707890.4968055.1261774/D=egroupweb/S=17> 
  056321> > > 
  98:HM/A=1595056/R=0/SIG=124p07ne0/*http:/ashnin.com/clk/muryutaitakena> 
  ttogyo> > ?YH=3707890&yhad=1595056> Click Here!> > 
  > > > >  > > <<A 
  href="">http://us.adserver.yahoo.com/l?> 
  M=244522.3707890.4968055.1261774/D=egroupmai> > 
  l/S=:HM/A=1595056/rand=910277976> > > > > > > 
  Send BUG REPORTS to bugs@xxxx> > Send SUGGESTIONS to 
  suggest@xxxx> > -----------------------------------------> 
  > Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx > 
  > (Web page: <A 
  href="">http://groups.yahoo.com/group/amiquote/messages/)> 
  > --------------------------------------------> > Check group FAQ 
  at:> > <A 
  href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
  > > > > Your use of Yahoo! Groups is subject to the 
  Yahoo!> > <<A 
  href="">http://docs.yahoo.com/info/terms/>  
  Terms of Service.Send 
  BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to 
  suggest@xxxxxxxxxxxxx-----------------------------------------Post 
  AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: <A 
  href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check 
  group FAQ at: <A 
  href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
  Your use of Yahoo! Groups is subject to the <A 
  href="">Yahoo! Terms of Service. 







Yahoo! Groups Sponsor


  ADVERTISEMENT 









Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx 
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html



Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.