[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] Robustivity



PureBytes Links

Trading Reference Links




Steve,
 
As you are probably aware....but for others on the 
list.
 
The first stage in the development of a trading 
system is its design.  This involves the development of a trading idea and 
then its implementation in some testable design. The second stage in this 
process is the preliminary testing of the trading model.
 
Testing comprises four main steps.
1. Verify that all the formulas and rules are being 
calculated as intended.
2. Asses whether the combinations of formulas and 
rules function as the theory would indicate.
3. Develop a preliminary idea of 
profitability.
4. Develop a preliminary idea of 
robustness.
 
Robustness is an 
important idea in trading model testing. The Webster's University dictionary 
provides the following definition: Robust : Powerfully 
built : Sturdy.
 
The key word is " Sturdy". In other words, a robust 
trading model is tough and long lasting. For our purposes, a robust trading 
model may be identified by three characteristics:
1. Profit over a wide range of 
variables.
2. Profit over a wide range of 
markets.
3. Profit over a wide range of market types and 
conditions.
 
In other words, a robust model will continue to 
perform profitably when markets change. Since markets change constantly, the 
more robust the model, the better.
<BLOCKQUOTE 
>
  ----- Original Message ----- 
  <DIV 
  >From: 
  <A title=kernish@xxxxxxxxxxx 
  href="">CedarCreekTrading 
  To: <A title=amibroker@xxxxxxxxxxxxxxx 
  href="">amibroker@xxxxxxxxxxxxxxx 
  Sent: Friday, October 31, 2003 4:12 
  PM
  Subject: Re: [amibroker] 
Robustivity
  
  Dave,
   
  Quite honestly, I haven't heard any definitions 
  of robust at this forum.  Lot's of folks are willing to say:  "This 
  is not robust...", but I have asked for specifics and definitions and nobody 
  is supplying the answers.  The system and how it's traded is not the 
  issue.  This is exactly why I decided to present the CMO5 on ETF's.  
  The concept isn't new and the data is OOS.  Without any tweaking or 
  modification, the approach continues to produce a reasonable equity 
  line.  
   
  Would you like for me to comment on the deviation 
  of the equity from a chosen linear regression?  Fred mentioned it would 
  be nice to see an equity curve (and there is one on each chart that I sent) 
  and that should count for one of Mark's nine robust measures.
   
  Since, I'm not bright enough to even guess at 
  nine measures of robustness (and so far, Mark hasn't shared), I must rely on 
  simple bench marks:  how does it perform over 8,000 issues, how does it 
  perform in different markets (futures, equities, ETF's), how does it perform 
  in different time periods, max eod dd's, max neg. excursions, avg. win/avg. 
  loss, expectancy, blah, blah, blah.
   
  Did I miss something or did someone actually 
  define robust?  If so, I hope it was better than the optimizing 
  thread.  Testing is necessary, optimizing is a spiraling death trap 
  (unless you name has DT as initials...then you understand the trap and play 
  the game on the edges of the black hole).  
   
  The CMO5 is a dandy indicator and the rules that 
  I apply to trigger the trades are simplistic.    Certainly, anyone 
  with decent software can better the results (well, maybe).  The trick is 
  to modify the approach ... without over-optimizing (tough trick).
   
  If over the next six months, this system, with 
  it's defined triggers, trades the DIA, QQQ, and SPY with decent profits...is 
  it robust?  How many years must it continue to produce steady results 
  before it can be called robust?  If we look back ten years from now, and 
  it has performed as it has in the last four years, is that robust?  Must 
  it also show the same kind of results in silver, bonds and soybeans?  If 
  it performs on EOD, data ... must it also knock em dead in one minute time 
  frames?  
   
  Here's a different view:  If a system 
  performs OOS for a period (x) of time and continues to perform into the 
  future, without adjustments...then I hereby declare that it is robust.  
  Why?  Because, just like technical analysis:  every input, that can 
  be considered, is discounted by an issue's price.  In technical analysis, 
  we discount all the information that flows in, out and around a stock and only 
  consider the price.  So, why not just judge a issue's robustness by 
  strictly pegging it to OOS results?  Period.  If these ETF's 
  continue to produce steady results for the next five years, I expect someone 
  to say:  "Well, maybe these ETF's with the CMO5 performed well for 
  the past eight years, but by my definition, this is not 
  robust". 
   
  I'm still waiting for the naysayers to define 
  robust and to post a system that can be monitored for results.  The CMO5 
  is the default indicator that I use to demonstrate the character of momentum 
  oscillators.  Many of us have come to realize that the indicator is not 
  the most important piece of the puzzle.  The grail is not a silver or 
  gold chalice.  
   
  Robust and optimizing are difficult terms to 
  define.  Any definition can be argued.  Robust is like beauty.  
  My wife is beautiful (my subjective opinion).  You might find her 
  unattractive.  
   
  The real silly part is that people want to argue 
  whether something is robust or not.  The rubber meets the road when you 
  trade.  Try flipping around sizeable accounts with something that is not 
  robust... money flies away.  
   
  Sorry that I haven't been able to answer your 
  question.  Why don't you direct you question to Mark.  He has a nine 
  point plan to evaluate what is or is not robust.  Is there a possibility 
  that one might over-optimize robustness?
   
  Take care,
   
  Steve
   
  <BLOCKQUOTE 
  >
    ----- Original Message ----- 
    <DIV 
    >From: 
    Dave Merrill 
    
    To: <A title=amibroker@xxxxxxxxxxxxxxx 
    href="">amibroker@xxxxxxxxxxxxxxx 
    Sent: Friday, October 31, 2003 12:33 
    PM
    Subject: RE: [amibroker] 
    Robustivity
    
    <SPAN 
    class=093340919-31102003>thanks for the specifics, glad to have 'em, even 
    though that's not what I was trying to ask (:-)
    <SPAN 
    class=093340919-31102003> 
    <SPAN 
    class=093340919-31102003>what I'm most interested in isn't the system 
    itself, but what you did to convince yourself that it's robust. the way it 
    was posted implied that you disagreed with other ideas of robust-ness/ivity 
    floating around here, so I'm trying to find out what your approach 
    is.
    <SPAN 
    class=093340919-31102003> 
    <SPAN 
    class=093340919-31102003>dave
    <SPAN 
    class=093340919-31102003> 
    <BLOCKQUOTE 
    >
      If I am using the CMO5 with triggers 
      of 34/-34, I would go back and start a test to evaluate this system and 
      triggers.  The starting period would be whatever date you pick 
      (1990, '97, 2000, etc.).  
       
      Next, I run the test over 315 trading 
      days (this period gives me results for approximately one year..it takes 
      "x" amount of periods to load the TRIX(21), which I use as a trend 
      identifier.  My approach produces about 10 to 15 round turn trades a 
      year... in each stock.  
       
      I then rank all issues by one 
      criteria:  percent return per day (while the money is in the 
      market).  If you only consider the percent per day contributions, I 
      think you will find that all other "book learned" ratios come out just 
      fine.  Numbers lie.  Would you rather trade a $100 stock that 
      returns $20 or a $20 stock that returns $10?  Percent per goes a long 
      way to normalizing the comparisons.
       
      I pick the 20 best percent per day 
      stocks and trade them for the next quarter.  At the end of the 
      quarter, I reevaluate the percentage per day contributions and reshuffle 
      the issues in play, if necessary.
       
      Symtems don't go bad, stocks and 
      commodities go bad.  Going bad is best defined by a change in the 
      pattern of supply and demand.  The cream rises to the top of the 
      list.    
       
      Is this optimizing?  Could be, 
      by some definitions.  If all the odds are even money, who would you 
      prefer to bet on:  Chicago or Kansas City?  KC is undefeated and 
      Chicago couldn't beat the local high school.  My money is on 
      KC.
       
      The stock betting setup is not 
      handicapped (like almost all games).  This is basically a even money 
      play (with subtractions for commission and 
      slippage...juice/vigorish).  If you have 9,000 issues to play, why 
      won't someone want to bet on the strongest performance?
       
      I know that the explanation might be 
      over simplified...but, the people who know me, in and out of this forum, 
      know that this is the way I do it.  I'm not crusading for 
      anything.  This works.  I've presented this simplistic approach 
      publicly to large groups and in a number of internet seminars. It 
      continues to crank out extraordinary profits. 
       
      Please let me know if the 
      paragraphs help to explain the ranking.
       
      Take care,
       
      Steve
       
       
      ----- Original Message ----- 
      <BLOCKQUOTE 
      >
        <DIV 
        >From: 
        Dave 
        Merrill 
        To: <A 
        title=amibroker@xxxxxxxxxxxxxxx 
        href="">amibroker@xxxxxxxxxxxxxxx 
        
        Sent: Friday, October 31, 2003 9:29 
        AM
        Subject: RE: [amibroker] 
        Robustivity
        
        <SPAN 
        class=496400216-31102003>steve, thanks for your 
        response.
        <SPAN 
        class=496400216-31102003> 
        <SPAN 
        class=496400216-31102003>from your msg subject and the way you presented 
        this system, I thought you were offering it as an example of one 
        you had objectively evaluated and determined to be robust. I 
        was interested in how you thought "robustivity" should be evaluated, 
        since you seemed to be contrasting your approach to walkforward 
        optimization and the various other system measures people were talking 
        about.
        <SPAN 
        class=496400216-31102003> 
        <SPAN 
        class=496400216-31102003>what I'm hearing in your response below isn't 
        what I would describe as a specific method for distinguishing 
        accidentally gorgeous backtest results from robustness. you do mention 
        testing also at faster time frames, which isn't a technique that's been 
        mentioned recently. but mostly, the robustness label here seems to come 
        from your integration of various aspects of your long experience with 
        it, like your visual sense of how it behaves. am I missing 
        something?
        <SPAN 
        class=496400216-31102003> 
        <SPAN 
        class=496400216-31102003>another question: you mention issue selection, 
        the idea of looking for stocks you think will trade well with a 
        particular indicator, rather than the other way around. how do you do 
        that? by measuring raw past growth trading that indicator? other 
        measures?
        <SPAN 
        class=496400216-31102003> 
        <SPAN 
        class=496400216-31102003>thanks again,
        <SPAN 
        class=496400216-31102003> 
        <SPAN 
        class=496400216-31102003>dave
        <BLOCKQUOTE 
        >
          <FONT face="Courier New" 
          color=#0000ff size=2>just for my understanding, in what sense is this 
          system "robust"? 
          <FONT face="Courier New" 
          color=#0000ff size=2> 
          Well, 
          first, this was presented to the public in the late 90's, at a series 
          of seminars that I conducted for Equis.  Same indicator, same 
          triggers, same everything.  This robust "thing" is a tough one to 
          define.  I'll try to explain what's important to me, but, it's 
          very subjective and just one person's opinion.  
          
          <FONT face=Arial 
          size=2> 
          <FONT face="Courier New" 
          color=#0000ff size=2>is it because 
          results are similar with different similar periods and 
          thresholds?
          <FONT face=Arial 
          size=2> 
          If you 
          take this CMO5 indicator and step down in time (5, 10, 60 
          minutes), you need to widen the triggers to obtain decent 
          results.  Other than that, it trades through time-zones with 
          very good results.
          <FONT face=Arial 
          size=2> 
          <FONT face="Courier New" 
          color=#0000ff size=2>that seems unlikely, since there isn't very far 
          to go from 5 to hit 1 and 0, which I'd guess are significantly 
          different. what sort of testing led you to decide on this period and 
          threshold, and this system for that matter?
          <FONT face=Arial 
          size=2> 
          If you're 
          referring to the CMO5...I first started testing it six years 
          ago.  I've tested and eyeballed every version of CMO(x).  
          I've created a few indicators that combines different periods of the 
          CMO.  For my money, for my style, this judge of momentum trades 
          more things, more accurately than any other indicator I am aware 
          of.  As I have begged many times:  give me something 
          better...I'll use it instead of this.
          <FONT face=Arial 
          size=2> 
          <FONT face="Courier New" 
          color=#0000ff size=2>is it robust 
          because it works well on many stocks, indexes and funds over a long 
          period of time? 
          <FONT face="Courier New" 
          color=#0000ff size=2><SPAN 
          class=468263723-30102003> 
          Yes, it 
          works well on many stocks and indexes.  I don't trade funds, but, 
          some fund managers, DTG members, use versions of the CMO to aid their 
          timing.  
          <FONT face=Arial 
          size=2> 
          <SPAN 
          class=468263723-30102003><FONT face="Courier New" color=#0000ff 
          size=2>because of the concepts behind the indicator 
          itself?
          <SPAN 
          class=468263723-30102003><FONT face="Courier New" color=#0000ff 
          size=2> 
          <SPAN 
          class=468263723-30102003>I process 
          visually.  The math is beyond me.  My bottom line has always 
          been the same:  give me an indicator that is smooth, yet 
          sensitive to intermediate and major market turns.  After gawking 
          hundreds of charts, everyday, for the last six years, I'm amazed at 
          how this indicator quantifies momentum.  I like versions of the 
          Stochastic RSI and the Standard Error Oscillator, but dollar for 
          dollar, the CMO does it for me.
          <SPAN 
          class=468263723-30102003><FONT face=Arial 
          size=2> 
          <SPAN 
          class=468263723-30102003><FONT face="Courier New" color=#0000ff 
          size=2>something 
          else?
          <SPAN 
          class=468263723-30102003><FONT face="Courier New" color=#0000ff 
          size=2><SPAN 
          class=468263723-30102003> 
          <SPAN 
          class=468263723-30102003><SPAN 
          class=468263723-30102003>I think there's a few other things to 
          mention.  First of all, the ETF's that I showed were chosen 
          because they represent a broad range of stocks and are popular trading 
          instruments.  Do I suggest trading these issues with this 
          system?  No way.  The CMO5 trades a lot of other issues with 
          better results than the ETF's.  I always allow the issues "to 
          pick themselves".  Trade the issues that return the greatest 
          percentages in a stable system.  
          
          <SPAN 
          class=468263723-30102003><SPAN 
          class=468263723-30102003> 
          <SPAN 
          class=468263723-30102003><SPAN 
          class=468263723-30102003>In it's stripped down version, as presented, 
          the CMO5 is an indicator that can return steady profits (see equity 
          lines) in it's rawest unoptimized form.  Is that robust?  
          
          <SPAN 
          class=468263723-30102003><SPAN 
          class=468263723-30102003> 
          <SPAN 
          class=468263723-30102003><SPAN 
          class=468263723-30102003>Robustness and optimizing/over-optimizing are 
          fascinating and misunderstood subjects.  Over the years, I've 
          constantly simplified my approaches.  I can improve on the 
          results of the three ETF's by simply "tweaking" the trigger 
          levels.  But, will it walk forward better than the default 
          triggers of 34/-34?  At least what I presented was out of 
          sample.  
          <SPAN 
          class=468263723-30102003><SPAN 
          class=468263723-30102003> 
          <SPAN 
          class=468263723-30102003><SPAN 
          class=468263723-30102003>If an approach does a good job of 
          identifying movement of supply and demand, the approach should not be 
          expected to work on all issues.  To say a system needs to 
          work on all  issues is total crap.   To say 
          that a system sucks because it doesn't work on XYZ is another large 
          pile.  Build simple things and concentrate on issue 
          selection.
          <SPAN 
          class=468263723-30102003><SPAN 
          class=468263723-30102003> 
          <FONT face=Arial 
          size=2>Optimization leads to dark and spooky places.  Ranking 
          leads you down the yellow brick road.
          <FONT face=Arial 
          size=2> 
          Take 
          care,
          <FONT face=Arial 
          size=2> 
          <FONT face=Arial 
          size=2>Steve
          <FONT face=Arial 
          size=2> 
          <BLOCKQUOTE 
          >
            <SPAN 
            class=468263723-30102003>steve, thanks for sharing this 
            (again).
            <SPAN 
            class=468263723-30102003> 
            <SPAN 
            class=468263723-30102003> 
            <SPAN 
            class=468263723-30102003>just for my understanding, in what sense is 
            this system "robust"? 
            <SPAN 
            class=468263723-30102003> 
            <SPAN 
            class=468263723-30102003>is it because results are similar with 
            different similar periods and thresholds? that seems unlikely, since 
            there isn't very far to go from 5 to hit 1 and 0, which I'd guess 
            are significantly different. what sort of testing led you to decide 
            on this period and threshold, and this system for that 
            matter?
            <SPAN 
            class=468263723-30102003> 
            <SPAN 
            class=468263723-30102003>is it robust because it works well on many 
            stocks, indexes and funds over a long period of time? 
            
            <SPAN 
            class=468263723-30102003> 
            <SPAN 
            class=468263723-30102003>because of the concepts behind the 
            indicator itself?
            <SPAN 
            class=468263723-30102003> 
            <SPAN 
            class=468263723-30102003>something else?
            <SPAN 
            class=468263723-30102003> 
            <SPAN 
            class=468263723-30102003> 
            <SPAN 
            class=468263723-30102003>I'm not disputing the system's value, which 
            I haven't tested yet. I'm trying to understand what kind of process 
            you go through to settle on a system and 
            settings.
            <SPAN 
            class=468263723-30102003> 
            <SPAN 
            class=468263723-30102003>thanks,
            <SPAN 
            class=468263723-30102003> 
            <SPAN 
            class=468263723-30102003>dave
            <SPAN 
            class=468263723-30102003> 
            <BLOCKQUOTE 
            >
              1.  This exact system was 
              presented over a year ago at this forum
              2.  The charts are OOS (since, 
              it's been posted publicly forever)
              3.  Rules are simple:  Buy 
              the opening of the next day when the CMO5 closes below -34 and 
              sell when it triggers above 34.
               
              Works on most issues (raw).  
              Works better if:  
               
              a.  You take trades only with 
              the trend
              b.  You protect yourself from 
              large drawdowns (stop)
              c.  You conjure a profit target 
              (limit)
              d.  You put in a time stop 
              
               
              This is the guts of an indicator and 
              a logical systematic approach.  Whistles and bells are 
              optional (but, in my opinion necessary).  Again, if you start 
              with a pig, the prom dress doesn't make it look any better.  
              Don't hang ornaments on a twisted Christmas 
            tree.Send 
        BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to 
        suggest@xxxxxxxxxxxxx-----------------------------------------Post 
        AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web 
        page: <A 
        href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check 
        group FAQ at: <A 
        href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
        Your use of Yahoo! Groups is subject to the <A 
        href="">Yahoo! Terms of 
        Service. Send BUG REPORTS to 
      bugs@xxxxxxxxxxxxxSend SUGGESTIONS to 
      suggest@xxxxxxxxxxxxx-----------------------------------------Post 
      AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: 
      <A 
      href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check 
      group FAQ at: <A 
      href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
      Your use of Yahoo! Groups is subject to the <A 
      href="">Yahoo! Terms of Service. 
      Send BUG REPORTS to bugs@xxxxxxxxxxxxxSend 
    SUGGESTIONS to 
    suggest@xxxxxxxxxxxxx-----------------------------------------Post 
    AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: 
    <A 
    href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check 
    group FAQ at: <A 
    href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
    Your use of Yahoo! Groups is subject to the <A 
    href="">Yahoo! Terms of Service. 
    
  Send BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to 
  suggest@xxxxxxxxxxxxx-----------------------------------------Post 
  AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: <A 
  href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check 
  group FAQ at: <A 
  href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
  Your use of Yahoo! Groups is subject to the <A 
  href="">Yahoo! Terms of Service. 
  
<BLOCKQUOTE 
><FONT 
  face="Courier New">---Outgoing mail is certified Virus 
  Free.Checked by AVG anti-virus system (<A 
  href="">http://www.grisoft.com).Version: 6.0.532 
  / Virus Database: 326 - Release Date: 
10/27/2003






Yahoo! Groups Sponsor


  ADVERTISEMENT 









Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx 
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html



Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.