[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[amibroker] Re: Robustivity



PureBytes Links

Trading Reference Links

Over-optimize robustness ?!

That's an interesting concept, but I think not possible due to mutual 
exclusivity but I suspect at least someone out there won't agree.

--- In amibroker@xxxxxxxxxxxxxxx, "CedarCreekTrading" <kernish@xxxx> 
wrote:
> Dave,
> 
> Quite honestly, I haven't heard any definitions of robust at this 
forum.  Lot's of folks are willing to say:  "This is not robust...", 
but I have asked for specifics and definitions and nobody is 
supplying the answers.  The system and how it's traded is not the 
issue.  This is exactly why I decided to present the CMO5 on ETF's.  
The concept isn't new and the data is OOS.  Without any tweaking or 
modification, the approach continues to produce a reasonable equity 
line.  
> 
> Would you like for me to comment on the deviation of the equity 
from a chosen linear regression?  Fred mentioned it would be nice to 
see an equity curve (and there is one on each chart that I sent) and 
that should count for one of Mark's nine robust measures.
> 
> Since, I'm not bright enough to even guess at nine measures of 
robustness (and so far, Mark hasn't shared), I must rely on simple 
bench marks:  how does it perform over 8,000 issues, how does it 
perform in different markets (futures, equities, ETF's), how does it 
perform in different time periods, max eod dd's, max neg. excursions, 
avg. win/avg. loss, expectancy, blah, blah, blah.
> 
> Did I miss something or did someone actually define robust?  If so, 
I hope it was better than the optimizing thread.  Testing is 
necessary, optimizing is a spiraling death trap (unless you name has 
DT as initials...then you understand the trap and play the game on 
the edges of the black hole).  
> 
> The CMO5 is a dandy indicator and the rules that I apply to trigger 
the trades are simplistic.    Certainly, anyone with decent software 
can better the results (well, maybe).  The trick is to modify the 
approach ... without over-optimizing (tough trick).
> 
> If over the next six months, this system, with it's defined 
triggers, trades the DIA, QQQ, and SPY with decent profits...is it 
robust?  How many years must it continue to produce steady results 
before it can be called robust?  If we look back ten years from now, 
and it has performed as it has in the last four years, is that 
robust?  Must it also show the same kind of results in silver, bonds 
and soybeans?  If it performs on EOD, data ... must it also knock em 
dead in one minute time frames?  
> 
> Here's a different view:  If a system performs OOS for a period (x) 
of time and continues to perform into the future, without 
adjustments...then I hereby declare that it is robust.  Why?  
Because, just like technical analysis:  every input, that can be 
considered, is discounted by an issue's price.  In technical 
analysis, we discount all the information that flows in, out and 
around a stock and only consider the price.  So, why not just judge a 
issue's robustness by strictly pegging it to OOS results?  Period.  
If these ETF's continue to produce steady results for the next five 
years, I expect someone to say:  "Well, maybe these ETF's with the 
CMO5 performed well for the past eight years, but by my definition, 
this is not robust". 
> 
> I'm still waiting for the naysayers to define robust and to post a 
system that can be monitored for results.  The CMO5 is the default 
indicator that I use to demonstrate the character of momentum 
oscillators.  Many of us have come to realize that the indicator is 
not the most important piece of the puzzle.  The grail is not a 
silver or gold chalice.  
> 
> Robust and optimizing are difficult terms to define.  Any 
definition can be argued.  Robust is like beauty.  My wife is 
beautiful (my subjective opinion).  You might find her unattractive.  
> 
> The real silly part is that people want to argue whether something 
is robust or not.  The rubber meets the road when you trade.  Try 
flipping around sizeable accounts with something that is not 
robust... money flies away.  
> 
> Sorry that I haven't been able to answer your question.  Why don't 
you direct you question to Mark.  He has a nine point plan to 
evaluate what is or is not robust.  Is there a possibility that one 
might over-optimize robustness?
> 
> Take care,
> 
> Steve
> 
>   ----- Original Message ----- 
>   From: Dave Merrill 
>   To: amibroker@xxxxxxxxxxxxxxx 
>   Sent: Friday, October 31, 2003 12:33 PM
>   Subject: RE: [amibroker] Robustivity
> 
> 
>   thanks for the specifics, glad to have 'em, even though that's 
not what I was trying to ask (:-)
> 
>   what I'm most interested in isn't the system itself, but what you 
did to convince yourself that it's robust. the way it was posted 
implied that you disagreed with other ideas of robust-ness/ivity 
floating around here, so I'm trying to find out what your approach is.
> 
>   dave
> 
>     If I am using the CMO5 with triggers of 34/-34, I would go back 
and start a test to evaluate this system and triggers.  The starting 
period would be whatever date you pick (1990, '97, 2000, etc.).  
> 
>     Next, I run the test over 315 trading days (this period gives 
me results for approximately one year..it takes "x" amount of periods 
to load the TRIX(21), which I use as a trend identifier.  My approach 
produces about 10 to 15 round turn trades a year... in each stock.  
> 
>     I then rank all issues by one criteria:  percent return per day 
(while the money is in the market).  If you only consider the percent 
per day contributions, I think you will find that all other "book 
learned" ratios come out just fine.  Numbers lie.  Would you rather 
trade a $100 stock that returns $20 or a $20 stock that returns $10?  
Percent per goes a long way to normalizing the comparisons.
> 
>     I pick the 20 best percent per day stocks and trade them for 
the next quarter.  At the end of the quarter, I reevaluate the 
percentage per day contributions and reshuffle the issues in play, if 
necessary.
> 
>     Symtems don't go bad, stocks and commodities go bad.  Going bad 
is best defined by a change in the pattern of supply and demand.  The 
cream rises to the top of the list.    
> 
>     Is this optimizing?  Could be, by some definitions.  If all the 
odds are even money, who would you prefer to bet on:  Chicago or 
Kansas City?  KC is undefeated and Chicago couldn't beat the local 
high school.  My money is on KC.
> 
>     The stock betting setup is not handicapped (like almost all 
games).  This is basically a even money play (with subtractions for 
commission and slippage...juice/vigorish).  If you have 9,000 issues 
to play, why won't someone want to bet on the strongest performance?
> 
>     I know that the explanation might be over simplified...but, the 
people who know me, in and out of this forum, know that this is the 
way I do it.  I'm not crusading for anything.  This works.  I've 
presented this simplistic approach publicly to large groups and in a 
number of internet seminars. It continues to crank out extraordinary 
profits. 
> 
>     Please let me know if the paragraphs help to explain the 
ranking.
> 
>     Take care,
> 
>     Steve
> 
> 
>     ----- Original Message ----- 
>       From: Dave Merrill 
>       To: amibroker@xxxxxxxxxxxxxxx 
>       Sent: Friday, October 31, 2003 9:29 AM
>       Subject: RE: [amibroker] Robustivity
> 
> 
>       steve, thanks for your response.
> 
>       from your msg subject and the way you presented this system, 
I thought you were offering it as an example of one you had 
objectively evaluated and determined to be robust. I was interested 
in how you thought "robustivity" should be evaluated, since you 
seemed to be contrasting your approach to walkforward optimization 
and the various other system measures people were talking about.
> 
>       what I'm hearing in your response below isn't what I would 
describe as a specific method for distinguishing accidentally 
gorgeous backtest results from robustness. you do mention testing 
also at faster time frames, which isn't a technique that's been 
mentioned recently. but mostly, the robustness label here seems to 
come from your integration of various aspects of your long experience 
with it, like your visual sense of how it behaves. am I missing 
something?
> 
>       another question: you mention issue selection, the idea of 
looking for stocks you think will trade well with a particular 
indicator, rather than the other way around. how do you do that? by 
measuring raw past growth trading that indicator? other measures?
> 
>       thanks again,
> 
>       dave
>         just for my understanding, in what sense is this 
system "robust"? 
> 
>         Well, first, this was presented to the public in the late 
90's, at a series of seminars that I conducted for Equis.  Same 
indicator, same triggers, same everything.  This robust "thing" is a 
tough one to define.  I'll try to explain what's important to me, 
but, it's very subjective and just one person's opinion.  
> 
>         is it because results are similar with different similar 
periods and thresholds?
> 
>         If you take this CMO5 indicator and step down in time (5, 
10, 60 minutes), you need to widen the triggers to obtain decent 
results.  Other than that, it trades through time-zones with very 
good results.
> 
>         that seems unlikely, since there isn't very far to go from 
5 to hit 1 and 0, which I'd guess are significantly different. what 
sort of testing led you to decide on this period and threshold, and 
this system for that matter?
> 
>         If you're referring to the CMO5...I first started testing 
it six years ago.  I've tested and eyeballed every version of CMO
(x).  I've created a few indicators that combines different periods 
of the CMO.  For my money, for my style, this judge of momentum 
trades more things, more accurately than any other indicator I am 
aware of.  As I have begged many times:  give me something 
better...I'll use it instead of this.
> 
>         is it robust because it works well on many stocks, indexes 
and funds over a long period of time? 
> 
>         Yes, it works well on many stocks and indexes.  I don't 
trade funds, but, some fund managers, DTG members, use versions of 
the CMO to aid their timing.  
> 
>         because of the concepts behind the indicator itself?
> 
>         I process visually.  The math is beyond me.  My bottom line 
has always been the same:  give me an indicator that is smooth, yet 
sensitive to intermediate and major market turns.  After gawking 
hundreds of charts, everyday, for the last six years, I'm amazed at 
how this indicator quantifies momentum.  I like versions of the 
Stochastic RSI and the Standard Error Oscillator, but dollar for 
dollar, the CMO does it for me.
> 
>         something else?
> 
>         I think there's a few other things to mention.  First of 
all, the ETF's that I showed were chosen because they represent a 
broad range of stocks and are popular trading instruments.  Do I 
suggest trading these issues with this system?  No way.  The CMO5 
trades a lot of other issues with better results than the ETF's.  I 
always allow the issues "to pick themselves".  Trade the issues that 
return the greatest percentages in a stable system.  
> 
>         In it's stripped down version, as presented, the CMO5 is an 
indicator that can return steady profits (see equity lines) in it's 
rawest unoptimized form.  Is that robust?  
> 
>         Robustness and optimizing/over-optimizing are fascinating 
and misunderstood subjects.  Over the years, I've constantly 
simplified my approaches.  I can improve on the results of the three 
ETF's by simply "tweaking" the trigger levels.  But, will it walk 
forward better than the default triggers of 34/-34?  At least what I 
presented was out of sample.  
> 
>         If an approach does a good job of identifying movement of 
supply and demand, the approach should not be expected to work on all 
issues.  To say a system needs to work on all  issues is total 
crap.   To say that a system sucks because it doesn't work on XYZ is 
another large pile.  Build simple things and concentrate on issue 
selection.
> 
>         Optimization leads to dark and spooky places.  Ranking 
leads you down the yellow brick road.
> 
>         Take care,
> 
>         Steve
> 
>           steve, thanks for sharing this (again).
> 
> 
>           just for my understanding, in what sense is this 
system "robust"? 
> 
>           is it because results are similar with different similar 
periods and thresholds? that seems unlikely, since there isn't very 
far to go from 5 to hit 1 and 0, which I'd guess are significantly 
different. what sort of testing led you to decide on this period and 
threshold, and this system for that matter?
> 
>           is it robust because it works well on many stocks, 
indexes and funds over a long period of time? 
> 
>           because of the concepts behind the indicator itself?
> 
>           something else?
> 
> 
>           I'm not disputing the system's value, which I haven't 
tested yet. I'm trying to understand what kind of process you go 
through to settle on a system and settings.
> 
>           thanks,
> 
>           dave
> 
>             1.  This exact system was presented over a year ago at 
this forum
>             2.  The charts are OOS (since, it's been posted 
publicly forever)
>             3.  Rules are simple:  Buy the opening of the next day 
when the CMO5 closes below -34 and sell when it triggers above 34.
> 
>             Works on most issues (raw).  Works better if:  
> 
>             a.  You take trades only with the trend
>             b.  You protect yourself from large drawdowns (stop)
>             c.  You conjure a profit target (limit)
>             d.  You put in a time stop 
> 
>             This is the guts of an indicator and a logical 
systematic approach.  Whistles and bells are optional (but, in my 
opinion necessary).  Again, if you start with a pig, the prom dress 
doesn't make it look any better.  Don't hang ornaments on a twisted 
Christmas tree.
> 
> 
>       Send BUG REPORTS to bugs@xxxx
>       Send SUGGESTIONS to suggest@xxxx
>       -----------------------------------------
>       Post AmiQuote-related messages ONLY to: 
amiquote@xxxxxxxxxxxxxxx 
>       (Web page: http://groups.yahoo.com/group/amiquote/messages/)
>       --------------------------------------------
>       Check group FAQ at: 
http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
> 
>       Your use of Yahoo! Groups is subject to the Yahoo! Terms of 
Service. 
> 
> 
> 
>     Send BUG REPORTS to bugs@xxxx
>     Send SUGGESTIONS to suggest@xxxx
>     -----------------------------------------
>     Post AmiQuote-related messages ONLY to: 
amiquote@xxxxxxxxxxxxxxx 
>     (Web page: http://groups.yahoo.com/group/amiquote/messages/)
>     --------------------------------------------
>     Check group FAQ at: 
http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
> 
>     Your use of Yahoo! Groups is subject to the Yahoo! Terms of 
Service. 
> 
> 
>         Yahoo! Groups Sponsor 
>        
>        
> 
>   Send BUG REPORTS to bugs@xxxx
>   Send SUGGESTIONS to suggest@xxxx
>   -----------------------------------------
>   Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx 
>   (Web page: http://groups.yahoo.com/group/amiquote/messages/)
>   --------------------------------------------
>   Check group FAQ at: 
http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
> 
>   Your use of Yahoo! Groups is subject to the Yahoo! Terms of 
Service.


------------------------ Yahoo! Groups Sponsor ---------------------~-->
Buy Ink Cartridges or Refill Kits for your HP, Epson, Canon or Lexmark
Printer at MyInks.com. Free s/h on orders $50 or more to the US & Canada.
http://www.c1tracking.com/l.asp?cid=5511
http://us.click.yahoo.com/mOAaAA/3exGAA/qnsNAA/GHeqlB/TM
---------------------------------------------------------------------~->

Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx 
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/