[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: How to spot over optimizing



PureBytes Links

Trading Reference Links

Herman,
As you may read at
http://money.cnn.com/2002/07/14/news/companies/bc.tech.worldcom.reut/i
ndex.htm
cooking is extended back to 2000 [and not only 2001, as it was 
previously believed.]
Cooking is more robust now...
By the way, if you have had last January a system filling all your 
requirements for WCOM,what would you do ?
In other words, mechanical systems are always dangerous, they do 
not "see" fundamentals.
And fundamentals may be fake enough, as we are informed several 
months [or years] later.
This lack of confidence we practice this period will have some 
influence over the whole market.
Volumes will be lower and the game will be "easier" for potential 
traders.
Unfortunately we do not have T/A tools for the lack of confidence, so 
the risk will be increased in any decision.
I think it is a kind of luxury to put these [correct] requirements 
and I can not see a real application.
[except to avoid trading]
Dimitris Tsokakis 
--- In amibroker@xxxx, "Herman van den Bergen" <psytek@xxxx> wrote:
> Optimizing has been discussed at several times on this list, but 
never have
> I seen a check-list that could help you identify over optimizing. I 
welcome
> additions to the small list below. If we can identify degrees of
> over-optimization we can use the same tool to rate robustness, this 
because
> they are somewhat complementary.
> 
> If any of the following statements are True you may be over-
optimizing:
> 
> 1) The system works on only one stock
> 2) The gains are erratic with a final big gain
> 3) Optimization values are scattered (random peaks on a 3D chart)
> 4) Gains are too good to be true
> 5) The equity line is flat
> 6) The system only works in recent years
> 7) It doesn't work out-of-sample (I use 8y dev, 2y out of sample 
testing)
> 8) Optimizing parameters vary widely for different time periods
> 9) Greater than 75% winning trades
> 7) ... what can you add?
> 
> It seem that if you overlay the equities for many systems many 
appear to
> fail for the same periods (use 10 year data), I take this as a sign 
that the
> system pick up on a true market anomaly. It seems to indicates that 
there
> are times that the market is incoherent (chaos rules) and that most 
systems
> will not perform well. Combining equities can produce a "Market 
Coherence"
> (please suggest a better name) indicator which would tell you when 
trading
> is highly risky. Or taking the inverse; a "Market Chaos" indicator. 
Brave
> traders will flip all their signals during those periods :-)
> 
> Since there are many parameters one can optimize I take note of the 
gain
> improvement that comes with selectively activating various stages 
of the
> system. I start with the most basic form and add/measure features 
as I add
> them. An unreasonable large jump in gain for any one change makes 
it suspect
> and I will pick it apart manually for a few trades.