[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: How to spot over optimizing



PureBytes Links

Trading Reference Links

Hi Herman,
Some of your definitions are quite arbitrary for me.
I will give an example:
I was studying last December an RSIt system and found a stock with 
almost ideal performance:
5trades/5winners/0losers since 1/2000.
I decided to follow this example.
Now, 7 months later, we have already 7trades/7winners/0losers and we 
walk steadily to 8/8/0.
The last short was at 2.30 and now the recent values are 1.75 to 1.80.
The 8th trade will be closed below 1.70 or [hopefully] lower, because 
of the bearish sentiment, because of the investing fear, because many 
share holders give up, because, because,..., we always hear 
some "reasons" to take the wrong decision...
Is there any reason to change this trading system because it is 
against 
#1, [yes, it is one stock]
#4, [yes, a +3500% is too good to be true in 30 ultra bearish months]
#6, [I did not even look at the before 1/2000 extra bullish [and 
catastrophic] period]
#9 [yes, 5/5/0 changed to 7/7/0 and it will be 8/8/0 in the next two 
weeks]
and maybe some other statements you refer ?
What would you do in my position ?
I would appreciate a reply, if we discuss for real trading and not 
academically.
Thank you for your interesting, as usual, mail.
Dimitris Tsokakis
--- In amibroker@xxxx, "Herman van den Bergen" <psytek@xxxx> wrote:
> Optimizing has been discussed at several times on this list, but 
never have
> I seen a check-list that could help you identify over optimizing. I 
welcome
> additions to the small list below. If we can identify degrees of
> over-optimization we can use the same tool to rate robustness, this 
because
> they are somewhat complementary.
> 
> If any of the following statements are True you may be over-
optimizing:
> 
> 1) The system works on only one stock
> 2) The gains are erratic with a final big gain
> 3) Optimization values are scattered (random peaks on a 3D chart)
> 4) Gains are too good to be true
> 5) The equity line is flat
> 6) The system only works in recent years
> 7) It doesn't work out-of-sample (I use 8y dev, 2y out of sample 
testing)
> 8) Optimizing parameters vary widely for different time periods
> 9) Greater than 75% winning trades
> 7) ... what can you add?
> 
> It seem that if you overlay the equities for many systems many 
appear to
> fail for the same periods (use 10 year data), I take this as a sign 
that the
> system pick up on a true market anomaly. It seems to indicates that 
there
> are times that the market is incoherent (chaos rules) and that most 
systems
> will not perform well. Combining equities can produce a "Market 
Coherence"
> (please suggest a better name) indicator which would tell you when 
trading
> is highly risky. Or taking the inverse; a "Market Chaos" indicator. 
Brave
> traders will flip all their signals during those periods :-)
> 
> Since there are many parameters one can optimize I take note of the 
gain
> improvement that comes with selectively activating various stages 
of the
> system. I start with the most basic form and add/measure features 
as I add
> them. An unreasonable large jump in gain for any one change makes 
it suspect
> and I will pick it apart manually for a few trades.