|
PureBytes Links
Trading Reference Links
|
Herman,
I haven't yet had an opportunity to expolore the 3d charting concept but
it sure sounds like a great way to evaluate for over optimization.
Personally, I haven't thought designing a system would be easy - I'm too
darned old for that! And I think the idea of having a written plan is
excellent - one that I plan to do.
Since we've been discussing the development of a system for QQQ and it
has a limited history (although NDX has been around longer and might be
and acceptable proxy) it complicates things - yes/no? With the wild
gyrations of the markets over the last 5 years don't you think you ought
to include the years 1997 thru 2002 to make sure the system has exposure
to those events? Be assured that I agree with your premises about
sample testing.. I have a feeling that Steve's chart that started this
latest round of discussions was optimized using dates beginning with
1/1/2000 to current which is highly optimized for a bear market. For
intstance start the backtest on 1/1/1998 and see what happens. That
worries me. How would you address this?
Thanks!
dingo
-----Original Message-----
From: Herman van den Bergen [mailto:psytek@x...]
Sent: Sunday, July 21, 2002 12:17 PM
To: amibroker@xxxxxxxxxxxxxxx
Subject: RE: [amibroker] StoRSI
Hi dingo,
There are two simple ways to verify that you did not over-optimize your
system:
1) I always chart each parameter I optimize to see if the parameter has
a
pattern, ie changes in some predictable fashion wrt another parameter
(don't
only plot parameters vs equity, analyze relationships between other
parameters as well). If I can i will plot two optimized parameters at
the
same time on a 3D plot. If I optimize 3 parameters I make one 3D chart
for
each value of the parameter with the smallest range - yes you get dozens
of
charts but you may start to see very interesting patterns! If you get
lots
of random peaks and you cannot see any kind of normal distribution you
have
probably over-optimized. Too many people think designing a trading
system is
simple...you should document it like a business plan - after all you
will
risk similar sums of money with it.
2) I always reserve a fair bit of data for out of sample testing. I
develop
may systems on data before 1/1/01 and test it out of sample on data
after
that date. I never adjust parameters to make it perform better in the
OOS
data. If it doesn't work i start from scratch, something i have done
hundreds of not thousands, of times. Ideally your equity curve shouldn't
show a ripple when you change data.
Not everybody believes in these two approaches because they think or
assume
it is impossible to get "perfect" out-of-sample performance for systems
developed on data from 2-10 years ago. But it is possible, for the QQQ
and
for many other stocks.
Good luck,
Herman.
-----Original Message-----
From: dingo [mailto:dingo@x...]
Sent: Sunday, July 21, 2002 11:39 AM
To: amibroker@xxxxxxxxxxxxxxx
Subject: RE: [amibroker] StoRSI
Hi WJ,
>>From all of my reading on this list it seems that every time on
ventures
off and starts optimizing every parameter in the foundation of a formula
that the dreadful "over optimization" term starts rearing its ugly head.
\
I'm far from an expert and am still a little confilicted on it. I can
understand the desire to have a "robust" solution for a basket of stocks
but I can't help but think that if you want to develop a tailored
solution for a very high profile vehicle like qqq it should be more than
ok.
dingo
|