|
PureBytes Links
Trading Reference Links
|
-----Original Message-----
From: Mark Johnson [mailto:janitor@xxxxxxxxxxxx]
Sent: Friday, May 07, 1999 9:45 AM
To: omega-list@xxxxxxxxxx
Subject: Scared of "Catscan"
[Snip]
On the other hand, good old Chuck LeBeau advocates developing
a different system for each different market. In that case,
Catscan is 39 different systems with an average of 45/39 = 1.15
parameters per system. I bet Chuck LeBeau loves it.
But I don't. I am nervous that it is excessively over-fitted.
I'll stick to systems that use the same code and the same
parameter values on all markets.
========== A broader view point.
As a general rule having the same parameters for all markets is solid
advice but it shouldn't prevent system users from experimenting.
Results from my trading and testing have shown that curve fitting and
failed performance is more often associated with entries that use
indicators. Where these indicators have specific values that only
work on that market. If the values for the indicators don't work on
other markets especially of the same type, for example S&P, NYFE or
VL, then it is likely that the entry has been curve fitted and is
unlikely to survive the test of time.
However, if the parameter values do work on a basket of markets and
the performance is acceptable, than I have found historical
performance using market specific parameters will continue into the
future.
The process of selecting a parameter is more closely associated with
getting the value that sits in the middle of a distribution where the
movement of distribution below the number doesn't put the system into
failed performance. This is why using the same number that works on
all markets is safer as Mark points out.
Finding the correct number, especially for long term systems, is
restricted by the systems ability to create a large enough sample of
trades that the results can be considered statistically valid.
Getting a large enough sample is only part of the problem. The other
part is looking at how the system's operation of each trade in the
data series behaves. It is possible to create some remarkable
systems that can't physically be executed or considered reasonable.
For example, testing a system that enters at some price and exiting
at another on the same daily bar. A single bar doesn't provide
enough resolution for this kind of test and must be considered
invalid until it is tested on intraday data. More often than not
these tweaked variables are not reliable. Being able to test on a
broad basket of markets with the same variables can improve the
validity of the performance results.
Trades that use separate and disconnected setup and entry rules can
reduce the potential for curve fitting but they won't prevent it.
Reversing a position with an opposing entry has the same problems as
straight entry.
Straight exit methods that use some underlying calculation don't seem
to suffer much from curve fitting as long as they aren't using some
form of a price indicator. For example, Chuck LeBeau's multiplier
value of true range can change from one market to another and still
provide future performance.
Different dollar values for initial protective stops also don't have
the curve fit problem as long as the samples are large enough and the
values work on other markets without creating disastrous results.
Using the same protective stop value on all markets may cause more
damage than finding the center of the distribution for all trades.
There is validity in understanding how far a trade moves against the
intended direction. John Sweeney calls this maximum adverse
excursion.
Using more than one type of exit can also improve the performance of
a system. Having more methods for getting out will increase the
number of parameters in a system and the large number of parameters
isn't necessarily a reflection on the quality of the system.
Where most systems I've bought and programmed seem to fail is that
they put all the magic into where they get in and do only cursory
work on getting out.
A while back Mark posted the results of a random entry system that
used what Chuck LeBeau called his Chandelier exit. This is an
important method for exiting a trade. This method works well because
its trip is calculated from the direction of movement. Adding this
method to an exit method that calculates its trip from a ratio of
equity earned can improve efficiency of your exit process because
each method isn't expected to handle all market conditions. Both of
these methods, which often add parameters, can have different values
from market to market and still perform well over time.
My point in all this is to add some experience to the argument on
multiple and different parameter values being a predictor of curve
fitting. Very often it is but I haven't found it to be an absolute
rule -- Just be careful.
I hope this helps.
Roger...
|