Greetings all --
There has been a lot of activity on this
thread. I'll not respond to each point individually, but will make a
couple of general comments.
I know David Aronson, speak with him
regularly, and collaborate with him on projects. I have a copy of his
book, "Evidence-Based Technical Analysis." His book is excellent and I
highly recommend it. I think David and I are in pretty close agreement
on most of the modeling, simulation, testing, and validation issues.
I
have spoken with Robert Pardo and have exchanged several emails and forum
postings with him. I think his earlier book was very good, particularly
at the time it was published. And his more recent book is not quite up
to those standards. There are several important areas he did not cover
and several areas where I see things considerably differently than
Robert.
I have spoken with and exchanged emails with Van Tharp, and I
have copies of his books "Trade Your Way to Financial Freedom" and "Definitive
Guide to Position Sizing." Both are excellent, and I recommend them both
highly. Be sure to get the second edition of Trade Your Way to Financial
Freedom -- it has some important corrections and clarifications.
Permit
me a short rant on my soapbox. I really dislike it when people claim
ownership of common terms. Tom DeMark, Robert Pardo, Van Tharp, and
others put Service Mark symbols on terms that they think are unique to them,
but are not. I appreciate Tharp's enthusiasm over what he calls System
Quality Number, but I wish he would not put the Service Mark symbol next to
every occurrence of it. And trying to Service Mark the term Position
Sizing is like a dietician service marking "calorie counting." Robert
Pardo claims "Walk Forward." I used exactly that term describing exactly
that process in research papers I delivered at conferences in the late
1960s. The mark has been registered, not by Robert, but by a company I
used to work for and with which Robert was not associated, over my strong
objection. End of rant.
System quality number is equivalent to
t-test. Systems with SQNs above 2 work well for exactly the same reasons
that systems with t-test scores above 2 work well. In fact, it is
possible to create a custom objective function that Is the t-test and use it
for optimization. Attendees at my workshops in Melbourne later this
month will see that demonstrated. Optimizing for the t-test of
expectancy is equivalent to optimizing for CAR, so don't bother creating the
custom function unless you have a better candidate for your objective function
than CAR.
Back to the topic at hand -----
There is No rule of
thumb to determine how long the in-sample period should be. The Only way
to determine that is by testing the model and the data together. And be
prepared for that length to change over time. Some writers suggest a
relationship between the number of free parameters and the number of data
points, or some proportional division of the available data. Those
techniques do work on industrial time-series data which is usually stationary,
but they do not work on financial time-series data which is non-stationary and
changes as trading systems become better at extracting inefficiencies from
it.
No matter how good the in-sample results look, no matter how high
the t-test score is, no matter how many closed trades are represented --
in-sample results have no value in estimating the future performance of the
system. None. The only information you have that gives any
indication of future performance are the out-of-sample results from testing on
data that was never used at all -- not even once -- during system
development.
Tomorrow is out-of-sample. The only way to prepare
for real-money trading tomorrow is to be rigorous during the system testing
and validation process. Anything less will overestimate the probability
of success.
Thanks for listening,
Howard