--- In
amibroker@xxxxxxxxxxxxxxx, Howard B <howardbandy@xxx> wrote:
>
> Greetings all --
>
> There has been a lot of activity on this thread. I'll not respond to each
> point individually, but will make a couple of general comments.
>
> I know David Aronson, speak with him regularly, and collaborate with him on
> projects. I have a copy of his book, "Evidence-Based Technical Analysis."
> His book is excellent and I highly recommend it. I think David and I are in
> pretty close agreement on most of the modeling, simulation, testing, and
> validation issues.
>
> I have spoken with Robert Pardo and have exchanged several emails and forum
> postings with him. I think his earlier book was very good, particularly at
> the time it was published. And his more recent book is not quite up to
> those standards. There are several important areas he did not cover and
> several areas where I see things considerably differently than Robert.
>
> I have spoken with and exchanged emails with Van Tharp, and I have copies of
> his books "Trade Your Way to Financial Freedom" and "Definitive Guide to
> Position Sizing." Both are excellent, and I recommend them both highly. Be
> sure to get the second edition of Trade Your Way to Financial Freedom -- it
> has some important corrections and clarifications.
>
> Permit me a short rant on my soapbox. I really dislike it when people claim
> ownership of common terms. Tom DeMark, Robert Pardo, Van Tharp, and others
> put Service Mark symbols on terms that they think are unique to them, but
> are not. I appreciate Tharp's enthusiasm over what he calls System Quality
> Number, but I wish he would not put the Service Mark symbol next to every
> occurrence of it. And trying to Service Mark the term Position Sizing is
> like a dietician service marking "calorie counting." Robert Pardo claims
> "Walk Forward." I used exactly that term describing exactly that process in
> research papers I delivered at conferences in the late 1960s. The mark has
> been registered, not by Robert, but by a company I used to work for and with
> which Robert was not associated, over my strong objection. End of rant.
>
> System quality number is equivalent to t-test. Systems with SQNs above 2
> work well for exactly the same reasons that systems with t-test scores above
> 2 work well. In fact, it is possible to create a custom objective function
> that Is the t-test and use it for optimization. Attendees at my workshops
> in Melbourne later this month will see that demonstrated. Optimizing for
> the t-test of expectancy is equivalent to optimizing for CAR, so don't
> bother creating the custom function unless you have a better candidate for
> your objective function than CAR.
>
> Back to the topic at hand -----
>
> There is No rule of thumb to determine how long the in-sample period should
> be. The Only way to determine that is by testing the model and the data
> together. And be prepared for that length to change over time. Some
> writers suggest a relationship between the number of free parameters and the
> number of data points, or some proportional division of the available data.
> Those techniques do work on industrial time-series data which is usually
> stationary, but they do not work on financial time-series data which is
> non-stationary and changes as trading systems become better at extracting
> inefficiencies from it.
>
> No matter how good the in-sample results look, no matter how high the t-test
> score is, no matter how many closed trades are represented -- in-sample
> results have no value in estimating the future performance of the system.
> None. The only information you have that gives any indication of future
> performance are the out-of-sample results from testing on data that was
> never used at all -- not even once -- during system development.
>
> Tomorrow is out-of-sample. The only way to prepare for real-money trading
> tomorrow is to be rigorous during the system testing and validation
> process. Anything less will overestimate the probability of success.
>
> Thanks for listening,
> Howard
>