|
PureBytes Links
Trading Reference Links
|
Howard:
Interesting points, as always.
Have you considered the corrolation between the insample fitness function and the out of sample profits in the next walk forward step?
That offers a way to objectivly chose the fitness function and in sample size to maximize the predictive power of the fitness function.
After all, we want the fitness function to choose the best parm set for the next out of sample step. A perfect fitness function would have a corrolation of 1 and one without any predictive power at all would be near zero.
In tests I have run, over a simple system, I have found average corrolations beween in sample fitness and out of sample profit of < .2. That is a very weak relationship, with more noise than signal.
I would be interested in what other developers find with other systems.
In general, longer in sample lengths have a better corrolation then lower. For example, walk forwards with 1 month in sample and 1 month out of sample had a near 0 corrolation between profit and the prior month fitness, while longer periods had higher corrolations up to the limit of my data.
Car was the worse performing of all of the fitness functions I tested. UPI was the best, but there was not a dramatic difference between it and the other commonly used risk adjusted methods such as CAR/MDD, sharpe, etc.
--- In amibroker@xxxxxxxxxxxxxxx, Howard B <howardbandy@xxx> wrote:
>
> Greetings all --
>
> There has been a lot of activity on this thread. I'll not respond to each
> point individually, but will make a couple of general comments.
>
> I know David Aronson, speak with him regularly, and collaborate with him on
> projects. I have a copy of his book, "Evidence-Based Technical Analysis."
> His book is excellent and I highly recommend it. I think David and I are in
> pretty close agreement on most of the modeling, simulation, testing, and
> validation issues.
>
> I have spoken with Robert Pardo and have exchanged several emails and forum
> postings with him. I think his earlier book was very good, particularly at
> the time it was published. And his more recent book is not quite up to
> those standards. There are several important areas he did not cover and
> several areas where I see things considerably differently than Robert.
>
> I have spoken with and exchanged emails with Van Tharp, and I have copies of
> his books "Trade Your Way to Financial Freedom" and "Definitive Guide to
> Position Sizing." Both are excellent, and I recommend them both highly. Be
> sure to get the second edition of Trade Your Way to Financial Freedom -- it
> has some important corrections and clarifications.
>
> Permit me a short rant on my soapbox. I really dislike it when people claim
> ownership of common terms. Tom DeMark, Robert Pardo, Van Tharp, and others
> put Service Mark symbols on terms that they think are unique to them, but
> are not. I appreciate Tharp's enthusiasm over what he calls System Quality
> Number, but I wish he would not put the Service Mark symbol next to every
> occurrence of it. And trying to Service Mark the term Position Sizing is
> like a dietician service marking "calorie counting." Robert Pardo claims
> "Walk Forward." I used exactly that term describing exactly that process in
> research papers I delivered at conferences in the late 1960s. The mark has
> been registered, not by Robert, but by a company I used to work for and with
> which Robert was not associated, over my strong objection. End of rant.
>
> System quality number is equivalent to t-test. Systems with SQNs above 2
> work well for exactly the same reasons that systems with t-test scores above
> 2 work well. In fact, it is possible to create a custom objective function
> that Is the t-test and use it for optimization. Attendees at my workshops
> in Melbourne later this month will see that demonstrated. Optimizing for
> the t-test of expectancy is equivalent to optimizing for CAR, so don't
> bother creating the custom function unless you have a better candidate for
> your objective function than CAR.
>
> Back to the topic at hand -----
>
> There is No rule of thumb to determine how long the in-sample period should
> be. The Only way to determine that is by testing the model and the data
> together. And be prepared for that length to change over time. Some
> writers suggest a relationship between the number of free parameters and the
> number of data points, or some proportional division of the available data.
> Those techniques do work on industrial time-series data which is usually
> stationary, but they do not work on financial time-series data which is
> non-stationary and changes as trading systems become better at extracting
> inefficiencies from it.
>
> No matter how good the in-sample results look, no matter how high the t-test
> score is, no matter how many closed trades are represented -- in-sample
> results have no value in estimating the future performance of the system.
> None. The only information you have that gives any indication of future
> performance are the out-of-sample results from testing on data that was
> never used at all -- not even once -- during system development.
>
> Tomorrow is out-of-sample. The only way to prepare for real-money trading
> tomorrow is to be rigorous during the system testing and validation
> process. Anything less will overestimate the probability of success.
>
> Thanks for listening,
> Howard
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|