|
PureBytes Links
Trading Reference Links
|
Hmmm. I see your point of encouraging system characteristics that we like such as small draw downs, trade frequency and holding period.
But, I keep thinking that there should be a relationship between in sample performance and out of sample.
For example:
If we use the typical default net gain objective function, we will tend to get a system that trades very frequently, but with very small gains per trade. If the out of sample average return degrades even a little, which it is likely too given the curve fitting bias, then the expectation per trade can shift to negative and loose money out of sample. Such is the beginning and end of many a eager young systems trader.
If we instead optimize the profit factor, then we will tend to get a system that trades very rarely both in sample and out and is more vulnerable to curve fitting due to small sample size.
It feels like there is a deeper relationship between out of sample performance and objective function, but it eludes me.
Maybe a Walk Forward Objective Efficiency? The correlation between in sample scores and out of sample profit? Hmm. I'll have to try that.
In sample size is another head scratcher. Maybe trial and error is the best that we can do. Not very elegant. But your point is good that it is model dependent.
-Doug
--- In amibroker@xxxxxxxxxxxxxxx, Howard B <howardbandy@xxx> wrote:
>
> Hi Doug --
>
> The objective function is not selected on a theoretical basis. It is
> selected by you by looking at a variety of trading results and selecting the
> objective function that best describes the alternatives that best fit your
> own style and personality. For a start, print out some equity curve
> charts. Lay them on the floor and look at them. Rearrange them and rank
> them in order from best to worst according to your own personal
> preferences. Select or design an objective function that scores highest on
> those charts you like and lowest on those you reject.
>
> I think there are several criteria that must be met:
>
> Having a trading system with a positive expectancy.
> Having an objective function that fits your trading style and personality.
> Having a trading system that passes walk forward validation.
>
> My experience is that objective functions that reward fairly short holding
> periods and smooth equity curves both fit my style and personality and
> perform reasonably well out-of-sample.
>
> Your experience with the tests you ran agree pretty well with mine. In
> particular, using net profit as the objective function usually (but not
> always) selects systems that do not perform well out-of-sample. But notice
> that net profit is often the default objective function for trading system
> development platforms. In fact, some platforms default to net profit and
> give no opportunity to use anything else. But those also usually do not
> provide the tools for system validation, such as automatic walk forward
> testing, that AmiBroker does.
>
> On the subject of adding data to the in-sample period. More is not always
> better. Determination of the length of the in-sample period is critical to
> getting good results. If there is too little data, the model will not
> detect enough patterns or enough of the pattern, and it will fail to
> recognize the pattern both in-sample and out-of-sample. If there is too
> much data, the extraneous / old / stale data will distract the model and the
> signal to noise ratio will drop. Experiment to find the right length. That
> length is very dependent on the model and the data it is processing.
>
>
> Thanks,
> Howard
>
>
> On Fri, May 8, 2009 at 1:36 PM, dloyer123 <dloyer123@xxx> wrote:
>
> >
> >
> > This raises a interesting point.
> >
> > If some objective functions are poor predictors of out of sample
> > performance, are there other objective functions that are good predictors of
> > out of sample performance? Or at least better?
> >
> > It seems like this is the kind of problem where there should be a optimal
> > answer. I wish that I had the math background to find it.
> >
> > The best that I could come up with is that there needs to be a measure of
> > performance and a measure of risk to balance it. There are several such
> > measures in common use and they will each tend to favor different solutions.
> >
> >
> > Since I am not able to pick one on a theoretical basis, I had to resort to
> > empirical testing using Ami's walk forward feature. As a result, I found
> > that net profit performed the worst of all of the objective functions I
> > tried in terms of out of sample profit. In my simple test, UPI performed the
> > best. CAR/MDD also performed well. I don't recall how the others did. I
> > believe that Sharpe was somewhere in the middle and k-ratio didnt do well at
> > all.
> >
> > However, this is just one simple system and I really wish that I had a
> > theoretical basis to hang my hat on it. Results for other systems may be
> > very different and it might be different if I repeated the test now, with a
> > extra year of test data.
> >
> > -Doug
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%40yahoogroups.com>, Howard B
> > <howardbandy@> wrote:
> >
> > > It is also interesting to note that some objective functions tend to
> > reward
> > > / select values for the logic and parameters that result in trading
> > systems
> > > that do not trade well out-of-sample. In particular, be careful using net
> > > profit, sharpe ratio, and so forth. By all means, define your own custom
> > > objective function metric, program it, and tell AmiBroker to use it.
> > >
> > > Thanks for listening,
> > > Howard
> > >
> > >
> >
> >
> >
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|