Re: [amibroker] Re: AmiBroker Measures of Performance

["Howard B" <howardbandy@xxxxxxxxx>]

Hi James --

You wrote:
"I really like the idea he presents about the objective function, but 
after basically saying that this is a personal decision, no 
information about why you may want to pick RAR vs CAR, or Sharpe vs 
K-raio, etc, nor anything regarding what kind of values typically 
are produced by a "good" system (obviously the definetion of good 
will vary based on objective function...kind of a chicken or the egg 
question!)"

Thanks for the kind words about my book.  

The choice of an objective function really is personal.  In my mind, it is not a chicken and egg concept -- it is clearly a personal preference.   The choice of the objective function plays a very large part in determining which trading systems pass your personal tests of acceptability. 

I do recommend looking at metrics that combine steepness of the equity curve, and either smoothness of equity curve or minimization of drawdown -- RRR, KRatio, Ulcer Performance Index, RAR/MDD, CAR/MDD, and so forth.  In each case, higher values are preferable.   In my experience and in discussions with lots of other traders, most people want high returns, but most are limited by the amount of drawdown they can stand.  Most traders stop trading a system because the drawdown has been excessive.   

There are people and organizations that seek high return in preference to low risk, and vice versa.  Which you are will determine what objective function you use.

As to what value of each is acceptable -- I recommend the procedure I describe in the book.  Run several tests.  For each test, plot and print the equity curve.  Record the values of the metrics that you think you might want to maximize or minimize for each of those tests and equity curves.  Arrange the results, based on the shape of the equity curve and the values of the metrics, into order as you prefer them -- best to worst.  If that places them in order by one of the metrics, that metric is a good choice to either be your objective function, or to be a primary component of your objective function.  Repeat the process of defining and refining the objective function and arranging the results until you have decided what metrics you want to use, and what values are desirable and what minimums are acceptable to you.

Note that some objective functions tend to select alternatives (when optimizing in-sample data) that perform better out-of-sample than other objective functions.  Those objective functions that I mentioned are some that do.  Other objective functions tend to select alternatives that fit to the noise rather than to the signal, and perform poorly out-of-sample.  Examples include total profit, CAR, percent winners, and other than do not include a component representing risk.  

Sharpe ratio is popular with some people because it measures performance relative to a risk-free standard and penalizes volatility.  Some money managers report their Sharpe ratio, so it is important to them.  Sharpe ratio suffers because the measure of volatility it uses is the standard deviation of trades (or of daily equity).  By doing that, it penalizes both negative volatility and positive volatility.  Most people agree that negative volatility is bad and should be minimized.  Depending on your trading philosophy, you may think positive volatility is bad or you may think it is good.  Your view of the value of Sharpe ratio will depend on your answer.  If you think positive volatility is good, then replace the standard deviation with the equivalent calculation using only the losing entries.  The metric that results from doing this goes by the name of semi-variance or Sortino ratio.  Having said all that ---- in my experience, Sharpe ratio is one of the metrics that tends to select alternatives that do not perform well out-of-sample.   But, depending on your trading systems, your experience may be different.

Continuing on with the importance of the objective function.  You will eventually want to use the walk forward method to automate the process of selecting the best alternative over an in-sample period, then test that specific alternative over the following out-of-sample period.  Each step in this process, and each transition from in-sample to out-of-sample, gives you one more look at what might happen when you begin trading the system.  The Best alternative is that one that is at the top of the list when the list is sorted into order based on the objective function.  Since the process is automatic, you will not have an opportunity to examine the performance of any of the alternatives that have different parameters -- the top one is the one that will be used.  So you must have confidence that the alternative at the top of the list is the one you prefer.

Thanks for listening,
Howard

On Sat, Apr 12, 2008 at 7:44 AM, brian_z111 <brian_z111@xxxxxxxxx> wrote:

James,

> I found it to be lacking in terms of
> information regarding metrics.

IMO Howard didn't set out to write a book on metrics.
I also believe he was constrained by size limits.

Over to the discussion:

Most of the material around only defines the maths - I don't know of
anyone I could recommend for in depth analysis of metrics - in fact
they are all pretty lame and it is a glaring omission from the
trading literature.

The only thing I could do was to figure out the pros and cons for
myself.

IMO most of them are a load of old rubbish - I am interested in root
causes and follow:

- trading as a binomial event i.e. Win/Loss +-error and Win%/Loss% +-
error, which predicts the estimated GeometricMean +- error.

EGM == arithmetic mean^2 - StDev^2 (from RalphVince - "Mathematics of
Money Management").

PowerFactor == (EGM +- StDev)^N is the driver of an equity curve
probability cone where the EGM is the reward, and the worstcase eq
curve is the risk (you choose your own risk tolerance) - it can be
annualised as %.

>From there it is over to RV and MM - everything else is second rate.

brian_z

--- In amibroker@xxxxxxxxxxxxxxx, "jamesfarrow2003"

<jamesfarrow2003@xxx> wrote:
>
> Well, I guess I am still looking. I just finshed Howard's book(in
> alittle over two days), and while I found it very readable, and for
> the most part captivating, I found it to be lacking in terms of
> information regarding metrics.
>
> I really like the idea he presents about the objective function,
but
> after basically saying that this is a personal decision, no
> information about why you may want to pick RAR vs CAR, or Sharpe vs
> K-raio, etc, nor anything regarding what kind of values typically
> are produced by a "good" system (obviously the definetion of good
> will vary based on objective function...kind of a chicken or the
egg
> question!)
>
> I am glad that I read the appendix, which is where there was a
> sample of coding an objective function - very useful. Also all of
> the info on walk forward testing and seperation of data is great (
> and like all good ideas, obvious once you learn about it:)
>
> I would definetly recommend(and already have to sevral co-workers)
> his book to someone looking for methods of system validation,
> particularly AmiBroker users. I am now looking for a book to help
> me define my personal objective functions!
>
> Another topic I need to research is at what point do you begin
> trading a system... after all back testing/optimization/walk
> forward/monte carlo is complete, or does it make more sense
> to "paper trade" for a while (maybe a set number of trades, or
> time)..again probably mostly a personal choice;)
>
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, Grant Noble <gruntus@> wrote:
> >
> > Howard's book is essential for grasping the essence of why any
> metric is useful or not. As for
> > detailed definitions of each metric just start at the UKB and go
> from there - Brian is assembling
> > loads of link to useful sites.
> >
> > jamesfarrow2003 wrote:
> > > Can anyone in this group recommend a good book which discusses
> the
> > > various metrics that AmiBroker computes based on backtest
> results?
> > >
> > > Thanks,
> > >
> > > james
> > >
> > >
> > > ------------------------------------
> > >
> > > Please note that this group is for discussion between users
only.
> > >
> > > To get support from AmiBroker please send an e-mail directly to
> > > SUPPORT {at} amibroker.com
> > >
> > > For NEW RELEASE ANNOUNCEMENTS and other news always check
DEVLOG:
> > > http://www.amibroker.com/devlog/
> > >
> > > For other support material please check also:
> > > http://www.amibroker.com/support.html
> > > Yahoo! Groups Links
> > >
> > >
> > >
> > >
> >
>

__._,_.___








Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to 
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html

Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___