Hi PS --
One way to get a feeling for values for metrics and objective functions
is run an optimization, giving you a range of results. If necessary,
peek into the future so you are certain to get some really good results.
Look through the list of results, pick some individual results with a
variety of values for the metrics you want to learn about.
Set the default value of the optimized variables to the specific values
you chose.
Run a single backtest. Plot the equity curve.
When you have done a few of these and have a feeling for the
characteristics of the systems you would like to trade, print out the
plot of the equity curves and write the values of the metrics and
objective functions on the printout.
-----------------------
Your question about interpretation.
My preference is for objective functions that reward equity growth and
penalize drawdown. CAR/MDD, RAR/MDD, RRR, Recovery Factor, K-Ratio,
Ulcer Performance Index, and Sharpe Ratio all do that. In all of these
cases, larger values are better.
The important results are the out-of-sample results. We are looking
for logic and parameter values that not only perform well in-sample,
but that also perform well out-of-sample.
When you run walk forward tests, you will find that some objective
functions give high rank to alternatives that do tend to perform well
out-of-sample, while other objective functions select alternatives that
often do not perform well out-of-sample. You will need to run some of
your own tests to get a feeling for how these work on your trading
systems.
My experience is that using net profit is usually a poor objective
function, although it is the default (and often the only) selection for
some trading systems development platforms other than AmiBroker.
Fund managers are evaluated on the Sharpe Ratio of their performance.
My experience is that systems selected using Sharpe Ratio tend to
perform poorly out-of-sample.
Standard Error is a measurement of the smoothness of the equity line --
smaller values are better. But optimizing to minimize standard error
alone may give high ranks to systems that have trading characteristics
that you do not want. For example, if you have the options set so that
the equity earns interest when ever it is not in a position, then using
standard error will reward alternatives that stay in cash and trade
infrequently.
------------------------
One of the very valuable features of AmiBroker is the capability for
the system developer to create whatever objective function he or she
wants to use. It does not have to be limited to those that are
distributed with AmiBroker and appear in the list of metrics. You
might want to combine metrics -- for example to reward alternatives
whose trading frequency suits your preferences, while also rewarding
equity growth and penalizing drawdown.
Thanks,
Howard
On Sat, Nov 28, 2009 at 11:33 PM,
potatosoupz
<potatosoupz@yahoo.com>
wrote:
I don't see any good definitions for the metrics below. I am
familiar of course with Sharpe, and Ulcer Index. I'm finding it a bit
hard to reconcile the differences. How would you reconcile a backtest
that has a max draw down % that is smaller than it's profit %, a low
ulcer index, a high Sharpe, but a very low k ratio (< .05)?
Ulcer Index in my mind is one of the best metrics outside of a much
deeper quantitative treatment of things. Thoughts?
CAR/MDD ?
RAR/MDD ?
Payoff Ratio ?
Standard Error ?
RRR ?
Recovery Factor ?