Profit Factor & System Analysis

["Paul Barnes" <paul@xxxxxxxxxxxx>]

I find Profit Factor useful as a quick filter to rank the results of
optimising a system with different parameters.  After an optimisation, I
will first sort the results by the number of trades and ignore all the
results with less than 30 trades.  After that I sort on the profit factor
and start to look harder at the parameter set with the best profit factor.

You can't just look at one number, (e.g. profit factor), because it is the
process of analysing a system back-to-front, upside-down and inside-out that
builds the confidence you need to persist with trading the system when it is
having a drawdown.

I want to see a smoothly rising equity curve.  I use an IncludeSystem with
all my testing to bring in systems that write equity data to files.  I then
have macros in Excel so that I can quickly open those files and graph the
equity curves for both closed trades, and daily equity.  If the system
doesn't perform consistently over time, then I reject it.

I look at the ratio of average profit per year to maximum drawdown to
compare different systems applied to the same commodity.  My best systems
are around 0.8.  Using the average profit per year allows comparison between
commodities with different amounts of historical data.

Having said, this, I agree with what others have written that the maximum
drawdown is not a dependable parameter of the system results.  When I'm
getting serious about a system, I use a Monte Carlo simulation (randomly
sampling from the trades thousands of times ) to determine maximum drawdown
at the 6.7 percentile.  I use the 6.7 percentile because this is equivalent
to the drawdown I could expect to encounter once in every fifteen years.
This feels about the maximum drawdown that I could tolerate without
despairing that the system was no longer working.  (Once I commit to trading
a system, I define performance at the 6.7 percentile for drawdown and number
of trades in a row that are losers.  This predetermines the point at which I
would judge that the trading performance is so bad that I have to give up on
the system.)

I think there is probably value in considering the system's time in the
market.  I'd be interested in hearing how others make use of this for
comparing systems.

Paul Barnes