|
PureBytes Links
Trading Reference Links
|
One of the goals in testing a trading system is to provide
some kind of answer to the question:
IS THIS A "GOOD" TRADING SYSTEM? DOES IT PERFORM WELL?
However, backtesting software merely calculates historical
statistics. It takes *human* *judgement* to interpret backtest
results and then come to an opinion whether they are good
or not-good. Observe that different people will interpret
the same test results in different ways: what is "good" for
me, might be "unacceptable" for you, and vice versa.
(For example, I might be psychologically fixated upon
net_profit and nothing else, whereas you might only care
about Sharpe_ratio, so I could love a system that had huge
profits and low Sharpe ratio, while you would hate it.)
One way to evaluate a trading system's backtest results is to
*compare* them to the backtest results from a benchmark reference.
Essentially this compares the investment results of the trading
system against the investment results of a well-known,
public-domain investment method. Examples of reference
investment methods include "BARRA Managed Futures Index",
"Risk-Free Interest Rate", "Buy and Hold the S&P 500",
"Buy and Hold Fidelity Magellan Mutual Fund", and so forth.
Comparing a trading system against a benchmark reference
investment method recognizes that an investor/speculator
has a choice: she can trade the trading system, or invest
in the reference investment method, whichever she prefers.
By comparing the two sets of backtest results, she can
decide which investment approach she likes best.
This makes evaluation easier: all you need to do is
(1) use your *human* *judgement* to decide which performance
measures are important TO YOU (Is it important TO YOU
that a system should have a high Profit Factor? A high
profit-per-trade? Low drawdown? Please decide.); then
(2) accumulate trading system backtest results including
backtest results of benchmark reference investment methods;
then (3) choose the one that scores the best on the
performance measures that matter most TO YOU. Voila.
Several writers in this forum have, in the recent past,
advocated "buy and hold the S&P" as a significant benchmark
reference investment. But a full set of backtest performance
data on this system (including its performance in the first
half of 1999) is hard to find. So I decided to perform some
historical backtests on this investment method. I chose the
PERL5 programming language for the project, and wrote all
code "from scratch".
TEST METHOD
TEST METHOD
I tested "Buy and Hold the S&P 500 Futures" .
PRICE DATA
I decided to use the actual, unadjusted, real commodity futures
contracts in this tests. Please note: this means
"continuous" contracts..... were NOT used
"perpetual" contracts...... were NOT used
"back-adjusted" contracts.. were NOT used
"spliced" contracts........ were NOT used
Instead, I used the actual contracts that were actually traded
at the actual Chicago Mercantile Exchange. These contracts
are born, they thrive a while, and then they expire. Only
the "day session" price data were used; Globex trading was
ignored. Not that it matters for buy-and-hold :-).
I wrote my software to perform actual roll-over trades, to exit an
existing position in a soon-to-expire contract, and to simultaneously
establish a new position in a contract with plenty of time till
expiration. As in real-life trading, these rollover trades are
performed using spread orders, and (as in real life), commissions
are charged and slippage is encountered on rollover trades.
I bought a single contract at the open on the very first day the
S&P 500 futures contracts were traded on the CME. (I bought one
June 1982 futures contract at market-on-open on 21 April 1982).
I held the position (using rollovers) until Friday afternoon,
July 23, 1999.
Note1: on November 1st, 1997, the CME divided the SP contract
value in half. Before then, a move of 1.00000 in the SP futures
was worth $500.00 ; afterwards, it was worth $250.00. I handled
this in testing by having my software trade one contract from
21Apr82 to 01Nov97, and two contracts from 02Nov97 onward.
Notice that this increases the "skid" due to commissions and
slippage after 01Nov97.
Note2: I use a calendar based rollover schedule, both in my real-
money trading, and in this backtest. I roll over SP500 futures
positions on the 8th calendar day of the delivery month
(i.e. on 3/8, 6/8, 9/8, and 12/8), or if the market is closed on
that date, the first market day thereafter. Thus I performed
my first rollover on 08 June 1982, selling my June contract and
buying a September contract.
TREATMENT OF CASH
Cash in the account was used to purchase 3-month T-Bills whose
yield was 5.00% annually. T-Bills were assumed to be redeemed
and repurchased on calendar days 1/15, 4/15, 7/15, and 10/15
(or the first market day thereafter). T bills were purchased
at a discounted price of (1/1.0125)*Face_value, and redeemed
3 months later for full Face_value. {note: 1.0125 quarterly
is 1.0500 annually}. Face value was allowed to be $10,000,
or $11,000, or any higher multiple of $1,000.
COMMISSIONS AND SLIPPAGE
T-bill redemptions were charged a commission of $40.00 regardless
of face value. (T-bill purchases were free). S&P futures
trades were charged a round-trip commission of $50.00 per
contract. Remember, trades were 1 contract before 01Nov97
and 2 contracts afterwards. Extremely modest (unrealistic?)
slippage was charged: 0.05 points (1 tick) per contract per
round trip.
STARTING ACCOUNT EQUITY
I ran the backtest with three different initial account
values. First I used an account value that was precisely
equal to buying the underlying index without leverage.
Since futures were priced at 116.35 , and the multiplier
was $500 per point, the underlying was worth $58,175.00
on the first day of trading. So I made one backtest run
with an initial account equity of $58,175.
I also did a backtest, starting the account with half
that amount ($29,088), corresponding to a "leverage" of
two-to-one. And I performed a third backtest, starting
the account equity at $120,000.
Why run the test three times with three account equity
values? To pacify observers who worry that initial
equity might unfairly distort the results. By using
three different account values, spanning a 4-to-1 range,
we can interpolate to any starting account equity that
a reader may desire.
TEST RESULTS
TEST RESULTS
SUMMARY OF PERFORMANCE STATISTICS
Starting equity....................... 29088.00 58175.00 120000.00
Final equity.......................... 671871.09 739612.41 883610.25
Total net profit...................... 642783.09 681437.41 763610.25
Profit from T-bills................... 157033.09 195687.41 277860.25
Profit from actual futures trades..... 485750.00 485750.00 485750.00
Compound Annual Growth Rate (percent). 19.95 15.87 12.26
Max Drawdown (percent)................ 51.68 40.06 26.92
(CAGR / MAXDD) Ratio.................. 0.386 0.396 0.456
Annual Standard Deviation (percent)... 25.98 18.36 12.13
Sharpe Ratio.......................... 0.575 0.592 0.598
Semideviation Ratio................... 0.796 0.802 0.809
Return Retracement Ratio.............. 2.196 2.527 3.359
Sterling Ratio........................ 1.364 1.285 1.166
Keltner K-Ratio....................... 0.000023 0.000024 0.000027
First day of test..................... 820421 820421 820421
Last day of test...................... 990723 990723 990723
# days................................ 4433 4433 4433
# rollover trades..................... 70 70 70
T-Bill interest rate.................. 0.050 0.050 0.050
Slippage per round trip trade......... 0.05 0.05 0.05
Commission on T-Bill purchases........ 40.00 40.00 40.00
Round trip commission on SP futures... 50.00 50.00 50.00
Total commissions..................... -6520.00 -6520.00 -6520.00
Total slippage........................ -1900.00 -1900.00 -1900.00
DETAILED RESULTS
The full and complete test outputs are on the World Wide Web; the
download URL is
http://www.mjohnson.com/buyhold/results.zip (130 Kbytes)
I have also put them in the AAFT's group email server "vault",
in the "DataFiles" folder. And the text of the message you are
reading now, is in the "TextDocs" folder.
Here are the first few lines of a backtest results file:
Open Trade Cash+Tbill Total Equity
===================================================================
820421 buy SP82M 116.35 0.00 58175.00 58175.00
820608 sell SP82M 108.50 -4000.00 54175.00 54175.00
820608 buy SP82U 107.80 0.00 54175.00 54175.00
820715 purc TBILL 53333.33 2875.00 54175.00 57050.00
820908 sell SP82U 122.65 7350.00 61525.00 61525.00
820908 buy SP82Z 123.25 0.00 61525.00 61525.00
821015 matu TBILL 54000.00 5850.00 62151.67 68001.67
First, we see that the account began with $58,175.00. One contract
of June 1982 S&P futures was bought at 116.35 on 21 April 1982.
On 08 June 1982, the June contract was sold at 108.50 and a September
contract was bought at 107.80. The June contract was sold at a
loss of 7.85 points (-$3925). Additionally, slippage of 0.05
points ($25) was encountered, and a commission of $50 was paid.
So the net profit on this trade was -$4000.00, as shown in the
second line above.
On 15 July 1982, cash in the account was used to purchase a
3 month T-bill whose face value was $54,000. Since this Tbill
yielded 5.00% annually (1.25% quarterly), the purchase price was
$53,333.33. { math: 54000 / 1.0125 = 53,333.33 }
On 08 September 1982, the futures position was rolled over
again, this time from the Sept contract into the Dec contract.
The Sept contract was sold for 122.65, for a net profit of
+$7350. (14.85 points profit, minus 0.05 points slippage,
minus 50 dollars commission).
On 15 October 1982, the T-bill matures. Three months ago
we bought this bill for $53,333.33 and now we redeem it
for $54,000 (minus $40 T-bill commission). So the account
equity is increased by $626.67 on this day.
Note: the examples above only show days on which trades
or T-bill purchases were made. The full and complete test
output files contain *daily* data, including those days
during the "hold" part of "buy-and-hold". Daily equity
fluctuations (due to price movements in the futures
contract) are shown, giving a complete "equity curve".
Power users will remark that the combination of the
"sort" command, and the "find" command (MS-DOS) [or the
"grep" command in Unix/Linux], is very handy for extracting
specific subsets of the data from the complete result
files. Want the futures trades ONLY? use "find" to grab
the "buy" and "sell" lines, then "sort". Done.
DISCUSSION OF RESULTS: BUY AND HOLD THE S&P FUTURES
DISCUSSION OF RESULTS: BUY AND HOLD THE S&P FUTURES
Immediately we see that the T-bill interest is extremely
significant. Depending on starting equity, T-bills earn
from a quarter to a third of the total profits!
Starting equity............ 29088.00 58175.00 120000.00
Profit from T-bills........ 157033.09 195687.41 277860.25
Profit from futures........ 485750.00 485750.00 485750.00
Wow. We must remember to include T-bills when testing
(and trading!) buy-and-hold.
Next we observe just how gut-wrenching the October 1987
drawdown would have been for buy-and-hold traders. This event
produced the maximum drawdown in the entire test; notice that
in one case the drawdown exceeded 50 percent!
Starting equity............... 29088.00 58175.00 120000.00
Max Drawdown (percent)........ 51.68 40.06 26.92
Compound Annual Growth Rate %. 19.95 15.87 12.26
(CAGR / MAXDD) Ratio.......... 0.386 0.396 0.456
In exchange for suffering this awful body-blow, what pleasures does
the buy-and-hold method provide? Somewhere between 12 and 20 percent
compound annual growth rate. If Compound_Annual_Growth_Rate is
"gain" and Max_Drawdown is "pain", then the gain-to-pain ratio
is simply appalling: between 0.4 and 0.45. The drawdown percentage
is two and a half times bigger than the compound annual growth rate.
This is horrifyingly poor, compared to many many other futures
trading systems.
Also poor are the Sharpe ratio results, the semideviation ratio
results, and the return retracement ratio results. As benchmarks
go, this one performs BADLY -- it will be easy to find mechanical
trading systems that score much better on these performance
measurements. But the "Buy and Hold the S&P Futures" investment
method performs ESPECIALLY poorly on Keltner's K-ratio test.
The K-ratio test (TASC magazine, March 1996, pp. 46-50) measures
the smoothness of a system's equity curve. This test is built
upon the assumption that a perfect ideal system would have an
equity curve that is a razor-straight line (with no zig-zags),
sloping gently upwards. In that case the K-Ratio would be an
infinitely big number. Lars Keltner says (page 50):
"I look for systems with a K-ratio of 1.0 or better for
individual commodities, and values of 3.0 or higher for
diversified portfolios of commodities."
Unfortunately, the "Buy and Hold the S&P 500 Futures" system
does not produce the kind of result that Keltner wants to see:
Starting equity........ 29088.00 58175.00 120000.00
Keltner K-Ratio........ 0.000023 0.000024 0.000027
It will be easy to find mechanical trading systems that score
much MUCH better on the K-ratio test, than this buy-and-hold
reference method.
CONCLUSION
CONCLUSION
A reference investment method, "Buy and Hold the S&P Futures",
was tested using actual contract data and performing actual
rollovers. Three different account equity values were tried.
Drawdown was very large, compared to Compound_Annual_Growth_Rate
(the ratio CAGR/MAXDD was only 0.39 to 0.46). According to the
K-ratio measurement, the equity curve was extremely non-smooth.
The author asserts that this benchmark reference system is a
"poor" performer, i.e. that readers will find it easy to
discover 100% mechanical futures trading systems that handily
outperform this reference system. Which is very good news.
I hope you enjoyed reading this.
Mark Johnson mark@xxxxxxxxxxxx
25 July 1999
--
Mark Johnson Silicon Valley, California mark@xxxxxxxxxxxx
"... The world will little note, nor long remember, what we
say here..." -Abraham Lincoln, "The Gettysburg Address"
|