|
PureBytes Links
Trading Reference Links
|
Hi Thomas –
Remember to apply the analysis to trades
in the “out of sample” period following the backtest, search, and
optimization “in sample” period. The best use of in sample
results is selection of parameter values or tickers to test forward. Do
not get excited over the profitability of the in sample results, no matter how
good they are.
The chi-squared test you describe is
testing only the proportion of winning trades among all trades. There may
be circumstances where that is exactly what you want. But if the system
is trading a stock directly (not being used as a filter), it is a poor
system. For example: a system that produces 100 winners, each of 1
percent, and 30 losers, each of 10 percent, will look like a good system under
a chi-squared test – 36 or so by the back of the envelope method. But
most people would prefer better return and less drawdown.
Howard
-----Original Message-----
From: Thomas Ludwig
[mailto:Thomas.Ludwig@xxxxxx]
Sent: Monday, November 03, 2003
1:49 PM
To: amibroker@xxxxxxxxxxxxxxx
Subject: [amibroker] Backtesting
and statistics
<font size=2
face="Courier New">Hi all!<font
size=2 face="Courier New">
The performance summary in the backtesting report
window contains a
lot of figures. But as useful as these figures are
- they do not
really tell us if the backtesting results
are statistically
significant.
Therefore I suggest to add a new figure proposed
by Arthur Merrill in
Stocks & Commodities some years ago (Bonus
Issue 1993). It's called
the "chi squared with one degree of freedom,
with the Yates
correction". It's calculated using the
following simple formula:
X^2 = ( |R-W| -1)^2 / (R+W)
where: R = number of times right
W = number of times wrong
Interpetation:
Below 3.84: Significance doubtful.
Above 3.84: Probably significant. Probability of
one in 20 that the
result was by chance.
Above 6.64: Significant. Probability of one in 100
that the result was
by chance.
Above 10.83: Highly significant. Probability of
one in 1,000 that the
result was by chance.
Let`s apply this formula to a simple example.
Let's assume that the
backtest gives 10 signals with 7 of them
profitable; let's also assume
that the other figures in the backtest report are
okay. So the results
look good at the first glance. However, the chi
squared figure is just
0.9 - very insignificant!
On the other hand, if the backtest gives 100
signals with 70 of them
profitable the chi squared figure is a whopping
15.21 - highly
significant!
Of course, these are very simple examples and the
difference between
them is rather obvious. But that might not be the
case in other
examples. And I'm afraid that even obvious things
are often
overlooked, especially if the other figures in the
backtest report
look good. Therefore in my opinion the chi
squared figure would be a
valuable addition for the correct interpretation
of the backtest
results.
Tomasz - any chance to add this?
Regards, Thomas
--
Thomas Ludwig
PGP key (RSA) available on request.
Fingerprint: 4356 55FF 3412 277A 9741 1CFA
B9A6 7B90
<font size=2
color="#003399" face="Times New Roman">Send
BUG REPORTS to <span
>bugs@xxxxxxxxxxxxx<font size=2
face="Courier New">
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: <font
size=2 face="Courier New">amiquote@xxxxxxxxxxxxxxx<font
size=2 face="Courier New"> <font
size=2 face="Courier New">
(Web page: <a
href="">http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: <a
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo!
Groups is subject to the Yahoo!
Terms of Service.
Yahoo! Groups Sponsor
ADVERTISEMENT
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
|