|
PureBytes Links
Trading Reference Links
|
On Wed, 5 Nov 2003 20:48:59 -0700, in Amibroker you wrote:
> Hi Thomas -
Hello Howard!
>
>
>
> Remember to apply the analysis to trades in the "out of sample" period
> following the backtest, search, and optimization "in sample" period. The
> best use of in sample results is selection of parameter values or tickers to
> test forward. Do not get excited over the profitability of the in sample
> results, no matter how good they are.
You are absolutely right. That's exactly how I'm doing my tests.
Nevertheless I think the chi-squared has some value because it
facilitates the judgment if a system is promising at a rather early
stage.
>
>
>
> The chi-squared test you describe is testing only the proportion of winning
> trades among all trades.
That's not quite correct. I you look at the nominator of the formula
you will notice that R-W is calculated as an absolute figure.
Therefore the chi-squared will also confirm that the bad results of a
lousy system are statistically significant.
> There may be circumstances where that is exactly
> what you want. But if the system is trading a stock directly (not being
> used as a filter), it is a poor system. For example: a system that produces
> 100 winners, each of 1 percent, and 30 losers, each of 10 percent, will look
> like a good system under a chi-squared test - 36 or so by the back of the
> envelope method. But most people would prefer better return and less
> drawdown.
I agree on that. That's why the chi-squared should only be used in
connection with the other figures in the performance report and a
thorough evaluation of the equity line.
Regards, Thomas
>
>
>
> Howard
>
>
>
> -----Original Message-----
> From: Thomas Ludwig [mailto:Thomas.Ludwig@xxxxxx]
> Sent: Monday, November 03, 2003 1:49 PM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: [amibroker] Backtesting and statistics
>
>
>
> Hi all!
> The performance summary in the backtesting report window contains a
> lot of figures. But as useful as these figures are - they do not
> really tell us if the backtesting results are statistically
> significant.
>
> Therefore I suggest to add a new figure proposed by Arthur Merrill in
> Stocks & Commodities some years ago (Bonus Issue 1993). It's called
> the "chi squared with one degree of freedom, with the Yates
> correction". It's calculated using the following simple formula:
>
> X^2 = ( |R-W| -1)^2 / (R+W)
> where: R = number of times right
> W = number of times wrong
>
> Interpetation:
> Below 3.84: Significance doubtful.
> Above 3.84: Probably significant. Probability of one in 20 that the
> result was by chance.
> Above 6.64: Significant. Probability of one in 100 that the result was
> by chance.
> Above 10.83: Highly significant. Probability of one in 1,000 that the
> result was by chance.
>
> Let`s apply this formula to a simple example. Let's assume that the
> backtest gives 10 signals with 7 of them profitable; let's also assume
> that the other figures in the backtest report are okay. So the results
> look good at the first glance. However, the chi squared figure is just
> 0.9 - very insignificant!
> On the other hand, if the backtest gives 100 signals with 70 of them
> profitable the chi squared figure is a whopping 15.21 - highly
> significant!
>
> Of course, these are very simple examples and the difference between
> them is rather obvious. But that might not be the case in other
> examples. And I'm afraid that even obvious things are often
> overlooked, especially if the other figures in the backtest report
> look good. Therefore in my opinion the chi squared figure would be a
> valuable addition for the correct interpretation of the backtest
> results.
>
> Tomasz - any chance to add this?
>
> Regards, Thomas
--
Thomas Ludwig
PGP key (RSA) available on request.
Fingerprint: 4356 55FF 3412 277A 9741 1CFA B9A6 7B90
------------------------ Yahoo! Groups Sponsor ---------------------~-->
Buy Ink Cartridges or Refill Kits for your HP, Epson, Canon or Lexmark
Printer at MyInks.com. Free s/h on orders $50 or more to the US & Canada.
http://www.c1tracking.com/l.asp?cid=5511
http://us.click.yahoo.com/mOAaAA/3exGAA/qnsNAA/GHeqlB/TM
---------------------------------------------------------------------~->
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
|