|
PureBytes Links
Trading Reference Links
|
On Mon, 03 Nov 2003 21:31:38 -0000, in Amibroker you wrote:
> If Tomasz ever did add this, it would much more useful for the report
> to contain the p values rather than raw chi-square. Only problem is,
> with any large backtest you'll get a lot of "significant" results by
> chance alone. Then you'd need another test, lol, to see if you have
> more significant results than you'd expect by chance given the size of
> the backtest.
I'm not saying that you should solely rely on the chi-square to
evaluate a backtest. But I think it gives valuable additional info
about an aspect that is probably often overlooked.
> I'm a huge fan of Arthur Merrill but can think of more
> useful (statistical) functionality to add (in my opinion) if Tomasz
> were ever so inclined.
What do you propose?
> BTW, this is very easy to do in a spreadsheet
> with results pasted from AA. You could even make a template so it
> would be nearly effortless.
Sure, but I think integrating this figure in the report window is
easier for most users.
Regards, Thomas
>
> --- In amibroker@xxxxxxxxxxxxxxx, Thomas Ludwig <Thomas.Ludwig@xxxx>
> wrote:
>> Hi all!
>> The performance summary in the backtesting report window contains a
>> lot of figures. But as useful as these figures are - they do not
>> really tell us if the backtesting results are statistically
>> significant.
>>
>> Therefore I suggest to add a new figure proposed by Arthur Merrill
> in
>> Stocks & Commodities some years ago (Bonus Issue 1993). It's called
>> the "chi squared with one degree of freedom, with the Yates
>> correction". It's calculated using the following simple formula:
>>
>> X^2 = ( |R-W| -1)^2 / (R+W)
>> where: R = number of times right
>> W = number of times wrong
>>
>> Interpetation:
>> Below 3.84: Significance doubtful.
>> Above 3.84: Probably significant. Probability of one in 20 that the
>> result was by chance.
>> Above 6.64: Significant. Probability of one in 100 that the result
> was
>> by chance.
>> Above 10.83: Highly significant. Probability of one in 1,000 that
> the
>> result was by chance.
>>
>> Let`s apply this formula to a simple example. Let's assume that the
>> backtest gives 10 signals with 7 of them profitable; let's also
> assume
>> that the other figures in the backtest report are okay. So the
> results
>> look good at the first glance. However, the chi squared figure is
> just
>> 0.9 - very insignificant!
>> On the other hand, if the backtest gives 100 signals with 70 of them
>> profitable the chi squared figure is a whopping 15.21 - highly
>> significant!
>>
>> Of course, these are very simple examples and the difference between
>> them is rather obvious. But that might not be the case in other
>> examples. And I'm afraid that even obvious things are often
>> overlooked, especially if the other figures in the backtest report
>> look good. Therefore in my opinion the chi squared figure would be
> a
>> valuable addition for the correct interpretation of the backtest
>> results.
>>
>> Tomasz - any chance to add this?
>>
>> Regards, Thomas
>> --
>> Thomas Ludwig
>> PGP key (RSA) available on request.
>> Fingerprint: 4356 55FF 3412 277A 9741 1CFA B9A6 7B90
>
>
>
> Send BUG REPORTS to bugs@xxxxxxxxxxxxx
> Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
> -----------------------------------------
> Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
> (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> --------------------------------------------
> Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
--
Thomas Ludwig
PGP key (RSA) available on request.
Fingerprint: 4356 55FF 3412 277A 9741 1CFA B9A6 7B90
------------------------ Yahoo! Groups Sponsor ---------------------~-->
Rent DVDs from home.
Over 14,500 titles. Free Shipping
& No Late Fees. Try Netflix for FREE!
http://us.click.yahoo.com/I3w.vC/hP.FAA/3jkFAA/GHeqlB/TM
---------------------------------------------------------------------~->
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
|