[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] Re: Statistical tests as custom metrics



PureBytes Links

Trading Reference Links

> Some of the 'academic' stats methods are not quite as good a fit to 
> trading as we first assume e.g. look at the recent discussion on 
> Fourier Transform, which comes from Signal Processing (electronics?) 
> and falls down rather spectacularly in trading because stockmarket 
> data isn't stationary.

For what is worth, FFT is most often applied to audio signals, that
are definitelly NOT stationary :-)  so that argument is not really valid :-)
There are methods (windowing+padding) to increase FFT resolution on short data samples,
and there are also methods different than FFT for spectal analysis. 
But that is rather broad subject ....

Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message ----- 
From: "brian_z111" <brian_z111@xxxxxxxxx>
To: <amibroker@xxxxxxxxxxxxxxx>
Sent: Wednesday, March 12, 2008 12:41 AM
Subject: [amibroker] Re: Statistical tests as custom metrics


> Thomas,
> 
> I have included Jeffs question with your opening query since they fit 
> so closely.
> 
>> looking at your algorithm I'm not seeing expected.  What exactly 
>>are 
>> you test significance against?
> 
> The expected value of an unbiased binomial event is 0.50 (50% win).
> 
> Since the (equity) market has a slight upward bias we need to use an 
> expected value, that includes the market bias, as the base line (or 
> detrend as per Howards/Aronsons methods).
> 
> (Personally I don't detrend as I like to condition myself to see the 
> markets as they actually are, and mentally factor in the trend, so I 
> use market bias benchmarking).
> 
>> > in "Quantitative Trading Systems" on p. 256, Howard describes a z-
>> score 
>> > test in order to evaluate the statistical significance of a 
> trading 
>> > system. While the formula is easy to write in AFL, I don't think 
>> that 
>> > it can be done as a custom metric since the system to be 
> evaluated 
>> is 
>> > compared with a Random System. Any idea how to sensibly implement 
>> it in 
>> > Amibroker?
> 
> At page 91,in his book, (Entries and Exits chapter) Howard gives some 
> very good (random entry) examples of how we can get an estimation for 
> the 'standardized binomial expectancy' of any market i.e. you can get 
> the mean expected wins for the actual market you are testing your 
> system in and use that in the Z score calculation - I think you would 
> be better to use random entries with an exit after a set number of 
> days == the average time your system trades are in the market. 
> 
> I am still learning AB myself so I am not sure if we can implement 
> Howards Z equation directly in AB - you will probably have to do it 
> outside somewhere - I haven't figured out how we can get the SD of a 
> trade series (from a backtest) in AB - anyway we don't have built in 
> stats tables (I suppose you could manually plug in the typical Z 
> scores).
> 
> I'm exporting to Excel and doing my evaluations there, but I don't 
> get that fancy.
> 
> > > I'm using another statistical test proposed by the late Arthur 
>> Merrill 
>> > some years ago in S&C. It's the "chi squared with one degree of 
>> > freedom, with the Yates correction". Here's how I implemented it 
> in 
>> AB:
>> > 
>> > //chi squared with one degree of freedom, with the Yates 
> correction
>> >     wi=st.GetValue("WinnersQty");
>> >     Lo=st.GetValue("LosersQty");
>> >     Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
>> >     bo.AddCustomMetric( "Chi-Squared modif.: >10.83: very     
>> > significant(1000:1), >6.64: significant (100:1) , >3.84: probably 
>> > significant (20:1), <3.84: significance doubtful", Chi ); 
>> > 
>> > What do you think about this metric? 
> 
> I think it is a very conservative measure.
> 
> One of the problems we have, in evaluation, is that 'academic' 
> statistics filtered into freelance trading via institutional 
> investing - nothing against academics or institutional traders but 
> their focus is somewhat different to freelance traders.
> 
> Some of the 'academic' stats methods are not quite as good a fit to 
> trading as we first assume e.g. look at the recent discussion on 
> Fourier Transform, which comes from Signal Processing (electronics?) 
> and falls down rather spectacularly in trading because stockmarket 
> data isn't stationary.
> 
> Most of the stats we are using assume stationarity and also assume 
> that data will be normal/ random i.e. it will have a normal 
> distribution and that the datapoints are independent of each other.
> 
> Neither is absolutely true, so the stats we are using are 
> approximations (of course the data we are using is only an 
> approximation anyway) - hence the doubts about Merrill's Chi.
> 
>> > While this metric doesn't tell you anything if your system is 
>> > profitable, it tells you if its signals are only pure coincidence 
>> > (simply put). It's remarkable that many systems that seem to be 
>> > promising according to the usual metrics, are below 3.84, i.e. 
>> > significance doubtful. You need either a rather high number of 
>> trades 
>> > or a very high percentage of winning trades to shift this metric 
>> > significantly higher. At least for (medium-term) EOD systems 
>> (that's 
>> > what I trade) this is not easy to achieve.
>> > 
> 
> Yes, it is very hard to find good trading systems.
> 
> This is what I have found - many tests that come up with nothing, 
> especially in the first two years.
> 
>>>Are there other "better" 
>> > statistical metrics? If yes - would you mind sharing the AFL code?
>> > 
> 
> Try Howards Z method, using his random code, to find your expected 
> win rate for your market and see how that works out.
> 
> I have started some original (to me) work, based on binomial 
> simulation of equity curves and the behaviour of random, 50/50, 
> trading systems.
> 
> It is only at the experimental, concept stage.
> I intend posting it to the UKB one day so that the mathematically 
> trained people in the forum can critique it (it might be a load of 
> old rubbish for all I know).
> 
> Based on that work I am using PowerFactor, with sample error, to 
> guestimate significance (I can quickly do that in my head).
> 
> Note that in PowerFactor the binomial component is considered to be 
> Gaussian, with independent variables, while the distribution of the 
> trades (ave%won/ave%lost) is not.
> 
> Because of that I only apply the significance test to the W/L 
> binomial component (I claim that carrying out stats analysis on the 
> compound system results is biased because of the non-normal nature of 
> the distribution etc).
> 
> For binomial events:
> 
> variance == sample error (sort of)
> 
> For 100 trades:
> 
> sample error = +_10%;
> expected random result (benchmark) == 50 wins;
> 
> a no win trade will have a range of:
> 
> 45-50 wins (one standard dev)
> 40-60 wins (two standard devs)
> 35- 65 (three standard devs) etc
> 
> So for 100 trades 60 wins doesn't happen all that often, if the coin 
> is a fair coin (a random event).
> 
> We are defintely going to take notice of 60/100 wins BUT it is 
> not 'out of this world' and we do not have certainty - we only have 
> the expectation that it is good - reality can, and does, dash 
> expectations on occasion.
> 
> Because of this, W/L results are never a sure thing.
> 
> To ensure against this take control of the ave%W/ave%L ratio - that 
> is something we can control via stops - if the W/L ratio turns out to 
> be a 'BlackSwan' our good stops will save us from crashing and 
> burning (keep us at low drawdowns).
> 
> Comparing to Chi (for 100 trades with a 60% win record):
> 
> Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
> 
> == ((60-40)-1)^2/(wi+Lo);
> == 19^2/100;
> == 361/100
> == 3.6
> == Not significant according to Chi but significant according to 
> brian (always look on the bright side of life!).
> 
> I agree that finding 60% winners, in any market or timeframe, is very 
> difficult - that is the reality of trading.
> 
> This problem is especially prevalent in mid - long term trading - say 
> indicators with long lookbacks are used - then the number of signals 
> available tends towards becoming a rare event and the trader then can 
> only see a small part of the longterm (10000 plus) trades - the 
> trader soon runs out of clean data and can't get high enough trade 
> counts.
> 
> That is why I like shorter term trading (intraday to 2-3 day cycles) 
> where I can take advantage of statistical smoothing (I quickly 
> approach my theoretical edge i.e. relative to the calendar days).
> 
> 
> As I said - please use 'my' theories at your own risk, at least until 
> after I post on the topic, and the mathematicians in the forum have a 
> chance to bash up my hypotheses.
> 
> 
> brian_z
> 
> 
> 
> 
> 
> 
> --- In amibroker@xxxxxxxxxxxxxxx, "jeffro861" <jeffro861@xxx> wrote:
>>
>> Ok, so the chi-squared tests for independence (real vs. expected) 
> so, 
>> looking at your algorithm I'm not seeing expected.  What exactly 
> are 
>> you test significance against?
>> 
>> --- In amibroker@xxxxxxxxxxxxxxx, Thomas Ludwig <Thomas.Ludwig@> 
>> wrote:
>> >
>> > Hello,
>> > 
>> > in "Quantitative Trading Systems" on p. 256, Howard describes a z-
>> score 
>> > test in order to evaluate the statistical significance of a 
> trading 
>> > system. While the formula is easy to write in AFL, I don't think 
>> that 
>> > it can be done as a custom metric since the system to be 
> evaluated 
>> is 
>> > compared with a Random System. Any idea how to sensibly implement 
>> it in 
>> > Amibroker?
>> > 
>> > I'm using another statistical test proposed by the late Arthur 
>> Merrill 
>> > some years ago in S&C. It's the "chi squared with one degree of 
>> > freedom, with the Yates correction". Here's how I implemented it 
> in 
>> AB:
>> > 
>> > //chi squared with one degree of freedom, with the Yates 
> correction
>> >     wi=st.GetValue("WinnersQty");
>> >     Lo=st.GetValue("LosersQty");
>> >     Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
>> >     bo.AddCustomMetric( "Chi-Squared modif.: >10.83: very     
>> > significant(1000:1), >6.64: significant (100:1) , >3.84: probably 
>> > significant (20:1), <3.84: significance doubtful", Chi ); 
>> > 
>> > While this metric doesn't tell you anything if your system is 
>> > profitable, it tells you if its signals are only pure coincidence 
>> > (simply put). It's remarkable that many systems that seem to be 
>> > promising according to the usual metrics, are below 3.84, i.e. 
>> > significance doubtful. You need either a rather high number of 
>> trades 
>> > or a very high percentage of winning trades to shift this metric 
>> > significantly higher. At least for (medium-term) EOD systems 
>> (that's 
>> > what I trade) this is not easy to achieve.
>> > 
>> > What do you think about this metric? Are there other "better" 
>> > statistical metrics? If yes - would you mind sharing the AFL code?
>> > 
>> > Best regards,
>> > 
>> > Thomas
>> >
>>
> 
> 
> 
> 
> Please note that this group is for discussion between users only.
> 
> To get support from AmiBroker please send an e-mail directly to 
> SUPPORT {at} amibroker.com
> 
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
> 
> For other support material please check also:
> http://www.amibroker.com/support.html
> 
> Yahoo! Groups Links
> 
> 
> 


Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to 
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html
 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:amibroker-digest@xxxxxxxxxxxxxxx 
    mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/