[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[amibroker] Re: Statistical tests as custom metrics



PureBytes Links

Trading Reference Links

> There are methods (windowing+padding) to increase FFT resolution on 
>short data samples,
> and there are also methods different than FFT for spectal analysis. 
> But that is rather broad subject ....

I have my own simple and direct methods of dealing with waves, so I 
haven't felt the need to go down those paths, but, in the interests 
of 'breadth of knowledge' I made a mental note to have a closer look 
at spectral analysis and ElliotWaves (based on feedback from the 
forum).

Thanks for your comments - always appreciated.

brian_z

--- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@xxx> 
wrote:
>
> > Some of the 'academic' stats methods are not quite as good a fit 
to 
> > trading as we first assume e.g. look at the recent discussion on 
> > Fourier Transform, which comes from Signal Processing 
(electronics?) 
> > and falls down rather spectacularly in trading because 
stockmarket 
> > data isn't stationary.
> 
> For what is worth, FFT is most often applied to audio signals, that
> are definitelly NOT stationary :-)  so that argument is not really 
valid :-)
> There are methods (windowing+padding) to increase FFT resolution on 
short data samples,
> and there are also methods different than FFT for spectal analysis. 
> But that is rather broad subject ....
> 
> Best regards,
> Tomasz Janeczko
> amibroker.com
> ----- Original Message ----- 
> From: "brian_z111" <brian_z111@xxx>
> To: <amibroker@xxxxxxxxxxxxxxx>
> Sent: Wednesday, March 12, 2008 12:41 AM
> Subject: [amibroker] Re: Statistical tests as custom metrics
> 
> 
> > Thomas,
> > 
> > I have included Jeffs question with your opening query since they 
fit 
> > so closely.
> > 
> >> looking at your algorithm I'm not seeing expected.  What exactly 
> >>are 
> >> you test significance against?
> > 
> > The expected value of an unbiased binomial event is 0.50 (50% 
win).
> > 
> > Since the (equity) market has a slight upward bias we need to use 
an 
> > expected value, that includes the market bias, as the base line 
(or 
> > detrend as per Howards/Aronsons methods).
> > 
> > (Personally I don't detrend as I like to condition myself to see 
the 
> > markets as they actually are, and mentally factor in the trend, 
so I 
> > use market bias benchmarking).
> > 
> >> > in "Quantitative Trading Systems" on p. 256, Howard describes 
a z-
> >> score 
> >> > test in order to evaluate the statistical significance of a 
> > trading 
> >> > system. While the formula is easy to write in AFL, I don't 
think 
> >> that 
> >> > it can be done as a custom metric since the system to be 
> > evaluated 
> >> is 
> >> > compared with a Random System. Any idea how to sensibly 
implement 
> >> it in 
> >> > Amibroker?
> > 
> > At page 91,in his book, (Entries and Exits chapter) Howard gives 
some 
> > very good (random entry) examples of how we can get an estimation 
for 
> > the 'standardized binomial expectancy' of any market i.e. you can 
get 
> > the mean expected wins for the actual market you are testing your 
> > system in and use that in the Z score calculation - I think you 
would 
> > be better to use random entries with an exit after a set number 
of 
> > days == the average time your system trades are in the market. 
> > 
> > I am still learning AB myself so I am not sure if we can 
implement 
> > Howards Z equation directly in AB - you will probably have to do 
it 
> > outside somewhere - I haven't figured out how we can get the SD 
of a 
> > trade series (from a backtest) in AB - anyway we don't have built 
in 
> > stats tables (I suppose you could manually plug in the typical Z 
> > scores).
> > 
> > I'm exporting to Excel and doing my evaluations there, but I 
don't 
> > get that fancy.
> > 
> > > > I'm using another statistical test proposed by the late 
Arthur 
> >> Merrill 
> >> > some years ago in S&C. It's the "chi squared with one degree 
of 
> >> > freedom, with the Yates correction". Here's how I implemented 
it 
> > in 
> >> AB:
> >> > 
> >> > //chi squared with one degree of freedom, with the Yates 
> > correction
> >> >     wi=st.GetValue("WinnersQty");
> >> >     Lo=st.GetValue("LosersQty");
> >> >     Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
> >> >     bo.AddCustomMetric( "Chi-Squared modif.: >10.83: very     
> >> > significant(1000:1), >6.64: significant (100:1) , >3.84: 
probably 
> >> > significant (20:1), <3.84: significance doubtful", Chi ); 
> >> > 
> >> > What do you think about this metric? 
> > 
> > I think it is a very conservative measure.
> > 
> > One of the problems we have, in evaluation, is that 'academic' 
> > statistics filtered into freelance trading via institutional 
> > investing - nothing against academics or institutional traders 
but 
> > their focus is somewhat different to freelance traders.
> > 
> > Some of the 'academic' stats methods are not quite as good a fit 
to 
> > trading as we first assume e.g. look at the recent discussion on 
> > Fourier Transform, which comes from Signal Processing 
(electronics?) 
> > and falls down rather spectacularly in trading because 
stockmarket 
> > data isn't stationary.
> > 
> > Most of the stats we are using assume stationarity and also 
assume 
> > that data will be normal/ random i.e. it will have a normal 
> > distribution and that the datapoints are independent of each 
other.
> > 
> > Neither is absolutely true, so the stats we are using are 
> > approximations (of course the data we are using is only an 
> > approximation anyway) - hence the doubts about Merrill's Chi.
> > 
> >> > While this metric doesn't tell you anything if your system is 
> >> > profitable, it tells you if its signals are only pure 
coincidence 
> >> > (simply put). It's remarkable that many systems that seem to 
be 
> >> > promising according to the usual metrics, are below 3.84, i.e. 
> >> > significance doubtful. You need either a rather high number of 
> >> trades 
> >> > or a very high percentage of winning trades to shift this 
metric 
> >> > significantly higher. At least for (medium-term) EOD systems 
> >> (that's 
> >> > what I trade) this is not easy to achieve.
> >> > 
> > 
> > Yes, it is very hard to find good trading systems.
> > 
> > This is what I have found - many tests that come up with nothing, 
> > especially in the first two years.
> > 
> >>>Are there other "better" 
> >> > statistical metrics? If yes - would you mind sharing the AFL 
code?
> >> > 
> > 
> > Try Howards Z method, using his random code, to find your 
expected 
> > win rate for your market and see how that works out.
> > 
> > I have started some original (to me) work, based on binomial 
> > simulation of equity curves and the behaviour of random, 50/50, 
> > trading systems.
> > 
> > It is only at the experimental, concept stage.
> > I intend posting it to the UKB one day so that the mathematically 
> > trained people in the forum can critique it (it might be a load 
of 
> > old rubbish for all I know).
> > 
> > Based on that work I am using PowerFactor, with sample error, to 
> > guestimate significance (I can quickly do that in my head).
> > 
> > Note that in PowerFactor the binomial component is considered to 
be 
> > Gaussian, with independent variables, while the distribution of 
the 
> > trades (ave%won/ave%lost) is not.
> > 
> > Because of that I only apply the significance test to the W/L 
> > binomial component (I claim that carrying out stats analysis on 
the 
> > compound system results is biased because of the non-normal 
nature of 
> > the distribution etc).
> > 
> > For binomial events:
> > 
> > variance == sample error (sort of)
> > 
> > For 100 trades:
> > 
> > sample error = +_10%;
> > expected random result (benchmark) == 50 wins;
> > 
> > a no win trade will have a range of:
> > 
> > 45-50 wins (one standard dev)
> > 40-60 wins (two standard devs)
> > 35- 65 (three standard devs) etc
> > 
> > So for 100 trades 60 wins doesn't happen all that often, if the 
coin 
> > is a fair coin (a random event).
> > 
> > We are defintely going to take notice of 60/100 wins BUT it is 
> > not 'out of this world' and we do not have certainty - we only 
have 
> > the expectation that it is good - reality can, and does, dash 
> > expectations on occasion.
> > 
> > Because of this, W/L results are never a sure thing.
> > 
> > To ensure against this take control of the ave%W/ave%L ratio - 
that 
> > is something we can control via stops - if the W/L ratio turns 
out to 
> > be a 'BlackSwan' our good stops will save us from crashing and 
> > burning (keep us at low drawdowns).
> > 
> > Comparing to Chi (for 100 trades with a 60% win record):
> > 
> > Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
> > 
> > == ((60-40)-1)^2/(wi+Lo);
> > == 19^2/100;
> > == 361/100
> > == 3.6
> > == Not significant according to Chi but significant according to 
> > brian (always look on the bright side of life!).
> > 
> > I agree that finding 60% winners, in any market or timeframe, is 
very 
> > difficult - that is the reality of trading.
> > 
> > This problem is especially prevalent in mid - long term trading - 
say 
> > indicators with long lookbacks are used - then the number of 
signals 
> > available tends towards becoming a rare event and the trader then 
can 
> > only see a small part of the longterm (10000 plus) trades - the 
> > trader soon runs out of clean data and can't get high enough 
trade 
> > counts.
> > 
> > That is why I like shorter term trading (intraday to 2-3 day 
cycles) 
> > where I can take advantage of statistical smoothing (I quickly 
> > approach my theoretical edge i.e. relative to the calendar days).
> > 
> > 
> > As I said - please use 'my' theories at your own risk, at least 
until 
> > after I post on the topic, and the mathematicians in the forum 
have a 
> > chance to bash up my hypotheses.
> > 
> > 
> > brian_z
> > 
> > 
> > 
> > 
> > 
> > 
> > --- In amibroker@xxxxxxxxxxxxxxx, "jeffro861" <jeffro861@> wrote:
> >>
> >> Ok, so the chi-squared tests for independence (real vs. 
expected) 
> > so, 
> >> looking at your algorithm I'm not seeing expected.  What exactly 
> > are 
> >> you test significance against?
> >> 
> >> --- In amibroker@xxxxxxxxxxxxxxx, Thomas Ludwig <Thomas.Ludwig@> 
> >> wrote:
> >> >
> >> > Hello,
> >> > 
> >> > in "Quantitative Trading Systems" on p. 256, Howard describes 
a z-
> >> score 
> >> > test in order to evaluate the statistical significance of a 
> > trading 
> >> > system. While the formula is easy to write in AFL, I don't 
think 
> >> that 
> >> > it can be done as a custom metric since the system to be 
> > evaluated 
> >> is 
> >> > compared with a Random System. Any idea how to sensibly 
implement 
> >> it in 
> >> > Amibroker?
> >> > 
> >> > I'm using another statistical test proposed by the late Arthur 
> >> Merrill 
> >> > some years ago in S&C. It's the "chi squared with one degree 
of 
> >> > freedom, with the Yates correction". Here's how I implemented 
it 
> > in 
> >> AB:
> >> > 
> >> > //chi squared with one degree of freedom, with the Yates 
> > correction
> >> >     wi=st.GetValue("WinnersQty");
> >> >     Lo=st.GetValue("LosersQty");
> >> >     Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
> >> >     bo.AddCustomMetric( "Chi-Squared modif.: >10.83: very     
> >> > significant(1000:1), >6.64: significant (100:1) , >3.84: 
probably 
> >> > significant (20:1), <3.84: significance doubtful", Chi ); 
> >> > 
> >> > While this metric doesn't tell you anything if your system is 
> >> > profitable, it tells you if its signals are only pure 
coincidence 
> >> > (simply put). It's remarkable that many systems that seem to 
be 
> >> > promising according to the usual metrics, are below 3.84, i.e. 
> >> > significance doubtful. You need either a rather high number of 
> >> trades 
> >> > or a very high percentage of winning trades to shift this 
metric 
> >> > significantly higher. At least for (medium-term) EOD systems 
> >> (that's 
> >> > what I trade) this is not easy to achieve.
> >> > 
> >> > What do you think about this metric? Are there other "better" 
> >> > statistical metrics? If yes - would you mind sharing the AFL 
code?
> >> > 
> >> > Best regards,
> >> > 
> >> > Thomas
> >> >
> >>
> > 
> > 
> > 
> > 
> > Please note that this group is for discussion between users only.
> > 
> > To get support from AmiBroker please send an e-mail directly to 
> > SUPPORT {at} amibroker.com
> > 
> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> > http://www.amibroker.com/devlog/
> > 
> > For other support material please check also:
> > http://www.amibroker.com/support.html
> > 
> > Yahoo! Groups Links
> > 
> > 
> >
>




Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to 
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html
 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:amibroker-digest@xxxxxxxxxxxxxxx 
    mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/