|
PureBytes Links
Trading Reference Links
|
> Some of the 'academic' stats methods are not quite as good a fit to
> trading as we first assume e.g. look at the recent discussion on
> Fourier Transform, which comes from Signal Processing (electronics?)
> and falls down rather spectacularly in trading because stockmarket
> data isn't stationary.
For what is worth, FFT is most often applied to audio signals, that
are definitelly NOT stationary :-) so that argument is not really valid :-)
There are methods (windowing+padding) to increase FFT resolution on short data samples,
and there are also methods different than FFT for spectal analysis.
But that is rather broad subject ....
Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: "brian_z111" <brian_z111@xxxxxxxxx>
To: <amibroker@xxxxxxxxxxxxxxx>
Sent: Wednesday, March 12, 2008 12:41 AM
Subject: [amibroker] Re: Statistical tests as custom metrics
> Thomas,
>
> I have included Jeffs question with your opening query since they fit
> so closely.
>
>> looking at your algorithm I'm not seeing expected. What exactly
>>are
>> you test significance against?
>
> The expected value of an unbiased binomial event is 0.50 (50% win).
>
> Since the (equity) market has a slight upward bias we need to use an
> expected value, that includes the market bias, as the base line (or
> detrend as per Howards/Aronsons methods).
>
> (Personally I don't detrend as I like to condition myself to see the
> markets as they actually are, and mentally factor in the trend, so I
> use market bias benchmarking).
>
>> > in "Quantitative Trading Systems" on p. 256, Howard describes a z-
>> score
>> > test in order to evaluate the statistical significance of a
> trading
>> > system. While the formula is easy to write in AFL, I don't think
>> that
>> > it can be done as a custom metric since the system to be
> evaluated
>> is
>> > compared with a Random System. Any idea how to sensibly implement
>> it in
>> > Amibroker?
>
> At page 91,in his book, (Entries and Exits chapter) Howard gives some
> very good (random entry) examples of how we can get an estimation for
> the 'standardized binomial expectancy' of any market i.e. you can get
> the mean expected wins for the actual market you are testing your
> system in and use that in the Z score calculation - I think you would
> be better to use random entries with an exit after a set number of
> days == the average time your system trades are in the market.
>
> I am still learning AB myself so I am not sure if we can implement
> Howards Z equation directly in AB - you will probably have to do it
> outside somewhere - I haven't figured out how we can get the SD of a
> trade series (from a backtest) in AB - anyway we don't have built in
> stats tables (I suppose you could manually plug in the typical Z
> scores).
>
> I'm exporting to Excel and doing my evaluations there, but I don't
> get that fancy.
>
> > > I'm using another statistical test proposed by the late Arthur
>> Merrill
>> > some years ago in S&C. It's the "chi squared with one degree of
>> > freedom, with the Yates correction". Here's how I implemented it
> in
>> AB:
>> >
>> > //chi squared with one degree of freedom, with the Yates
> correction
>> > wi=st.GetValue("WinnersQty");
>> > Lo=st.GetValue("LosersQty");
>> > Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
>> > bo.AddCustomMetric( "Chi-Squared modif.: >10.83: very
>> > significant(1000:1), >6.64: significant (100:1) , >3.84: probably
>> > significant (20:1), <3.84: significance doubtful", Chi );
>> >
>> > What do you think about this metric?
>
> I think it is a very conservative measure.
>
> One of the problems we have, in evaluation, is that 'academic'
> statistics filtered into freelance trading via institutional
> investing - nothing against academics or institutional traders but
> their focus is somewhat different to freelance traders.
>
> Some of the 'academic' stats methods are not quite as good a fit to
> trading as we first assume e.g. look at the recent discussion on
> Fourier Transform, which comes from Signal Processing (electronics?)
> and falls down rather spectacularly in trading because stockmarket
> data isn't stationary.
>
> Most of the stats we are using assume stationarity and also assume
> that data will be normal/ random i.e. it will have a normal
> distribution and that the datapoints are independent of each other.
>
> Neither is absolutely true, so the stats we are using are
> approximations (of course the data we are using is only an
> approximation anyway) - hence the doubts about Merrill's Chi.
>
>> > While this metric doesn't tell you anything if your system is
>> > profitable, it tells you if its signals are only pure coincidence
>> > (simply put). It's remarkable that many systems that seem to be
>> > promising according to the usual metrics, are below 3.84, i.e.
>> > significance doubtful. You need either a rather high number of
>> trades
>> > or a very high percentage of winning trades to shift this metric
>> > significantly higher. At least for (medium-term) EOD systems
>> (that's
>> > what I trade) this is not easy to achieve.
>> >
>
> Yes, it is very hard to find good trading systems.
>
> This is what I have found - many tests that come up with nothing,
> especially in the first two years.
>
>>>Are there other "better"
>> > statistical metrics? If yes - would you mind sharing the AFL code?
>> >
>
> Try Howards Z method, using his random code, to find your expected
> win rate for your market and see how that works out.
>
> I have started some original (to me) work, based on binomial
> simulation of equity curves and the behaviour of random, 50/50,
> trading systems.
>
> It is only at the experimental, concept stage.
> I intend posting it to the UKB one day so that the mathematically
> trained people in the forum can critique it (it might be a load of
> old rubbish for all I know).
>
> Based on that work I am using PowerFactor, with sample error, to
> guestimate significance (I can quickly do that in my head).
>
> Note that in PowerFactor the binomial component is considered to be
> Gaussian, with independent variables, while the distribution of the
> trades (ave%won/ave%lost) is not.
>
> Because of that I only apply the significance test to the W/L
> binomial component (I claim that carrying out stats analysis on the
> compound system results is biased because of the non-normal nature of
> the distribution etc).
>
> For binomial events:
>
> variance == sample error (sort of)
>
> For 100 trades:
>
> sample error = +_10%;
> expected random result (benchmark) == 50 wins;
>
> a no win trade will have a range of:
>
> 45-50 wins (one standard dev)
> 40-60 wins (two standard devs)
> 35- 65 (three standard devs) etc
>
> So for 100 trades 60 wins doesn't happen all that often, if the coin
> is a fair coin (a random event).
>
> We are defintely going to take notice of 60/100 wins BUT it is
> not 'out of this world' and we do not have certainty - we only have
> the expectation that it is good - reality can, and does, dash
> expectations on occasion.
>
> Because of this, W/L results are never a sure thing.
>
> To ensure against this take control of the ave%W/ave%L ratio - that
> is something we can control via stops - if the W/L ratio turns out to
> be a 'BlackSwan' our good stops will save us from crashing and
> burning (keep us at low drawdowns).
>
> Comparing to Chi (for 100 trades with a 60% win record):
>
> Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
>
> == ((60-40)-1)^2/(wi+Lo);
> == 19^2/100;
> == 361/100
> == 3.6
> == Not significant according to Chi but significant according to
> brian (always look on the bright side of life!).
>
> I agree that finding 60% winners, in any market or timeframe, is very
> difficult - that is the reality of trading.
>
> This problem is especially prevalent in mid - long term trading - say
> indicators with long lookbacks are used - then the number of signals
> available tends towards becoming a rare event and the trader then can
> only see a small part of the longterm (10000 plus) trades - the
> trader soon runs out of clean data and can't get high enough trade
> counts.
>
> That is why I like shorter term trading (intraday to 2-3 day cycles)
> where I can take advantage of statistical smoothing (I quickly
> approach my theoretical edge i.e. relative to the calendar days).
>
>
> As I said - please use 'my' theories at your own risk, at least until
> after I post on the topic, and the mathematicians in the forum have a
> chance to bash up my hypotheses.
>
>
> brian_z
>
>
>
>
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, "jeffro861" <jeffro861@xxx> wrote:
>>
>> Ok, so the chi-squared tests for independence (real vs. expected)
> so,
>> looking at your algorithm I'm not seeing expected. What exactly
> are
>> you test significance against?
>>
>> --- In amibroker@xxxxxxxxxxxxxxx, Thomas Ludwig <Thomas.Ludwig@>
>> wrote:
>> >
>> > Hello,
>> >
>> > in "Quantitative Trading Systems" on p. 256, Howard describes a z-
>> score
>> > test in order to evaluate the statistical significance of a
> trading
>> > system. While the formula is easy to write in AFL, I don't think
>> that
>> > it can be done as a custom metric since the system to be
> evaluated
>> is
>> > compared with a Random System. Any idea how to sensibly implement
>> it in
>> > Amibroker?
>> >
>> > I'm using another statistical test proposed by the late Arthur
>> Merrill
>> > some years ago in S&C. It's the "chi squared with one degree of
>> > freedom, with the Yates correction". Here's how I implemented it
> in
>> AB:
>> >
>> > //chi squared with one degree of freedom, with the Yates
> correction
>> > wi=st.GetValue("WinnersQty");
>> > Lo=st.GetValue("LosersQty");
>> > Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
>> > bo.AddCustomMetric( "Chi-Squared modif.: >10.83: very
>> > significant(1000:1), >6.64: significant (100:1) , >3.84: probably
>> > significant (20:1), <3.84: significance doubtful", Chi );
>> >
>> > While this metric doesn't tell you anything if your system is
>> > profitable, it tells you if its signals are only pure coincidence
>> > (simply put). It's remarkable that many systems that seem to be
>> > promising according to the usual metrics, are below 3.84, i.e.
>> > significance doubtful. You need either a rather high number of
>> trades
>> > or a very high percentage of winning trades to shift this metric
>> > significantly higher. At least for (medium-term) EOD systems
>> (that's
>> > what I trade) this is not easy to achieve.
>> >
>> > What do you think about this metric? Are there other "better"
>> > statistical metrics? If yes - would you mind sharing the AFL code?
>> >
>> > Best regards,
>> >
>> > Thomas
>> >
>>
>
>
>
>
> Please note that this group is for discussion between users only.
>
> To get support from AmiBroker please send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> For other support material please check also:
> http://www.amibroker.com/support.html
>
> Yahoo! Groups Links
>
>
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|