|
PureBytes Links
Trading Reference Links
|
Thomas,
I have included Jeffs question with your opening query since they fit
so closely.
> looking at your algorithm I'm not seeing expected. What exactly
>are
> you test significance against?
The expected value of an unbiased binomial event is 0.50 (50% win).
Since the (equity) market has a slight upward bias we need to use an
expected value, that includes the market bias, as the base line (or
detrend as per Howards/Aronsons methods).
(Personally I don't detrend as I like to condition myself to see the
markets as they actually are, and mentally factor in the trend, so I
use market bias benchmarking).
> > in "Quantitative Trading Systems" on p. 256, Howard describes a z-
> score
> > test in order to evaluate the statistical significance of a
trading
> > system. While the formula is easy to write in AFL, I don't think
> that
> > it can be done as a custom metric since the system to be
evaluated
> is
> > compared with a Random System. Any idea how to sensibly implement
> it in
> > Amibroker?
At page 91,in his book, (Entries and Exits chapter) Howard gives some
very good (random entry) examples of how we can get an estimation for
the 'standardized binomial expectancy' of any market i.e. you can get
the mean expected wins for the actual market you are testing your
system in and use that in the Z score calculation - I think you would
be better to use random entries with an exit after a set number of
days == the average time your system trades are in the market.
I am still learning AB myself so I am not sure if we can implement
Howards Z equation directly in AB - you will probably have to do it
outside somewhere - I haven't figured out how we can get the SD of a
trade series (from a backtest) in AB - anyway we don't have built in
stats tables (I suppose you could manually plug in the typical Z
scores).
I'm exporting to Excel and doing my evaluations there, but I don't
get that fancy.
> > I'm using another statistical test proposed by the late Arthur
> Merrill
> > some years ago in S&C. It's the "chi squared with one degree of
> > freedom, with the Yates correction". Here's how I implemented it
in
> AB:
> >
> > //chi squared with one degree of freedom, with the Yates
correction
> > wi=st.GetValue("WinnersQty");
> > Lo=st.GetValue("LosersQty");
> > Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
> > bo.AddCustomMetric( "Chi-Squared modif.: >10.83: very
> > significant(1000:1), >6.64: significant (100:1) , >3.84: probably
> > significant (20:1), <3.84: significance doubtful", Chi );
> >
> > What do you think about this metric?
I think it is a very conservative measure.
One of the problems we have, in evaluation, is that 'academic'
statistics filtered into freelance trading via institutional
investing - nothing against academics or institutional traders but
their focus is somewhat different to freelance traders.
Some of the 'academic' stats methods are not quite as good a fit to
trading as we first assume e.g. look at the recent discussion on
Fourier Transform, which comes from Signal Processing (electronics?)
and falls down rather spectacularly in trading because stockmarket
data isn't stationary.
Most of the stats we are using assume stationarity and also assume
that data will be normal/ random i.e. it will have a normal
distribution and that the datapoints are independent of each other.
Neither is absolutely true, so the stats we are using are
approximations (of course the data we are using is only an
approximation anyway) - hence the doubts about Merrill's Chi.
> > While this metric doesn't tell you anything if your system is
> > profitable, it tells you if its signals are only pure coincidence
> > (simply put). It's remarkable that many systems that seem to be
> > promising according to the usual metrics, are below 3.84, i.e.
> > significance doubtful. You need either a rather high number of
> trades
> > or a very high percentage of winning trades to shift this metric
> > significantly higher. At least for (medium-term) EOD systems
> (that's
> > what I trade) this is not easy to achieve.
> >
Yes, it is very hard to find good trading systems.
This is what I have found - many tests that come up with nothing,
especially in the first two years.
>>Are there other "better"
> > statistical metrics? If yes - would you mind sharing the AFL code?
> >
Try Howards Z method, using his random code, to find your expected
win rate for your market and see how that works out.
I have started some original (to me) work, based on binomial
simulation of equity curves and the behaviour of random, 50/50,
trading systems.
It is only at the experimental, concept stage.
I intend posting it to the UKB one day so that the mathematically
trained people in the forum can critique it (it might be a load of
old rubbish for all I know).
Based on that work I am using PowerFactor, with sample error, to
guestimate significance (I can quickly do that in my head).
Note that in PowerFactor the binomial component is considered to be
Gaussian, with independent variables, while the distribution of the
trades (ave%won/ave%lost) is not.
Because of that I only apply the significance test to the W/L
binomial component (I claim that carrying out stats analysis on the
compound system results is biased because of the non-normal nature of
the distribution etc).
For binomial events:
variance == sample error (sort of)
For 100 trades:
sample error = +_10%;
expected random result (benchmark) == 50 wins;
a no win trade will have a range of:
45-50 wins (one standard dev)
40-60 wins (two standard devs)
35- 65 (three standard devs) etc
So for 100 trades 60 wins doesn't happen all that often, if the coin
is a fair coin (a random event).
We are defintely going to take notice of 60/100 wins BUT it is
not 'out of this world' and we do not have certainty - we only have
the expectation that it is good - reality can, and does, dash
expectations on occasion.
Because of this, W/L results are never a sure thing.
To ensure against this take control of the ave%W/ave%L ratio - that
is something we can control via stops - if the W/L ratio turns out to
be a 'BlackSwan' our good stops will save us from crashing and
burning (keep us at low drawdowns).
Comparing to Chi (for 100 trades with a 60% win record):
Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
== ((60-40)-1)^2/(wi+Lo);
== 19^2/100;
== 361/100
== 3.6
== Not significant according to Chi but significant according to
brian (always look on the bright side of life!).
I agree that finding 60% winners, in any market or timeframe, is very
difficult - that is the reality of trading.
This problem is especially prevalent in mid - long term trading - say
indicators with long lookbacks are used - then the number of signals
available tends towards becoming a rare event and the trader then can
only see a small part of the longterm (10000 plus) trades - the
trader soon runs out of clean data and can't get high enough trade
counts.
That is why I like shorter term trading (intraday to 2-3 day cycles)
where I can take advantage of statistical smoothing (I quickly
approach my theoretical edge i.e. relative to the calendar days).
As I said - please use 'my' theories at your own risk, at least until
after I post on the topic, and the mathematicians in the forum have a
chance to bash up my hypotheses.
brian_z
--- In amibroker@xxxxxxxxxxxxxxx, "jeffro861" <jeffro861@xxx> wrote:
>
> Ok, so the chi-squared tests for independence (real vs. expected)
so,
> looking at your algorithm I'm not seeing expected. What exactly
are
> you test significance against?
>
> --- In amibroker@xxxxxxxxxxxxxxx, Thomas Ludwig <Thomas.Ludwig@>
> wrote:
> >
> > Hello,
> >
> > in "Quantitative Trading Systems" on p. 256, Howard describes a z-
> score
> > test in order to evaluate the statistical significance of a
trading
> > system. While the formula is easy to write in AFL, I don't think
> that
> > it can be done as a custom metric since the system to be
evaluated
> is
> > compared with a Random System. Any idea how to sensibly implement
> it in
> > Amibroker?
> >
> > I'm using another statistical test proposed by the late Arthur
> Merrill
> > some years ago in S&C. It's the "chi squared with one degree of
> > freedom, with the Yates correction". Here's how I implemented it
in
> AB:
> >
> > //chi squared with one degree of freedom, with the Yates
correction
> > wi=st.GetValue("WinnersQty");
> > Lo=st.GetValue("LosersQty");
> > Chi = (abs(wi-Lo)-1)^2/(wi+Lo);
> > bo.AddCustomMetric( "Chi-Squared modif.: >10.83: very
> > significant(1000:1), >6.64: significant (100:1) , >3.84: probably
> > significant (20:1), <3.84: significance doubtful", Chi );
> >
> > While this metric doesn't tell you anything if your system is
> > profitable, it tells you if its signals are only pure coincidence
> > (simply put). It's remarkable that many systems that seem to be
> > promising according to the usual metrics, are below 3.84, i.e.
> > significance doubtful. You need either a rather high number of
> trades
> > or a very high percentage of winning trades to shift this metric
> > significantly higher. At least for (medium-term) EOD systems
> (that's
> > what I trade) this is not easy to achieve.
> >
> > What do you think about this metric? Are there other "better"
> > statistical metrics? If yes - would you mind sharing the AFL code?
> >
> > Best regards,
> >
> > Thomas
> >
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|