Re: [amibroker] Re: Aronson Detrending Market {Was Detrending... log}, AmiBroker Email List Archive

2008/3/3, Louis Préfontaine <rockprog80@xxxxxxxxx>:

Hi Mike,

Thank you so much for your reply. I really appreciate your step-by-step advices. This is really amazing to see the work you've done with Aronson's ideas!

I tried to follow your step-by-step guide but unfortunately it seems there may be a mistake in the following code:

SetTradeDelays(1, 1, 1, 1); // All trades on next Open afterEOD Signal
SetBacktestMode(backtestRegular); // One symbol, no redundant signals
SetOption("InitialEquity", 100000);
SetOption("AccountMargin", 100);

For some reason I get an error Line 2, Col.: 16 Error 30. Syntax error.

Did I do something wrong? Is it possible to do the same thing in another way?

Anyway I now fully understand what we're trying to do here. After I got the ALR for each day, all that will be needed will be to subtract it from day1 to day 2 log. Then we will need to set rules with +1 and -1 value... Well, I am not 100% sure I fully understand, but it's getting close. If I can only get past this error...

Thanks a lot!

Louis

2008/3/2, Mike <sfclimbers@xxxxxxxxx>:

I've provided an example of how to use it below. I'm still working on
this myself, so take all of this with a measure of caution. I have
not yet gone through all results to verify that there aren't any
bugs...

If I understand Aronson correctly, after you have have detrended the
market for the desired period, you then apply your trading strategy
to generate new log daily returns by multiplying your system state
(+1 for long, -1 for short) by the detrended log daily returns. You
then take the mean log daily return generated by your strategy and
compare it to the mean log daily return of the detrended daily
returns (i.e. zero).

The purpose is to remove the "conjoint effect of position bias and
trend" from your strategy results.

If the two means are found to be different (i.e. your strategy
generates non zero mean) with statistical significance (e.g. using
student's t-test), then you may conclude* that the returns generated
by the strategy are not purely by chance.

I use a "*" there because Aronson continues to say that a simple t-
test is NOT sufficient if you have used data mining to produce your
strategy. In that case, your results will still include a data mining
bias that must be removed by applying the Bootstrap method with
White's reality check, or else the Monte Carlo analysis.

Now for what you're really looking for, a concrete example :)

Still in Chapter 1 of the book, Aronson refers to an example of
taking signals based on the Dow Jones Transportation Average (DTX--X
by my data provider), and applying those signals to trade the SP-500.
The strategy has a long bias and is defined as follows:

Draw a moving average of DTX--X
Draw an upper band 3% above the moving average
Draw a lower band 3% below the moving average
Short the SP-500 when the DTX--X crosses below the lower band.
Long the SP-500 at all other times.

I believe that I have captured these rules and added backtesting
support in the script below. After running the backtest, a new symbol
will be added to your system holding the returns in the Close field
(e.g. ~SP-500Returns).

Aronson does not say what the moving average should be. It is
actually a bit of a challenge to find a period where the above is
profitable! However, after optimizing for the moving average, I found
that the period 4/27/1998 - 12/31/2000 will show you fully invested
for the entire range with a 13.69% CAR.

Applying my Detrend.afl script (posted in my last note) against the
SP-500 over those dates will give a mean log daily return of 2.40265E-
10 (i.e. zero).

You can compute the mean log daily return by opening the Quote Editor
(e.g. select ~SP-500 in symbols tree, then use Symbol | Quote
Editor... menu item) and copying all the data into Excel, then
summing all the Close values for the given date range and dividing by
the number of quotes in that range (678 in this example).

Running the script below against the SP-500 for the same dates will
will give a mean log daily return of 0.000428057

Again, copy ~SP-500Returns from Quote Editor into Excel and compute
the mean from the Close data.

Peforming a Student's T-Test over the two sets of log daily returns,
assuming a shared variance and hypothesizing a shared mean of zero,
returns a one tailed p-value of 0.272346792. This is well above 0.05
(the p-value required to reject the hypothesis with 95% confidence)
and so we cannot reject the hypothesis.

In other words, despite a 13.69% CAR over the period tested, the
strategy is no better than chance alone and thus should not be traded.

If the p-value had been .05 or less, we still could not conclude
(according to Aronson) that the strategy was any good because I used
Optimization (MA period ranging from 5 to 200 increments of 5) to
find the best rule (data mining bias) and because I used the 3% band
width suggested in the book (snooping bias - have no idea how much
optimization was applied to reach that number). Applying Monte Carlo
analysis would remove the data mining bias (my next effort). There's
no getting around the snooping bias.

1. Copy paste the script below to a file on your machine (say
c:\Program Files\AmiBroker\Formulas\Custom\Results.afl). Make sure
that you correct any formatting that gets messed up from this post,
such that AmiBroker likes everything. Use the Tools | Verify Syntax
menu from the code editor.

2. Open a chart on the symbol that you want to detrend. For example;
Aronson used the SP-500 for all his tests.

3. Open the Automatic Analysis Window

4. Click the "Pick" button and select the script that you just saved
(i.e. Results.afl).

5. Select "current symbol" for the Apply To.

6. Select "from" for the Range, and enter a from date and a to date
(e.g. from 4/27/1998 to 12/31/2000).

7. Click on Backtest

A new symbol will be added to your system having the same name as the
original, but prefixed with a "~" and appended with "Return", for
example "~SP-500Return". This symbol will appear in Market 253 and
contain detrended performance results for the range selected and
zeros for all other dates.

For each bar in the detrended symbol, the information will be
arranged as follows:

Close: The detrended log daily return of the strategy (i.e. Pos0 x log
(Open2/Open1) - ALR).

Note: As per Aronson, Pos0 refers to the strategy signal (+1 for
long, -1 for short), Open2 refers to the Open two days from now,
Open1 refers to the Open one day from now, ALR refers to the average
log daily return of the market (not your strategy results) over the
period being detrended.

As always, comments, corrections and enhancements are welcomed.

Again, I'm a software developer, not a statistics guy, and this is my
interpretation of the book. Do your own research before accepting
taking this as acurate.

If anyone else has done any work in this area, I would very much like
to hear if their approach agrees with my interpretation.

Thanks,

Mike

SetTradeDelays(1, 1, 1, 1); // All trades on next Open after
EOD signal
SetBacktestMode(backtestRegular); // One symbol, no redundant signals
SetOption("InitialEquity", 100000);
SetOption("AccountMargin", 100);

Plot(Close, "Close", colorBlue, styleLine);

SetForeign("DTX--X");
center = MA(Close, Optimize("MA", 40, 5, 200, 5));
upper = 1.03 * center;
lower = .97 * center;

Buy = Cross(Close, lower);
Sell = Cross(lower, Close);
Short = Sell;
Cover = Buy;

Plot(Close, "DTX--X Close", colorYellow, styleLine);
Plot(upper, "DTX--X Upper", colorGreen, styleDashed);
Plot(center, "DTX--X MA", colorPink, styleLine);
Plot(lower, "DTX--X Lower", colorRed, styleDashed);

RestorePriceArrays();

BuyPrice = SellPrice = ShortPrice = CoverPrice = Open;

PlotShapes(shapeUpArrow * Buy, colorGreen, 0, Close, -30);
PlotShapes(shapeDownArrow * Sell, colorRed, 0 , Close, -30);
PlotShapes(shapeHollowDownArrow * Short, colorRed, 0 , Close, -40) ;
PlotShapes(shapeHollowUpArrow * Cover, colorGreen, 0, Close, -40);

isLong = Flip(Buy, Sell);
isShort = Flip(Short, Cover);
market = Foreign("~" + Name(), "C", false);

AddToComposite(IIF(isLong, 1 * market, 0), "~" + Name()
+ "Return", "C", atcFlagDefaults | atcFlagEnableInBackTest);
AddToComposite(IIF(isShort, -1 * market, 0), "~" + Name()
+ "Return", "C", atcFlagDefaults | atcFlagEnableInBackTest);

--- In amibroker@xxxxxxxxxxxxxxx, "Louis Préfontaine"
<rockprog80@xxx> wrote:
>
> Hi again,
>
> I've searched thru the manual and I think that += is simply a
shortcut. I
> wrote total = total+ raw[i]; instead and there is no more error!
>
> I did exactly as you said and got the sign with those huge bars all
topping
> at 0.267615. I think I understand what you say, but what is OI?
>
> The thing is: I am not sure how to use this. I think what I need
is to make
> an average of all this data and then subtract this average from
each day log
> of the actual stock to detrend. Am I correct? Would you be kind
enough to
> give me some tips about how to use this new information to actually
backtest
> one of my rules and see how it performs when it is detrended?
>
> Thanks a lot!
>
> Louis
>
> 2008/3/2, Louis Préfontaine <rockprog80@xxx>:
> >
> > Hi Mike,
> >
> > Thank you so much for your reply!
> >
> > This is EXACTLY what I am looking for, from the Aronson's book!
> >
> > I can't wait to make it work... Right now there is a small
problem with
> > the formula... I get an error message for this parti total += raw
[i]; Ln12:
> > col:8:Error 30. Syntax error.
> >
> > I tried to change the += for = "" == and it works... Is it
possible that
> > the AB version that I have doesn't recognize the +=? Or maybe
there is an
> > error with the +=? Is it possible to get to the same result in
any other
> > way?
> >
> > Thanks a lot!
> >
> > Louis
> >
> > 2008/3/2, Mike <sfclimbers@xxx>:

> > >
> > > Hi,
> > >
> > > Based on your formula, I assume that you are referring to
Chapter 1
> > > of David Aronson's book: Evidence Based Technical Analysis.
> > >
> > > That being the case, I am providing a script below.
> > >
> > > However, I believe that the formula that you originally posted
is not
> > > correct. Aronson's formula calls for multiplying your boolean
> > > strategy signal (i.e. +1 for long vs. -1 for short) by the
detrended
> > > daily returns, *not* the Close by the returns!
> > >
> > > Also, the book does not go into detail for tri-state strategies
(i.e.
> > > long/neutral/short) nor for long/neutral or short/neutral
strategies.
> > > I'm assuming that plugging in a signal value of 0 would be
acceptable
> > > for a neutral position, but haven't researched that yet. So,
just be
> > > careful how you use the data once you've detrended it.
> > >
> > > Anyway, here is a script that I believe will detrend the market
> > > returns as per the book. Currently, the script is intended for
> > > detrending a single symbol. I have not yet got around to making
it
> > > work against a watchlist of symbols (coming soon).
> > >
> > > 1. Copy paste the script below to a file on your machine (say
> > > c:\Program Files\AmiBroker\Formulas\Custom\Detrend.afl). Make
sure
> > > that you correct any formatting that gets messed up from this
post,
> > > such that AmiBroker likes everything. Use the Tools | Verify
Syntax
> > > menu from the code editor.
> > >
> > > 2. Open a chart on the symbol that you want to detrend. For
example;
> > > Aronson used the SP-500 for all his tests.
> > >
> > > 3. Open the Automatic Analysis Window
> > >
> > > 4. Click the "Pick" button and select the script that you just
saved
> > > (i.e. Detrend.afl).
> > >
> > > 5. Select "current symbol" for the Apply To.
> > >
> > > 6. Select "from" for the Range, and enter a from date and a to
date
> > > (e.g. from 1/1/2007 to 12/31/2007).
> > >
> > > 7. Click on Backtest
> > >
> > > A new symbol will be added to your system having the same name
as the
> > > original, but prefixed with a "~", for example "~SP-500". This
symbol
> > > will appear in Market 253 and contain detrended market
information
> > > for the range selected (e.g. all of 2007 as above) and zeros
for all
> > > other dates.
> > >
> > > For each bar in the detrended symbol, the information will be
> > > arranged as follows:
> > >
> > > Open: The unadjusted log daily return (i.e. log(Open2/Open1)).
> > >
> > > High: The total sum of all unadjusted log daily returns.
> > >
> > > Low: The average of all unadjusted log daily returns (i.e. ALR).
> > >
> > > Close: The detrended log daily return (i.e. log(Open2/Open1) -
ALR).
> > >
> > > OI: The number of bars over which the data has been detrended.
> > >
> > > Note: As per Aronson, Open2 refers to the Open two days from
now,
> > > Open1 refers to the Open one day from now, ALR refers to the
average
> > > log return over the period being detrended.
> > >
> > > Note: I have used the natural logarithm in my code (i.e. ln), as
> > > opposed to the base 10 logarithm (i.e. log10). I don't know if
that
> > > makes a difference.
> > >
> > > Note: To find your detrended strategy results, you still must
write
> > > your own code to calculate which of the detrended daily returns
your
> > > strategy would pick up, and which sign to use (+/-) when
multiplying
> > > by the detrended return for that day.
> > >
> > > Note: I ran this script against SP-500 for the entire year of
2007.
> > > With my data source, the average detrended log daily return
(i.e. all
> > > the Close values of ~SP-500 divided by 251 actual trading days)
ended
> > > up being -6.00797E-10 which is effectively zero. So, I'm
assuming
> > > that it works.
> > >
> > > Corrections and enhancements welcomed :)
> > >
> > > Mike
> > >
> > > procedure Detrend(compositeName) {
> > > local range; range = Status("barinrange");
> > > local raw; raw = log(Ref(Open, 2)/Ref(Open, 1));
> > > local total; total = 0;
> > > local count; count = 0;
> > > local offset; offset = 0;
> > >
> > > for (i = 0; i < BarCount; i++) {
> > > if (range[i]) {
> > > if (NOT IsNull(raw[i])) {
> > > count++;
> > > total += raw[i];
> > > }
> > > }
> > > }
> > >
> > > if (count > 0) {
> > > AddToComposite(IIF(range, raw, Null), "~" + compositeName, "O",
> > > atcFlagDefaults | atcFlagEnableInBackTest);
> > >
> > > offset = total/count;
> > > raw = IIF(IsNull(raw), offset, raw);
> > >
> > > AddToComposite(IIF(range, raw - offset, Null), "~" +
> > > compositeName, "C", atcFlagDefaults | atcFlagEnableInBackTest);
> > > AddToComposite(IIF(range, total, Null), "~" +
compositeName, "H",
> > > atcFlagDefaults | atcFlagEnableInBackTest);
> > > AddToComposite(IIF(range, offset, Null), "~" +
> > > compositeName, "L", atcFlagDefaults | atcFlagEnableInBackTest);
> > > AddToComposite(IIF(range, count, Null), "~" +
compositeName, "I",
> > > atcFlagDefaults | atcFlagEnableInBackTest);
> > > } else {
> > > AddToComposite(Null, "~" + compositeName, "X", atcFlagDefaults |
> > > atcFlagEnableInBackTest);
> > > }
> > > }
> > >
> > > Buy = Sell = Short = Cover = 0;
> > > Detrend(Name());
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%40yahoogroups.com>,
> > > "louisprefontaine" <rockprog80@>
> > > wrote:
> > > >
> > > > Anybody can help?
> > > >
> > > > Thanks,
> > > >
> > > > Louis
> > > >
> > > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com>, "Louis
> > > Préfontaine" <rockprog80@>
> > > > wrote:
> > > > >
> > > > > I am trying to build a formula to "detrend" the market.
> > > > >
> > > > > What I want to set is something like this
> > > > >
> > > > > Close of day 0 * ( log (open day2/open day 1) - average log
> > > > return of
> > > > > every day of the data available.
> > > > >
> > > > > Anybody can do that?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Louis
> > > > >
> > > >
> > >
> > >
> > >
> >
> >
>