Re: -->To Mike --> Re: [amibroker] Re: Aronson Detrending Market {Was Detrending... log}, AmiBroker Email List Archive

2008/3/25, Mike <sfclimbers@xxxxxxxxx>:

Louis,

It seems clear that we are not communicating effectively. So, I'll
just wish you luck in your studies, except to suggest that you might
be better off not using detrended data at all.

Instead, you might be better off trying to follow Bandy's suggestion
(Quantitative Trading Systems) of using a random entry that takes
positions at an equivalent rate as your strategy, then holding those
positions for a fixed period equal to the average hold time of your
strategy.

If you can beat the random entry, you may be on to something. Don't
be surprised at how difficult that turns out to be!

I believe that the approach is described in his chapter on entries
and exits, including sample code. Though, you will need to rework the
random entry generation logic if your strategy differs significantly
from the example of the book.

Mike

--- In amibroker@xxxxxxxxxxxxxxx, "Louis Préfontaine"
<rockprog80@xxx> wrote:
>
> Hi Mike,
>
> Thank you very much for your time. I thought I was ready to
understand
> this, but I have to face the fact that I simply don't understand
what you
> say.
>
> I ran the second script, but I see no buying/selling signal
whatsoever. I
> only see white and black bars; the white bars goes from 0 to
positive number
> like 0.0021478 (only an example) and same thing with black but
with a
> negative number. I understand that the zero is important because
it
> represents the detrended data, but I really don't understand how to
make any
> use of this.
>
> What I am really trying to do (and maybe we weren't trying to do
the same
> thing) is having a script that would actually detrend a market so
as I could
> test a particular rule (e.g. 3/20 MA crossover) and instead of
having the
> real profit return (e.g. 5%) I would get the return minus the
detrended bias
> (to consider the fact that the market was probably trending for the
last
> years).
>
> I don't know if what I am looking for is possible, but in Aronson's
book it
> seems very important. All his analysis of a S&P500 rule is based
on a
> detrended market; e.g. the rule has a 4% profit on detrended data.
>
> I tried to understand this part:
>
> If your strategy, as added to the second script and run against your
> Yahoo real market data, has you going Short on the first day then
> Long on each of the next 2 days, the resulting new detrended series
> becomes:
>
> -.01 * -1 (i.e. short) = +.01
> +.03 * +1 (i.e. long) = +.03
> -.02 * +1 (i.e. long) = -.02
>
> This represents what your strategy would capture from "the detrended
> market" giving a daily average of (+.01 + .03 - .02)/3 = .0067,
which
> is greater than zero.
>
> but, well, I don't understand what you mean because you seem to say
there is
> a relation between the buying/selling signals and how the market
would be
> detrended, where in my understanding the detrending process should
be
> objective and not based on signals but simply on Average log
difference
> between each bars.
>
> Wouldn't it be possible to simply have a script that would detrend
> automatically a market so a rule could be backtested on the
detrended data?
>
> Thanks for your help,
>
> Louis
>
>
>
> 2008/3/25, Mike <sfclimbers@xxx>:

> >
> > Louis,
> >
> > Yes, you are confused. Reread the end of the first chapter where
> > Aronson explains that your strategy must still be run against the
> > real market data (i.e. your Yahoo data, NOT the detrended data).
You
> > do not optimize a script over detrended data, you *validate* a
script
> > against detrended data.
> >
> > All your optimization work should still be done exactly the way
you
> > would normally do it. Once you have found a strategy that you are
> > happy with, only the resulting strategy *signal* is multiplied
> > against the detrended market data (i.e. either a -1 for short or
a +1
> > for long).
> >
> > If you study my second script example, you will see that all it
does
> > is multiply the detrended data of the first script by a +1 or a -1
> > based on the signal of a strategy which was run against real
market
> > data for the same period.
> >
> > Look at the chart of the sample second script I gave and see that
the
> > signal is produced when the close crosses the offset of a moving
> > average. That signal was produced on real market data!
> >
> > Read the code and understand what every single line is doing. Put
> > aside what you think it *should* be doing, and understand what it
> > actually *is* doing. Only then will you be able to make any use
of it.
> >
> > In summary:
> >
> > If the detrended data from the first script gives the following
log
> > percent daily change series:
> >
> > -.01
> > +.03
> > -.02
> >
> > This represents "the detrended market" giving a daily average of
(-
> > .01 + .03 - .02)/3 = 0.
> >
> > If your strategy, as added to the second script and run against
your
> > Yahoo real market data, has you going Short on the first day then
> > Long on each of the next 2 days, the resulting new detrended
series
> > becomes:
> >
> > -.01 * -1 (i.e. short) = +.01
> > +.03 * +1 (i.e. long) = +.03
> > -.02 * +1 (i.e. long) = -.02
> >
> > This represents what your strategy would capture from "the
detrended
> > market" giving a daily average of (+.01 + .03 - .02)/3 = .0067,
which
> > is greater than zero.
> >
> > The question now becomes; Is a value of .0067 significant? When
you
> > have enough daily observations, a t-test will give you the answer.
> >
> > Mike
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com>, "Louis
> > Préfontaine"
> > <rockprog80@> wrote:
> > >
> > > Hi Mike,
> > >
> > > I am not sure to understand if your two scripts can actually see
> > the results
> > > of the rule on a detrended market. Correct me if I am wrong, but
> > the first
> > > script allow to see the avg. log (ALR), and the second shows the
> > buy/sell
> > > signals on the "detrended" new quotes, but it still doesn't
help me
> > to test
> > > the rule on the detrended market.
> > >
> > > Maybe I am confused, but I still don't see how I can backtest my
> > rule on the
> > > "detrended" quotes (which can't really be applied to a rule
which
> > is looking
> > > at "High" values when the "High" value in this case is a
> > constant). Right
> > > now, even after reading you again and again, I am still stuck
with
> > one on
> > > side the "detrended" quotes in which only the close matters and
the
> > > buying/selling signals without any way to optimize and backtest
the
> > actual
> > > rules.
> > >
> > > Can you help me with that?
> > >
> > > Thanks,
> > >
> > > Louis
> > >
> > >
> > >
> > >
> > >
> > > 2008/3/25, Mike <sfclimbers@>:
> >
> > > >
> > > > Louis,
> > > >
> > > > Forget the book for a minute. Just try to understand the
> > objective.
> > > > Until you understand what is trying to be accomplished, you
will
> > not
> > > > get anywhere.
> > > >
> > > > Given two sets of daily changes, can you say with confidence
that
> > the
> > > > daily changes did not come from the same strategy?
> > > >
> > > > If the average daily change from the first set of numbers is
zero,
> > > > and the average daily change from the second set of numbers
is NOT
> > > > zero. Then, is the strategy that generated the second set of
daily
> > > > changes really better than the first? Or, was it just a lucky
> > sample
> > > > from a strategy whose true daily change really is equal to the
> > first
> > > > (i.e. really is zero after all).
> > > >
> > > > The hypothesis is that the two sets of daily changes came from
> > > > equivalent strategies (i.e. both zero predictive ability).
> > > >
> > > > The test is to calculate the probability of drawing a sample
> > (whose
> > > > average was the non zero average found in the second set of
daily
> > > > changes), assuming that the true average for the second
strategy
> > is
> > > > the same as the true average for the first strategy (i.e both
true
> > > > averages equal to zero).
> > > >
> > > > If the probability is less than your cutoff (e.g. 5%), then
reject
> > > > the hypothesis and conclude that the strategy that returned
the
> > non
> > > > zero average daily change must be better than the one that
> > generated
> > > > the zero average daily change (i.e. the strategy generating
non
> > zero
> > > > daily change has predictive ability).
> > > >
> > > > You may find the following link helpful:
> > > > http://www.socialresearchmethods.net/kb/stat_t.php
> > > >
> > > > The first script generates the first set of daily changes (log
> > > > percent change) for the period being studied. These changes
were
> > not
> > > > generated by any strategy. They are just a detrended
> > representation
> > > > of what the market actually did.
> > > >
> > > > The second script generates the second set of daily changes
based
> > on
> > > > the buy sell signal of your strategy as applied against the
actual
> > > > detrended market changes. They are the results of what your
> > strategy
> > > > would capture from the detrended market.
> > > >
> > > > Running the second script will not give you a performance
number
> > per
> > > > se. It will merely allow you to calculate the second detrended
> > > > average daily return (i.e. log percent change, presumably non
> > zero)
> > > > that must be compared to that of the first script (zero).
> > > >
> > > > If you want to see the actual performance of your strategy
(e.g.
> > > > compounded average return), you still need to look at the
> > AmiBroker
> > > > backtest results.
> > > >
> > > > Also, if you go back and re-read my earlier comments. After
> > running
> > > > the first script, the High holds the total of all daily
changes
> > for
> > > > the period, the Lo holds the number of bars in the period.
> > Dividing
> > > > the first by the second gives you the average daily change
for the
> > > > period, which is then subtracted from the *individual* daily
> > changes,
> > > > giving the *detrended* daily changes in the _Close_ such that
if
> > you
> > > > were to take the average of the Close you would get zero. The
only
> > > > values you are interested in are the Close values. The rest
are
> > just
> > > > intermediary values used to calculate the Close.
> > > >
> > > > The second script applies the strategy signal against the
Close of
> > > > the first script to generate the second set of daily changes
> > (again,
> > > > in the Close), the average of which you want to compare to the
> > > > results of the first.
> > > >
> > > > I haven't run against the full 25 year period to compare
against
> > the
> > > > book for any given strategy. Depending on data providers, you
may
> > get
> > > > very different numbers. Particularly if Aronson used the
actual
> > > > contents of the SP-500 for each year, as opposed to only the
> > > > survivors that currently appear in today's index.
> > > >
> > > > Mike
> > > >
> > > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%

40yahoogroups.com><amibroker%

> > 40yahoogroups.com>, "Louis
> > > > Préfontaine"
> > > > <rockprog80@> wrote:
> > > > >
> > > > > Hi Mike,
> > > > >
> > > > > I took a break from the detrending procedure because I
thought I
> > > > had a lot
> > > > > to learn before been able to apply it. Now I think I am
ready
> > for
> > > > some more
> > > > > testing. I know you took a lot of time to explain how you
were
> > > > able to do
> > > > > build the detrending process, and so I will try not to make
> > > > yourself repeat
> > > > > yourself.
> > > > >
> > > > > I did exactly what Aronson did to try the method. I used the
> > > > S&P500 from
> > > > > November 1980 (November 1, 1980) to Junes 2005 (June 30,
2005)
> > and
> > > > used the
> > > > > 91 days channel breakout:
> > > > >
> > > > > Buy=C>Ref (HHV (C,91),-1);
> > > > > Sell= C< Ref (LLV (C,91),-1);
> > > > > Short=Sell;
> > > > > Cover=Buy;
> > > > >
> > > > > I used Yahoo data and used your detrend script:
> > > > >
> > > > > procedure Detrend(compositeName) {
> > > > > local range; range = Status("barinrange");
> > > > > local raw; raw = log(Ref(Open, 2)/Ref(Open, 1));
> > > > > local total; total = 0;
> > > > > local count; count = 0;
> > > > > local offset; offset = 0;
> > > > >
> > > > > for (i = 0; i < BarCount; i++) {
> > > > > if (range[i]) {
> > > > > if (NOT IsNull(raw[i])) {
> > > > > count++;
> > > > > total += raw[i];
> > > > > }
> > > > > }
> > > > > }
> > > > >
> > > > > if (count > 0) {
> > > > > AddToComposite(IIF(range, raw, Null), "~" +
compositeName, "O",
> > > > > atcFlagDefaults | atcFlagEnableInBackTest);
> > > > >
> > > > > offset = total/count;
> > > > > raw = IIF(IsNull(raw), offset, raw);
> > > > >
> > > > > AddToComposite(IIF(range, raw - offset, Null), "~" +
> > > > > compositeName, "C", atcFlagDefaults |
atcFlagEnableInBackTest);
> > > > > AddToComposite(IIF(range, total, Null), "~" +
> > compositeName, "H",
> > > > > atcFlagDefaults | atcFlagEnableInBackTest);
> > > > > AddToComposite(IIF(range, offset, Null), "~" +
> > > > > compositeName, "L", atcFlagDefaults |
atcFlagEnableInBackTest);
> > > > > AddToComposite(IIF(range, count, Null), "~" +
> > compositeName, "I",
> > > > > atcFlagDefaults | atcFlagEnableInBackTest);
> > > > > } else {
> > > > > AddToComposite(Null, "~" + compositeName, "X",
atcFlagDefaults |
> > > > > atcFlagEnableInBackTest);
> > > > > }
> > > > > }
> > > > >
> > > > > Buy = Sell = Short = Cover = 0;
> > > > > Detrend(Name());
> > > > >
> > > > > The values I got (in the new file created by your script)
are
> > the
> > > > following:
> > > > >
> > > > >
> > > > > Hi: 2.21199
> > > > > Lo: 0.000355397
> > > > >
> > > > > And that's why I am stuck.
> > > > >
> > > > > I tried to apply the 91-days channel breakout to the new
file,
> > but
> > > > after
> > > > > reflexion it makes no sense because the Hi value in that
file
> > is a
> > > > constant
> > > > > while in "real life" it isn't. So I am kind of stucked
there,
> > still
> > > > > wondering how to apply the rule to the "detrended" data.
> > > > >
> > > > > If you could help me with that, I would appreciate a lot!
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Louis
> > > > >
> > > > > p.s. For anyone actually trying to do this yourself without
> > having
> > > > read
> > > > > Aronson, Aronson says the result is +-4% annualized on
detrended
> > > > data.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 2008/3/3, Mike <sfclimbers@>:
> > > > > >
> > > > > > Louis,
> > > > > >
> > > > > > The first script (Detrend.afl) will detrend the market.
> > Nothing
> > > > more
> > > > > > to do. If you want to chart the results, open the symbol
in a
> > > > chart
> > > > > > and plot the Close using line format (i.e. not bar or
> > > > candlesticks).
> > > > > > You will see what appears like an oscillator centered on
zero,
> > > > since
> > > > > > all the values average out. Similarly, if you plotted a
> > > > histogram, of
> > > > > > the Close values, you would get a normal distribution
> > centered at
> > > > > > zero.
> > > > > >
> > > > > > You must code your own strategy into the second script,
> > replacing
> > > > the
> > > > > > example buy/sell/short/cover with your own logic.
> > > > > >
> > > > > > If you are not using a separate symbol to base your
signals
> > on,
> > > > you
> > > > > > can remove the SetForeign/RestorePrice arrays. The parts
that
> > are
> > > > > > most important are the setting of trade delays (since the
> > > > detrended
> > > > > > market data was produced with that assumption) and the
> > generating
> > > > of
> > > > > > a new composite.
> > > > > >
> > > > > > SetTradeDelays(1, 1, 1, 1);
> > > > > >
> > > > > > ...
> > > > > >
> > > > > > isLong = Flip(Buy, Sell);
> > > > > > isShort = Flip(Short, Cover);
> > > > > > market = Foreign("~" + Name(), "C", false);
> > > > > >
> > > > > > AddToComposite(IIF(isLong, 1 * market, 0), "~" + Name()
> > > > > > + "Return", "C", atcFlagDefaults |
atcFlagEnableInBackTest);
> > > > > > AddToComposite(IIF(isShort, -1 * market, 0), "~" + Name()
> > > > > > + "Return", "C", atcFlagDefaults |
atcFlagEnableInBackTest);
> > > > > >
> > > > > > Once your buy/sell/cover/short logic is in place, there is
> > nothing
> > > > > > more to do, the resulting composite (e.g. ~SP-500Return)
will
> > > > already
> > > > > > have every thing you need. If you were to plot a
histogram of
> > the
> > > > > > Close values, you want to see a normal distribution
centered
> > on a
> > > > > > value greater than zero, indicating better than chance
> > results.
> > > > > >
> > > > > > Since the second script is *your* trading logic, yes, when
> > > > running it
> > > > > > the buy/sell signals and all AmiBroker performance data
are
> > > > relevant.
> > > > > > If the system does not perform well based on your personal
> > > > criteria,
> > > > > > then you really don't care whether or not it is
statistically
> > > > > > significant, because you wouldn't want to trade it anyway!
> > > > > >
> > > > > > The only thing that you need to do with the two
composites is
> > to
> > > > > > compare the results.
> > > > > >
> > > > > > Take the mean of the detrended market (e.g. use excel to
add
> > all
> > > > > > Close values found in ~SP-500 and divide by the number of
> > values)
> > > > and
> > > > > > compare it to the mean of your strategy results (again,
add
> > all
> > > > Close
> > > > > > values of ~SP-500Results and divide by the number of
values).
> > > > > >
> > > > > > If a t-test shows them to be statistically significant,
you
> > > > *might*
> > > > > > be on to something. According to Aronson (by my
> > interpretation)
> > > > you
> > > > > > would then have to correct for data mining bias.
> > > > > >
> > > > > > Note that this is just one approach. Others prefer the
much
> > > > simpler
> > > > > > walk forward analysis and visually evaluating the equity
> > curve.
> > > > Walk
> > > > > > forward and equity curve evaluation will rule out most of
your
> > > > ideas
> > > > > > long before you ever need to start doing the statistical
> > > > validation.
> > > > > >
> > > > > > Delaying the statistics until after you've got a nice walk
> > forward
> > > > > > equity curve will get you on your way much quicker.
Though, if
> > > > > > possible it would still be preferable to do the
statistics at
> > that
> > > > > > time.
> > > > > >
> > > > > > Walk forward is built in to the most recent release of
> > AmiBroker,
> > > > and
> > > > > > also available in the free edition of Intelligent
Optimizer
> > (see
> > > > IO
> > > > > > under Files section of this forum). My understanding is
that
> > > > > > AmiBroker and IO treat the handling of trades accross
> > boundries
> > > > > > differently, so don't be surprised if you get slightly
> > different
> > > > > > results between them.
> > > > > >
> > > > > > Again, I'm no statistician, I wrote the code quickly in
the
> > late
> > > > > > hours of the night, and I may not have understood
correctly
> > the
> > > > > > process described in the book. So, use all of this as a
> > starting
> > > > > > point, but don't accept any of it as Gospel.
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com><amibroker%
> > 40yahoogroups.com><amibroker%
> > > > 40yahoogroups.com>, "Louis
> > > > > > Préfontaine"
> > > > > > <rockprog80@> wrote:
> > > > > > >
> > > > > > > Hi again Mike,
> > > > > > >
> > > > > > > Ok I tried the formula without the "setbacktestmode"
and it
> > > > seems
> > > > > > to work.
> > > > > > > I did use the first formula to get the log results of
spx.xo
> > > > (S&P
> > > > > > 500) from
> > > > > > > January 9 2007 to December 31, 2007. Then I added all
the
> > > > results
> > > > > > and
> > > > > > > divided by the number of days. -->But what do I do with
this
> > > > > > number then?
> > > > > > > (I know it's the number I want to substract from each
day
> > > > result,
> > > > > > but od I
> > > > > > > have to do it manually?)
> > > > > > >
> > > > > > > Then I applied the second formula to the same spx.xo
for the
> > > > same
> > > > > > days.
> > > > > > > When clicking on "backtesting" buying/selling orders
were
> > > > issued.
> > > > > > Are those
> > > > > > > relevant, or do I simply need to look at what's in the
new
> > > > > > ~spx.xoReturn?
> > > > > > > (I changed DTX--X for spx.xo everywhere in the
formula). And
> > > > when
> > > > > > I open
> > > > > > > this ~spx.xoReturn I get the detrended results but how
to
> > use
> > > > them
> > > > > > with the
> > > > > > > actual stock? I mean: I get all those 0. 000x numbers;
how
> > can I
> > > > > > actually
> > > > > > > use my trading system to test my positive/negative bias
with
> > > > this?
> > > > > > >
> > > > > > > Thank you a lot for your help. Your formulas really
rock.
> > > > Thanks!
> > > > > > >
> > > > > > > Louis
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 2008/3/2, Mike <sfclimbers@>:
> > > > > > > >
> > > > > > > > I've provided an example of how to use it below. I'm
still
> > > > > > working on
> > > > > > > > this myself, so take all of this with a measure of
> > caution. I
> > > > have
> > > > > > > > not yet gone through all results to verify that there
> > aren't
> > > > any
> > > > > > > > bugs...
> > > > > > > >
> > > > > > > > If I understand Aronson correctly, after you have have
> > > > detrended
> > > > > > the
> > > > > > > > market for the desired period, you then apply your
trading
> > > > > > strategy
> > > > > > > > to generate new log daily returns by multiplying your
> > system
> > > > state
> > > > > > > > (+1 for long, -1 for short) by the detrended log daily
> > > > returns.
> > > > > > You
> > > > > > > > then take the mean log daily return generated by your
> > > > strategy and
> > > > > > > > compare it to the mean log daily return of the
detrended
> > daily
> > > > > > > > returns (i.e. zero).
> > > > > > > >
> > > > > > > > The purpose is to remove the "conjoint effect of
position
> > > > bias and
> > > > > > > > trend" from your strategy results.
> > > > > > > >
> > > > > > > > If the two means are found to be different (i.e. your
> > strategy
> > > > > > > > generates non zero mean) with statistical significance
> > (e.g.
> > > > using
> > > > > > > > student's t-test), then you may conclude* that the
returns
> > > > > > generated
> > > > > > > > by the strategy are not purely by chance.
> > > > > > > >
> > > > > > > > I use a "*" there because Aronson continues to say
that a
> > > > simple
> > > > > > t-
> > > > > > > > test is NOT sufficient if you have used data mining to
> > produce
> > > > > > your
> > > > > > > > strategy. In that case, your results will still
include a
> > data
> > > > > > mining
> > > > > > > > bias that must be removed by applying the Bootstrap
method
> > > > with
> > > > > > > > White's reality check, or else the Monte Carlo
analysis.
> > > > > > > >
> > > > > > > > Now for what you're really looking for, a concrete
> > example :)
> > > > > > > >
> > > > > > > > Still in Chapter 1 of the book, Aronson refers to an
> > example
> > > > of
> > > > > > > > taking signals based on the Dow Jones Transportation
> > Average
> > > > (DTX-
> > > > > > -X
> > > > > > > > by my data provider), and applying those signals to
trade
> > the
> > > > SP-
> > > > > > 500.
> > > > > > > > The strategy has a long bias and is defined as
follows:
> > > > > > > >
> > > > > > > > Draw a moving average of DTX--X
> > > > > > > > Draw an upper band 3% above the moving average
> > > > > > > > Draw a lower band 3% below the moving average
> > > > > > > > Short the SP-500 when the DTX--X crosses below the
lower
> > band.
> > > > > > > > Long the SP-500 at all other times.
> > > > > > > >
> > > > > > > > I believe that I have captured these rules and added
> > > > backtesting
> > > > > > > > support in the script below. After running the
backtest,
> > a new
> > > > > > symbol
> > > > > > > > will be added to your system holding the returns in
the
> > Close
> > > > > > field
> > > > > > > > (e.g. ~SP-500Returns).
> > > > > > > >
> > > > > > > > Aronson does not say what the moving average should
be.
> > It is
> > > > > > > > actually a bit of a challenge to find a period where
the
> > > > above is
> > > > > > > > profitable! However, after optimizing for the moving
> > average,
> > > > I
> > > > > > found
> > > > > > > > that the period 4/27/1998 - 12/31/2000 will show you
fully
> > > > > > invested
> > > > > > > > for the entire range with a 13.69% CAR.
> > > > > > > >
> > > > > > > > Applying my Detrend.afl script (posted in my last
note)
> > > > against
> > > > > > the
> > > > > > > > SP-500 over those dates will give a mean log daily
return
> > of
> > > > > > 2.40265E-
> > > > > > > > 10 (i.e. zero).
> > > > > > > >
> > > > > > > > You can compute the mean log daily return by opening
the
> > Quote
> > > > > > Editor
> > > > > > > > (e.g. select ~SP-500 in symbols tree, then use Symbol
|
> > Quote
> > > > > > > > Editor... menu item) and copying all the data into
Excel,
> > then
> > > > > > > > summing all the Close values for the given date range
and
> > > > > > dividing by
> > > > > > > > the number of quotes in that range (678 in this
example).
> > > > > > > >
> > > > > > > > Running the script below against the SP-500 for the
same
> > dates
> > > > > > will
> > > > > > > > will give a mean log daily return of 0.000428057
> > > > > > > >
> > > > > > > > Again, copy ~SP-500Returns from Quote Editor into
Excel
> > and
> > > > > > compute
> > > > > > > > the mean from the Close data.
> > > > > > > >
> > > > > > > > Peforming a Student's T-Test over the two sets of log
> > daily
> > > > > > returns,
> > > > > > > > assuming a shared variance and hypothesizing a shared
> > mean of
> > > > > > zero,
> > > > > > > > returns a one tailed p-value of 0.272346792. This is
well
> > > > above
> > > > > > 0.05
> > > > > > > > (the p-value required to reject the hypothesis with
95%
> > > > > > confidence)
> > > > > > > > and so we cannot reject the hypothesis.
> > > > > > > >
> > > > > > > > In other words, despite a 13.69% CAR over the period
> > tested,
> > > > the
> > > > > > > > strategy is no better than chance alone and thus
should
> > not be
> > > > > > traded.
> > > > > > > >
> > > > > > > > If the p-value had been .05 or less, we still could
not
> > > > conclude
> > > > > > > > (according to Aronson) that the strategy was any good
> > because
> > > > I
> > > > > > used
> > > > > > > > Optimization (MA period ranging from 5 to 200
increments
> > of
> > > > 5) to
> > > > > > > > find the best rule (data mining bias) and because I
used
> > the
> > > > 3%
> > > > > > band
> > > > > > > > width suggested in the book (snooping bias - have no
idea
> > how
> > > > much
> > > > > > > > optimization was applied to reach that number).
Applying
> > Monte
> > > > > > Carlo
> > > > > > > > analysis would remove the data mining bias (my next
> > effort).
> > > > > > There's
> > > > > > > > no getting around the snooping bias.
> > > > > > > >
> > > > > > > > 1. Copy paste the script below to a file on your
machine
> > (say
> > > > > > > > c:\Program
Files\AmiBroker\Formulas\Custom\Results.afl).
> > Make
> > > > sure
> > > > > > > > that you correct any formatting that gets messed up
from
> > this
> > > > > > post,
> > > > > > > > such that AmiBroker likes everything. Use the Tools |
> > Verify
> > > > > > Syntax
> > > > > > > > menu from the code editor.
> > > > > > > >
> > > > > > > > 2. Open a chart on the symbol that you want to
detrend.
> > For
> > > > > > example;
> > > > > > > > Aronson used the SP-500 for all his tests.
> > > > > > > >
> > > > > > > > 3. Open the Automatic Analysis Window
> > > > > > > >
> > > > > > > > 4. Click the "Pick" button and select the script that
you
> > just
> > > > > > saved
> > > > > > > > (i.e. Results.afl).
> > > > > > > >
> > > > > > > > 5. Select "current symbol" for the Apply To.
> > > > > > > >
> > > > > > > > 6. Select "from" for the Range, and enter a from date
and
> > a to
> > > > > > date
> > > > > > > > (e.g. from 4/27/1998 to 12/31/2000).
> > > > > > > >
> > > > > > > > 7. Click on Backtest
> > > > > > > >
> > > > > > > > A new symbol will be added to your system having the
same
> > > > name as
> > > > > > the
> > > > > > > > original, but prefixed with a "~" and appended
> > with "Return",
> > > > for
> > > > > > > > example "~SP-500Return". This symbol will appear in
Market
> > > > 253 and
> > > > > > > > contain detrended performance results for the range
> > selected
> > > > and
> > > > > > > > zeros for all other dates.
> > > > > > > >
> > > > > > > > For each bar in the detrended symbol, the information
> > will be
> > > > > > > > arranged as follows:
> > > > > > > >
> > > > > > > > Close: The detrended log daily return of the strategy
> > (i.e.
> > > > Pos0
> > > > > > x log
> > > > > > > > (Open2/Open1) - ALR).
> > > > > > > >
> > > > > > > > Note: As per Aronson, Pos0 refers to the strategy
signal
> > (+1
> > > > for
> > > > > > > > long, -1 for short), Open2 refers to the Open two days
> > from
> > > > now,
> > > > > > > > Open1 refers to the Open one day from now, ALR refers
to
> > the
> > > > > > average
> > > > > > > > log daily return of the market (not your strategy
results)
> > > > over
> > > > > > the
> > > > > > > > period being detrended.
> > > > > > > >
> > > > > > > > As always, comments, corrections and enhancements are
> > > > welcomed.
> > > > > > > >
> > > > > > > > Again, I'm a software developer, not a statistics
guy, and
> > > > this
> > > > > > is my
> > > > > > > > interpretation of the book. Do your own research
before
> > > > accepting
> > > > > > > > taking this as acurate.
> > > > > > > >
> > > > > > > > If anyone else has done any work in this area, I would
> > very
> > > > much
> > > > > > like
> > > > > > > > to hear if their approach agrees with my
interpretation.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Mike
> > > > > > > >
> > > > > > > > SetTradeDelays(1, 1, 1, 1); // All trades on next Open
> > after
> > > > > > > > EOD signal
> > > > > > > > SetBacktestMode(backtestRegular); // One symbol, no
> > redundant
> > > > > > signals
> > > > > > > > SetOption("InitialEquity", 100000);
> > > > > > > > SetOption("AccountMargin", 100);
> > > > > > > >
> > > > > > > > Plot(Close, "Close", colorBlue, styleLine);
> > > > > > > >
> > > > > > > > SetForeign("DTX--X");
> > > > > > > > center = MA(Close, Optimize("MA", 40, 5, 200, 5));
> > > > > > > > upper = 1.03 * center;
> > > > > > > > lower = .97 * center;
> > > > > > > >
> > > > > > > > Buy = Cross(Close, lower);
> > > > > > > > Sell = Cross(lower, Close);
> > > > > > > > Short = Sell;
> > > > > > > > Cover = Buy;
> > > > > > > >
> > > > > > > > Plot(Close, "DTX--X Close", colorYellow, styleLine);
> > > > > > > > Plot(upper, "DTX--X Upper", colorGreen, styleDashed);
> > > > > > > > Plot(center, "DTX--X MA", colorPink, styleLine);
> > > > > > > > Plot(lower, "DTX--X Lower", colorRed, styleDashed);
> > > > > > > >
> > > > > > > > RestorePriceArrays();
> > > > > > > >
> > > > > > > > BuyPrice = SellPrice = ShortPrice = CoverPrice = Open;
> > > > > > > >
> > > > > > > > PlotShapes(shapeUpArrow * Buy, colorGreen, 0, Close, -
30);
> > > > > > > > PlotShapes(shapeDownArrow * Sell, colorRed, 0 ,
Close, -
> > 30);
> > > > > > > > PlotShapes(shapeHollowDownArrow * Short, colorRed, 0 ,
> > > > Close, -
> > > > > > 40) ;
> > > > > > > > PlotShapes(shapeHollowUpArrow * Cover, colorGreen, 0,
> > Close, -
> > > > 40);
> > > > > > > >
> > > > > > > > isLong = Flip(Buy, Sell);
> > > > > > > > isShort = Flip(Short, Cover);
> > > > > > > > market = Foreign("~" + Name(), "C", false);
> > > > > > > >
> > > > > > > > AddToComposite(IIF(isLong, 1 * market, 0), "~" + Name
()
> > > > > > > > + "Return", "C", atcFlagDefaults |
> > atcFlagEnableInBackTest);
> > > > > > > > AddToComposite(IIF(isShort, -1 * market, 0), "~" +
Name()
> > > > > > > > + "Return", "C", atcFlagDefaults |
> > atcFlagEnableInBackTest);
> > > > > > > >
> > > > > > > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com><amibroker%
> > 40yahoogroups.com><amibroker%
> > > > 40yahoogroups.com><amibroker%
> > > > > > 40yahoogroups.com>, "Louis
> > > > > > > > Préfontaine"
> > > > > > > > <rockprog80@> wrote:
> > > > > > > > >
> > > > > > > > > Hi again,
> > > > > > > > >
> > > > > > > > > I've searched thru the manual and I think that += is
> > simply
> > > > a
> > > > > > > > shortcut. I
> > > > > > > > > wrote total = total+ raw[i]; instead and there is no
> > more
> > > > error!
> > > > > > > > >
> > > > > > > > > I did exactly as you said and got the sign with
those
> > huge
> > > > bars
> > > > > > all
> > > > > > > > topping
> > > > > > > > > at 0.267615. I think I understand what you say, but
> > what is
> > > > OI?
> > > > > > > > >
> > > > > > > > > The thing is: I am not sure how to use this. I think
> > what I
> > > > need
> > > > > > > > is to make
> > > > > > > > > an average of all this data and then subtract this
> > average
> > > > from
> > > > > > > > each day log
> > > > > > > > > of the actual stock to detrend. Am I correct? Would
you
> > be
> > > > kind
> > > > > > > > enough to
> > > > > > > > > give me some tips about how to use this new
information
> > to
> > > > > > actually
> > > > > > > > backtest
> > > > > > > > > one of my rules and see how it performs when it is
> > > > detrended?
> > > > > > > > >
> > > > > > > > > Thanks a lot!
> > > > > > > > >
> > > > > > > > > Louis
> > > > > > > > >
> > > > > > > > > 2008/3/2, Louis Préfontaine <rockprog80@>:
> > > > > > > > > >
> > > > > > > > > > Hi Mike,
> > > > > > > > > >
> > > > > > > > > > Thank you so much for your reply!
> > > > > > > > > >
> > > > > > > > > > This is EXACTLY what I am looking for, from the
> > Aronson's
> > > > > > book!
> > > > > > > > > >
> > > > > > > > > > I can't wait to make it work... Right now there
is a
> > small
> > > > > > > > problem with
> > > > > > > > > > the formula... I get an error message for this
parti
> > > > total +=
> > > > > > raw
> > > > > > > > [i]; Ln12:
> > > > > > > > > > col:8:Error 30. Syntax error.
> > > > > > > > > >
> > > > > > > > > > I tried to change the += for = "" == and it
works...
> > Is it
> > > > > > > > possible that
> > > > > > > > > > the AB version that I have doesn't recognize the
+=?
> > Or
> > > > maybe
> > > > > > > > there is an
> > > > > > > > > > error with the +=? Is it possible to get to the
same
> > > > result in
> > > > > > > > any other
> > > > > > > > > > way?
> > > > > > > > > >
> > > > > > > > > > Thanks a lot!
> > > > > > > > > >
> > > > > > > > > > Louis
> > > > > > > > > >
> > > > > > > > > > 2008/3/2, Mike <sfclimbers@>:
> > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > Based on your formula, I assume that you are
> > referring
> > > > to
> > > > > > > > Chapter 1
> > > > > > > > > > > of David Aronson's book: Evidence Based
Technical
> > > > Analysis.
> > > > > > > > > > >
> > > > > > > > > > > That being the case, I am providing a script
below.
> > > > > > > > > > >
> > > > > > > > > > > However, I believe that the formula that you
> > originally
> > > > > > posted
> > > > > > > > is not
> > > > > > > > > > > correct. Aronson's formula calls for multiplying
> > your
> > > > > > boolean
> > > > > > > > > > > strategy signal (i.e. +1 for long vs. -1 for
short)
> > by
> > > > the
> > > > > > > > detrended
> > > > > > > > > > > daily returns, *not* the Close by the returns!
> > > > > > > > > > >
> > > > > > > > > > > Also, the book does not go into detail for tri-
state
> > > > > > strategies
> > > > > > > > (i.e.
> > > > > > > > > > > long/neutral/short) nor for long/neutral or
> > > > short/neutral
> > > > > > > > strategies.
> > > > > > > > > > > I'm assuming that plugging in a signal value of
0
> > would
> > > > be
> > > > > > > > acceptable
> > > > > > > > > > > for a neutral position, but haven't researched
that
> > > > yet. So,
> > > > > > > > just be
> > > > > > > > > > > careful how you use the data once you've
detrended
> > it.
> > > > > > > > > > >
> > > > > > > > > > > Anyway, here is a script that I believe will
> > detrend the
> > > > > > market
> > > > > > > > > > > returns as per the book. Currently, the script
is
> > > > intended
> > > > > > for
> > > > > > > > > > > detrending a single symbol. I have not yet got
> > around to
> > > > > > making
> > > > > > > > it
> > > > > > > > > > > work against a watchlist of symbols (coming
soon).
> > > > > > > > > > >
> > > > > > > > > > > 1. Copy paste the script below to a file on your
> > machine
> > > > > > (say
> > > > > > > > > > > c:\Program
> > Files\AmiBroker\Formulas\Custom\Detrend.afl).
> > > > > > Make
> > > > > > > > sure
> > > > > > > > > > > that you correct any formatting that gets
messed up
> > from
> > > > > > this
> > > > > > > > post,
> > > > > > > > > > > such that AmiBroker likes everything. Use the
Tools
> > |
> > > > Verify
> > > > > > > > Syntax
> > > > > > > > > > > menu from the code editor.
> > > > > > > > > > >
> > > > > > > > > > > 2. Open a chart on the symbol that you want to
> > detrend.
> > > > For
> > > > > > > > example;
> > > > > > > > > > > Aronson used the SP-500 for all his tests.
> > > > > > > > > > >
> > > > > > > > > > > 3. Open the Automatic Analysis Window
> > > > > > > > > > >
> > > > > > > > > > > 4. Click the "Pick" button and select the script
> > that
> > > > you
> > > > > > just
> > > > > > > > saved
> > > > > > > > > > > (i.e. Detrend.afl).
> > > > > > > > > > >
> > > > > > > > > > > 5. Select "current symbol" for the Apply To.
> > > > > > > > > > >
> > > > > > > > > > > 6. Select "from" for the Range, and enter a from
> > date
> > > > and a
> > > > > > to
> > > > > > > > date
> > > > > > > > > > > (e.g. from 1/1/2007 to 12/31/2007).
> > > > > > > > > > >
> > > > > > > > > > > 7. Click on Backtest
> > > > > > > > > > >
> > > > > > > > > > > A new symbol will be added to your system
having the
> > > > same
> > > > > > name
> > > > > > > > as the
> > > > > > > > > > > original, but prefixed with a "~", for
example "~SP-
> > > > 500".
> > > > > > This
> > > > > > > > symbol
> > > > > > > > > > > will appear in Market 253 and contain detrended
> > market
> > > > > > > > information
> > > > > > > > > > > for the range selected (e.g. all of 2007 as
above)
> > and
> > > > zeros
> > > > > > > > for all
> > > > > > > > > > > other dates.
> > > > > > > > > > >
> > > > > > > > > > > For each bar in the detrended symbol, the
> > information
> > > > will
> > > > > > be
> > > > > > > > > > > arranged as follows:
> > > > > > > > > > >
> > > > > > > > > > > Open: The unadjusted log daily return (i.e. log
> > > > > > (Open2/Open1)).
> > > > > > > > > > >
> > > > > > > > > > > High: The total sum of all unadjusted log daily
> > returns.
> > > > > > > > > > >
> > > > > > > > > > > Low: The average of all unadjusted log daily
returns
> > > > (i.e.
> > > > > > ALR).
> > > > > > > > > > >
> > > > > > > > > > > Close: The detrended log daily return (i.e. log
> > > > > > (Open2/Open1) -
> > > > > > > > ALR).
> > > > > > > > > > >
> > > > > > > > > > > OI: The number of bars over which the data has
been
> > > > > > detrended.
> > > > > > > > > > >
> > > > > > > > > > > Note: As per Aronson, Open2 refers to the Open
two
> > days
> > > > from
> > > > > > > > now,
> > > > > > > > > > > Open1 refers to the Open one day from now, ALR
> > refers
> > > > to the
> > > > > > > > average
> > > > > > > > > > > log return over the period being detrended.
> > > > > > > > > > >
> > > > > > > > > > > Note: I have used the natural logarithm in my
code
> > (i.e.
> > > > > > ln), as
> > > > > > > > > > > opposed to the base 10 logarithm (i.e. log10). I
> > don't
> > > > know
> > > > > > if
> > > > > > > > that
> > > > > > > > > > > makes a difference.
> > > > > > > > > > >
> > > > > > > > > > > Note: To find your detrended strategy results,
you
> > still
> > > > > > must
> > > > > > > > write
> > > > > > > > > > > your own code to calculate which of the
detrended
> > daily
> > > > > > returns
> > > > > > > > your
> > > > > > > > > > > strategy would pick up, and which sign to use
(+/-)
> > when
> > > > > > > > multiplying
> > > > > > > > > > > by the detrended return for that day.
> > > > > > > > > > >
> > > > > > > > > > > Note: I ran this script against SP-500 for the
> > entire
> > > > year
> > > > > > of
> > > > > > > > 2007.
> > > > > > > > > > > With my data source, the average detrended log
daily
> > > > return
> > > > > > > > (i.e. all
> > > > > > > > > > > the Close values of ~SP-500 divided by 251
actual
> > > > trading
> > > > > > days)
> > > > > > > > ended
> > > > > > > > > > > up being -6.00797E-10 which is effectively zero.
> > So, I'm
> > > > > > > > assuming
> > > > > > > > > > > that it works.
> > > > > > > > > > >
> > > > > > > > > > > Corrections and enhancements welcomed :)
> > > > > > > > > > >
> > > > > > > > > > > Mike
> > > > > > > > > > >
> > > > > > > > > > > procedure Detrend(compositeName) {
> > > > > > > > > > > local range; range = Status("barinrange");
> > > > > > > > > > > local raw; raw = log(Ref(Open, 2)/Ref(Open, 1));
> > > > > > > > > > > local total; total = 0;
> > > > > > > > > > > local count; count = 0;
> > > > > > > > > > > local offset; offset = 0;
> > > > > > > > > > >
> > > > > > > > > > > for (i = 0; i < BarCount; i++) {
> > > > > > > > > > > if (range[i]) {
> > > > > > > > > > > if (NOT IsNull(raw[i])) {
> > > > > > > > > > > count++;
> > > > > > > > > > > total += raw[i];
> > > > > > > > > > > }
> > > > > > > > > > > }
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > if (count > 0) {
> > > > > > > > > > > AddToComposite(IIF(range, raw, Null), "~" +
> > > > > > compositeName, "O",
> > > > > > > > > > > atcFlagDefaults | atcFlagEnableInBackTest);
> > > > > > > > > > >
> > > > > > > > > > > offset = total/count;
> > > > > > > > > > > raw = IIF(IsNull(raw), offset, raw);
> > > > > > > > > > >
> > > > > > > > > > > AddToComposite(IIF(range, raw - offset,
Null), "~" +
> > > > > > > > > > > compositeName, "C", atcFlagDefaults |
> > > > > > atcFlagEnableInBackTest);
> > > > > > > > > > > AddToComposite(IIF(range, total, Null), "~" +
> > > > > > > > compositeName, "H",
> > > > > > > > > > > atcFlagDefaults | atcFlagEnableInBackTest);
> > > > > > > > > > > AddToComposite(IIF(range, offset, Null), "~" +
> > > > > > > > > > > compositeName, "L", atcFlagDefaults |
> > > > > > atcFlagEnableInBackTest);
> > > > > > > > > > > AddToComposite(IIF(range, count, Null), "~" +
> > > > > > > > compositeName, "I",
> > > > > > > > > > > atcFlagDefaults | atcFlagEnableInBackTest);
> > > > > > > > > > > } else {
> > > > > > > > > > > AddToComposite(Null, "~" + compositeName, "X",
> > > > > > atcFlagDefaults |
> > > > > > > > > > > atcFlagEnableInBackTest);
> > > > > > > > > > > }
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > Buy = Sell = Short = Cover = 0;
> > > > > > > > > > > Detrend(Name());
> > > > > > > > > > >
> > > > > > > > > > > --- In amibroker@xxxxxxxxxxxxxxx<amibroker%
40yahoogroups.com>
> > <amibroker%
> > 40yahoogroups.com><amibroker%
> > > > 40yahoogroups.com><amibroker%
> > > > > > 40yahoogroups.com><amibroker%40yahoogroups.com>,
> > > > > > > > > > > "louisprefontaine" <rockprog80@>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Anybody can help?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Louis
> > > > > > > > > > > >
> > > > > > > > > > > > --- In amibroker@xxxxxxxxxxxxxxx<amibroker%
40yahoogroups.com>
> > <amibroker%
> > 40yahoogroups.com><amibroker%
> > > > 40yahoogroups.com><amibroker%
> > > > > > 40yahoogroups.com><amibroker%
> > > > > > > > 40yahoogroups.com>, "Louis
> > > > > > > > > > > Préfontaine" <rockprog80@>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > I am trying to build a formula to "detrend"
the
> > > > market.
> > > > > > > > > > > > >
> > > > > > > > > > > > > What I want to set is something like this
> > > > > > > > > > > > >
> > > > > > > > > > > > > Close of day 0 * ( log (open day2/open day
1) -
> > > > average
> > > > > > log
> > > > > > > > > > > > return of
> > > > > > > > > > > > > every day of the data available.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Anybody can do that?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Louis
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>