1. Find the HHV and compute the difference between
HHV and current value.
2. Rank / sort issues based on the difference
value.
3. Select the top 500 from the sorted
list.
Sorting and ranking code has been discussed before
... look at the UKB site as a starting point
----- Original Message -----
Sent: Saturday, July 12, 2008 6:01
PM
Subject: Re: [amibroker] Re: Paul Ho:
Memory Challenges with Great Ranking Tool
Hi,
Let's say that what I consider best is how close to
HHV value of 52 weeks. How can I use this code to generate a backtest of
precisely 500 symbols?
Thanks,
Louis
2008/7/12 Ara Kaloustian < ara1@xxxxxxxxxx>:
First, you need to define what you consider the
"best".
Then you can use your criteria to select
them
-----
Original Message -----
Sent:
Saturday, July 12, 2008 5:13 PM
Subject:
Re: [amibroker] Re: Paul Ho: Memory Challenges with Great Ranking
Tool
Hi, I tried to run this code and I just clicked
"verify syntax" and it was gone... kind of crashed or made me wait
forever... Is it possible to use this code on 8000 tickers
and then select the 500 "best" tickers in the list and use them for
backtest? How would one do
that? Thanks, Louis
2008/7/12 Paul Ho < paul.tsho@xxxxxxxxx>:
upgrade to the latest version of AB
Can some one help me i try to run this ranking tool but get the
next errors RestorePriceArrays();
n =
!IsNull(mRoc);
m = !IsNull(mRSI);
roccount
+= -------------^
Error 30. Syntax error
n =
!IsNull(mRoc);
m = !IsNull(mRSI);
roccount +=
n;
rsicount += -------------^
Error 30. Syntax
error
than the result for only one ticker in my watchlist shows
up
regards Rene
--- In amibroker@xxxxxxxxxxxxxxx, "bruce1r"
<brucer@xx> wrote: > > Tomasz - > >
Great diagnosis. I should've have thought of it. I've seen it
happen > to a number of the FT users. That is why we
stressed cache settings > in the 2007 conference class that you
helped with. > > Ken - > > There is no
reason to run anything > 5200 bars with a FT database. >
Soon that will change to a lower number at next weekend's
conversion. > And, the cache size setting needs to big big
enough to handle > 500 > symbols. For FT, that is at least
> 83 MB. FWIW, I run 5200 bars, > 1000 symbols, and 160MB
cache. > > Paul - > > In the interest of
conveying some info constructively, let me explain > why I
posted what I did. > > First, though, I think you might
want to re-consider abount a range > algorithm being slower. A
2*N algorithm like the range algorithm will > almost ALWAYS
be faster than a N*N algorithm - it will be much faster >
for large N. > > I hope that Ken is able to get a N*N
approach fast enough. I totally > agree with you that it can
be. He is still too slow, though. > > But, the other
reason that I suggested range values to Ken was than he > is
using it for ranking and probably rotational systems. This is
what > many from the FastTrack community gravitate toward.
He will have to > confirm that part. But, in that setting, the
range percentages can > capture info about closeness of mixed
scores that ordinal ranking can > mask. If it fits the score
distribution characteristics, this can be > used to minimize
rotation with little impact on fitness - which is > typically a
goal. > > Lastly, about the Pad and Align, I can only
tell you that my > experience is different. In a 20 year
database like the FastTrack > database like I think that Ken is
using, if you only need a few years > of backtest, then
aligning to a symbol with a short history will > dramatically
speed things up (thanks to Tomasz's implementation). > >
Anyway, this has been an interesting discussion. Let me know if
you > need any details about the above. I'm getting back to some
other > trading work. > > -- BruceR >
> > --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko"
<groups@> wrote: > > > > Hello, > >
> > Indeed I have run this code on larger watch list and can
confirm > Paul timings. > > > > I am running
Athonl 64 x2 @2GHz > > > > For 100 symbols it is
just 50 seconds. > > For 500 symbols (SP500) and history from
1992 till now (16 years) it > is just 13 minutes. >
> If the same 500 symbols are explored using QuickAFL turned on
and > last 4 years only, the time shrinks to 6 minutes 49
seconds . > > These are actual timings, not progress bar
estimate. > > Yes the time grows slightly as process
progresses but it is not that > surprising considering the
fact it outputs > > 500 * number of quotes lines (1.3 million
lines for 10 year history) > > > > I think this
timings are quite OK for N*N algorithm. > > > >
Memory also is not an issue. Running it on 500 symbols 16
years > history with full caching enabled > > caused
that AmiBroker consumed 175MB of RAM. > > > > If
you are getting timings in hours, I suspect that you are using >
sub-optimum cache settings. Please go to > >
Tools->Preferences, "Data" tab and increase "in-memory" cache to
at > least 500 symbols (the size of watch list under
test). > > If cache is too small it will force many disk
accesses. With cache > large enough - everything will be in
RAM. > > > > Best regards, > > Tomasz
Janeczko > > amibroker.com > > ----- Original Message
----- > > From: Paul Ho > > To: amibroker@xxxxxxxxxxxxxxx > > Sent:
Monday, July 07, 2008 9:29 AM > > Subject: RE: [amibroker]
Re: Paul Ho: Memory Challenges with Great > Ranking
Tool > > > > > > Ken > > As you
know, the algorithm that I gave you increase dramatically > with
the size of the watchlist. However, the times that you stated >
isn't in line with what I experience. I tried my code (which
is > similar to yours, with exception that it also stores the
value in OI) > on 2 machines. One very old machine, a single
core AMD which is about > 5 years old, Window tells me it is
an AMD XP 2600 with 1G of ram. I > use a watchlist of 472
symbols, running from 1/1/2000 to now. the time > taken is
6.5 minutes (from the progress bar). and on a newer machine >
Core 2 duo E6600, it took just over 1 minute. So this is very >
different to the one and a half hour that you are talking
about. > > So I am wondering what machine you're running on.
and what kind of > AA setting you use? What I will suggest
is that you DONT Check pad > and align, this will increase the
number of bars by quite a lot. You > can check QuickAFL
though. > > I have also given you a few suggestions in one of
the other posts, > including monthly bars, normalisation of
scores etc > http://finance.groups.yahoo.com/group/amibroker/message/126336.
Did > you take a look at that? > > Despite its
shortcoming, I think the N^2 algorithm will still > perform
better/faster than Bruce's suggestions. Pad and Align &/or
ATC > will slow it down even more. In the past, I have made
a ranking dll > which uses a 2 dimensional array, and basically
insert the stock into > the right ranking order as each
symbol is scanned, a little bit like > Fred's algorithm but on a
total array basis. That is certainly very > fast. In addition,
Tomasz's custom Backtester code has the potential > also to be
quite fast without the use of dll. I have ported the code > so
it stores directly in the OI during after optimization. However,
it > comes back with an internal error in certain instances
if the > watchlist gets over 1400 symbols and no of bars is more
than 2500. I > have sent it to Tomasz for him to have a look at,
when he comes back. > I can share that with you. But my
point is that the N^2 idea should be > fast enough for what
you are talking about ~500 stocks. You see there > is a big
difference between 1 minute and 1 and half hour. > > Send me
a private email if you like, I'm curious why there such a >
big difference. > > Cheers > > > >
> > > > > > > > >
---------------------------------------------------------- -------- >
> From: amibroker@xxxxxxxxxxxxxxx > [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Ken
Close > > Sent: Monday, 7 July 2008 7:40 AM > > To:
amibroker@xxxxxxxxxxxxxxx > > Subject: RE:
[amibroker] Re: Paul Ho: Memory Challenges with > Great Ranking
Tool > > > > > > Bruce: > >
> > Thanks for giving me another way to go. > >
> > In case you have been following (or remember) this topic
from a > ways back, > > Fred was generous enough to
write me a code concept for doing > all of this > >
(but end of range only) and I successfully converted it to my >
"endpoint" > > recipe of 11 or so indicators. It did ok for
relatively large > watchlists > > (even the RUT) but
could not handle very well some much larger > watchlists
in > > the many 1000s. But, it gave me a combined ranking on
any end > point date I > > set in the AA window. My
absolute minimum requirement is to create > >
combo-rank-scores on a monthly basis but I was pleasantly >
surprised when > > Paul Ho served up a concept to create
daily combo-rank-scores on > a daily > > basis, but
then euphoria changed to despair as I encountered the > n^2
time > > factor. > > > > So, thanks to you
and others I have a variety of ways to > consider getting
to > > the end of this problem. > > > > 1.
Your suggestion of normalized indicators and using a final >
percentage > > value as the combo rank. > > >
> 2. Taking Fred's code and finding a way to manipulate the >
EndofRange date, > > basically repeating his code over and
over on the same watchlist > but with > > changing
EndofRange dates. (I still have to do something with > the
collection > > of combo-scores I will accumulate by date, but
that is another > issue.*** see > > below) I have even
considered manually repeating the process to > the
end > > point (only 12 runs per year x the 8 years I want to
test over). > > > > 3. Taking Paul Ho's code
which ranks daily and either living > with the > >
limitation in the Watchlist population or running the thing
over > night. > > Since I have speed problems now
with 2 indicators and 150+ > symbols, I > > probably
will drop off the cliff with 11 indicators and the same >
150+ > > symbols. An alternate which I plan to test next is
to see how > Paul's code > > performs on a Weekly or
even Monthly compressed basis, although > if symbol >
> number is controlling and not barcount, then this will not
do > much good.) > > > > 4. Using Tomasz's
suggestion of the custombacktester, making 11 >
separate > > runs, then somehow combining the 11 different
output reports, > coming up with > > a combo-rank that
way. > > > > If you were approaching this, can you
guess and say which > approach you would > >
concentrate on. Right now, number 4 looks like it actually
might > be the > > least programming and execution
intensive, but I am not sure. I > also have > > to
have a way of updating the entire system as time goes > forward.
That will > > bring an additional set of challenges I am
sure. > > > > Thanks for stepping in. > >
> > Ken > > > > *** Paul Ho shared a
small COM code snippit that sticks an > indicator nicely >
> into the OI field of a symbol, so that is the approach I want
to > take once I > > have the combo-rank to stick
in the right place. Talk about > complex...... > >
> > PS: Bruce, if you are still reading, would I have a
better chance of > > executing my task in Trade vs
Amibroker (sorry Thomasz)? > > > > -----Original
Message----- > > From: amibroker@xxxxxxxxxxxxxxx > [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf > > Of
bruce1r > > Sent: Sunday, July 06, 2008 5:03 PM > >
To: amibroker@xxxxxxxxxxxxxxx > > Subject:
[amibroker] Re: Paul Ho: Memory Challenges with Great > Ranking
Tool > > > > Ken - > > > > I'm
too involved with something else right now, but let me see > if
I can > > offer quick suggestion. First - > >
> > 1. Tomasz is pointing out the solutions in (N^2) time
are never > practical > > past some limit. That
means that the execution time goes up with > the
square > > of the number of items - ticker in this case.
There are a couple of > > programming tricks that you can
play, but I don't think that > they are going > > to
get you where you want to go - > > > > For example,
programming tricks can be used to make the N^2 >
comparison > > matrix "triangular". This reduces the
comparisons by half. > > > > You might use Pad and
Align to a ticker with a short history to > cut the
time > > further. > > > > But, this is
still going to leave you in a long timeframe. > > >
> 2. It looks like you are trying to add unbounded indicators
and > use the > > ordinal values to normalize them
so that they can be combined. > > Use of the custom
backtester would still require that you > generate
output > > for each indicator and then combine them. >
> > > Another approach might be to go out of the box a
little and > question your > > basic assumption. Here's
what I mean. > > > > Ordinal values can be used to
convert unbounded ranges (such as > ROC) to > >
bounded values. But they can do some strange things to outliers.
> > For example, consider these points. Say they are for
tickers > A,B,C,D,E on a > > particular day - >
> > > 0, 20, 21, 22, 200 > > > > The
point 22 is ranked #2 (higher value better) when it is not >
near the top. > > > > ON THE OTHER HAND, range
value can be used also to convert > unbounded data to >
> bounded. THEY REQUIRE A PRE-SCAN TO KNOW THE MIN AND MAX.
> > For the range above, it would convert to the following
percentages - > > > > 0, 10, 10.5, 11,
100 > > > > This has some advantages for certain
data distributions, but some > > disadvantages for
others. For data where the probability of > outliers is >
> low, it yields similar results. > > > > SO,
HERE'S WHAT YOU MIGHT DO. > > > > 1. Take a
watchlist and start a Exploration pass. When > >
Status("stocknum") == 0, loop through the list and find the >
global Min and > > Max for each bar across all of the tickers
for a given indicator > and store > > it in an ATC
in the H and L fields. For RSI and ROC, you would > have 2
ATC's > > - say ~MINMAX_ROC and ~MINMAX_RSI. This is 1 pass
of all N tickers. > > > > 2. Continuing on for
stocknum 0 and for 1 - N, calculate the ROC > and RSI
and > > convert it to a percentage of the MIN and MAX range
that you > stored in the > > ATC's for each bar
- > > > > rangepcnt = ( tickscore - tickglobalmin )
/ ( tickmax - > tickglobalmin > > ) * 100; > >
> > 3. Now you can combine the range values because they
are > normalized. > > If you divide by the number of
indicators, you'll end up with a > combined > >
percentage. > > > > Now, while this is not an
ordinal rank, it works perfectly well > for scoring >
> and is a solution in 2*N time. BTW - this reference won't
mean > much to most > > here, but should to you - Ed
Gilbert detailed this in Trade doc > almost a > >
decade ago. > > > > -- BruceR > > >
> --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko"
<groups@> wrote: > > > > > >
Hello, > > > > > > No, look again. The code I
provided gives the sort is ON BAR > BY BAR > >
basis. > > > > > > Best regards, > >
> Tomasz Janeczko > > > amibroker.com > > > ----- Original
Message ----- > > > From: Ken Close > > >
To: amibroker@xxxxxxxxxxxxxxx > > > Sent:
Sunday, July 06, 2008 9:08 PM > > > Subject: RE:
[amibroker] Paul Ho: Memory Challenges with Great > >
Ranking Tool > > > > > > > > >
Tomasz: > > > > > > Thanks for all the help
you give to so many people, me included. > > >
> > > However, while I did as you suggested with the
custombacktester, > > and looked into the output file it
produces, I am at a loss to > know how to > > use the
data it contains. It is not all of the data that I need. >
> > > > > I want the ordinal ranking of multiple
indicators, add them all > > together, per bar and per
symbol, and use the final sum, of the > ORDINAL > >
ranks, as the ranking value for all symbols. > > >
> > > This output represents what I want (but it is only
for two > > indicators). I want to turn this into my "recipe"
which will have > > approximately 8 to 10
indicators. > > > > > > > > >
> > > I ran the custom backtest, opened the output.html
file, and see > > that the symbols are sorted by the
ranking value and it is > indeed an ordinal > > value.
But, the sort is done only once (probably as a lastbar > >
basis) and Paul Ho sorting algorithm gives me ordinal values
for > each bar > > for each symbol (displayed above
using a lastbar basis). > > > > > > You say
Paul's code is inefficient, and maybe it is because it >
> sorts all symbols by all bars. Can you suggest a change to
the > specific > > code that would do what I want, but
more efficiently? > > > > > > Again, thanks
for all that you do. > > > > > > Ken >
> > > > > > > > > >
> > >
---------------------------------------------------------- >
> -- > > > From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] > > On Behalf Of
Tomasz Janeczko > > > Sent: Sunday, July 06, 2008 1:39
PM > > > To: amibroker@xxxxxxxxxxxxxxx > > > Subject:
Re: [amibroker] Paul Ho: Memory Challenges with Great > >
Ranking Tool > > > > > > > > >
Hello, > > > > > > The code is inefficient
because it repeats the sorting N*N times > > where N is
number of symbols, while > > > only N times is
enough. > > > > > > Ranking is a process that
is done during first pass of backtest. > > It is
implemented efficiently. > > > We can use this built-in
process easily using custom backtest > > procedure as
shown here: > > > > > > Note that this
formula will not produce output in AA directly. > >
Instead it will produce a HTML > > > file (output.html)
that you can later import to AA using AA, > >
File->Import > > > > > > Also please be
warned that produced files are huge and attempt to > >
load such big HTML file > > > into Internet Explorer
instead will easily hang IE. > > > > > >
PositionScore = ROC( C, 14 ) + 1000; // WHAT YOU WANT TO
RANK > > > > > >
SetOption("MaxOpenPositions", 10 ); > > >
SetBacktestMode( backtestRegularRaw ); > > > Buy=1;
> > > Sell=0; > > >
SetCustomBacktestProc(""); > > > if(
Status("action")==actionPortfolio ) > > > { > >
> bo = GetBacktesterObject(); > > > > > >
bo.PreProcess(); > > > > > > dt =
DateTime(); > > > > > > fh =
fopen("output.html", "w" ); > > > > > >
> > >
fputs ("<TABLE><TR><TH>Symbol</TH><TH>Date/Time</TH><TH>Rank</TH></TR>\n", >
> fh ); > > > > > > for( i = 0; i <
BarCount; i++ ) > > > { > > > k = 1;
> > > for( sig = bo.GetFirstSignal( i ); sig; sig
= > > bo.GetNextSignal( i ) ) > > > { >
> > Line = "<TR><TD>" + sig.Symbol +
"</TD><TD>" + > > > DateTimeToStr( dt[ i ] )
+ "</TD><TD>" + k + > >
"</TD></TR>\n"; > > > fputs( Line, fh );
> > > k++; > > > } > > >
} > > > > > > bo.PostProcess(); > >
> > > > fputs( "</TABLE>", fh ); > >
> fclose( fh ); > > > } > > > >
> > > > > Best regards, > > > Tomasz
Janeczko > > > amibroker.com > > > ----- Original
Message ----- > > > From: Ken Close > > >
To: amibroker@xxxxxxxxxxxxxxx > > > Sent:
Sunday, July 06, 2008 5:35 PM > > > Subject: [amibroker]
Paul Ho: Memory Challenges with Great > > Ranking
Tool > > > > > > > > >
Paul: > > > > > > my initial euphoria has
turned somewhat downward as I attempt to > > apply the
code below (just two indicators) to larger Watchlists. You >
> sounded (from other messages) like someone who knows the ins
and > outs of > > memory management with AB, and
perhaps can comment on how to > keep the code > > below
from "bogging down". > > > > > > In spite of
my many years with AB and its array processing, my > >
mind still has a problem wrapping around what this code is
doing > and why > > (and whether) larger populated
Watchlists will ever be able to work. > > > >
> > I initially tested against the DJ-30 (30 symbols) and all
went > > well, fairly quickly, perhaps 10-15
seconds. > > > > > > I then tried the NDX
(100 symbols) and things went more slowly > > but
finished. I noticed the symbols appearing in the AA window >
more slowly. > > > > > > I have not been able
to nor wanted to wait for the SP-500, as > > the symbols
appear more and more slowly and the est time counter > was
saying > > something like 1 1/2 hours to complete 500
symbols. > > > > > > I was assuming that the
code had to collect or process all > > symbols before it
could make comparisons among them---this is > probably
false > > or else why would processed symbols start to appear
in the AA > window while > > it is still accessing
symbols. > > > > > > What suggestions can
you make, given your understanding of the > > code and
AB, that would minimize the processing of large member >
watchlists? > > > > > > Can adding a
SetBarsRequired in the right place limit the number > >
of lookback bars that are processed, and thus speed up
execution? > > > > > > As the number of
indicators I wish to process into a "Total > > Rank" score
increases, I imagine that executing this code will > get
slower > > and slower and may not be possible at all. Would
you agree? > > > > > > Thanks for any added
help. > > > > > > Ken > > >
> > > > > > > > > > >
---------------------------------------------------------- >
> > From: amibroker@xxxxxxxxxxxxxxx > > [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Ken
Close > > > Sent: Saturday, July 05, 2008 10:47 AM >
> > To: amibroker@xxxxxxxxxxxxxxx > > > Subject:
[amibroker] What a Great Ranking Tool > > > > >
> > > > Paul Ho has come up with a supurb ranking
tool. I have expanded > > it to two indicators. Feel free
to expand the code structure to > any number > > of
indicators. > > > > > > Possible next step:
stick the Tot_Rank values into the OI field > > for the
symbols, then Plot the Ranks for a visual representation >
of "where > > the symbol is over time". > > >
> > > The possibilities are endless (or at least enlarged
because of > > Paul's code idea). Thanks Paul for your
creative input. > > > > > > Ken > >
> > > > // Ranking_Alt01.afl KSC 07/05/2008 >
> > > > > // Original code by Paul Ho, Amibroker
list 07/05/2008 > > > > > > // Modifications
and expansions by Ken Close 07/05/2008 > > > > >
> > > > > > > // Will ordinal rank every
symbol in watchlist for every bar. > > > > >
> > > > > > > > > > >
> > mOwnROC = ROC(C, 14); > > > > > >
mOwnRSI = RSIa(C, 14); > > > > > > mRoc =
0; > > > > > > mRSI = 0; > > >
> > > list = CategoryGetSymbols(categoryWatchlist,
16); > > > > > > ROCcount[0] = rocrank[0] =
0; > > > > > > RSIcount[0] = RSIrank[0] =
0; > > > > > > for(i = 0; (sym =
StrExtract(list, i)) != ""; i++) > > > > > >
{ > > > > > > SetForeign(sym); > >
> > > > mRoc = ROC(C, 14); > > > >
> > mRSI = RSIa(C, 14); > > > > > >
RestorePriceArrays(); > > > > > > n =
!IsNull(mRoc); > > > > > > m =
!IsNull(mRSI); > > > > > > roccount +=
n; > > > > > > rsicount += m; > >
> > > > rocrank = IIf(Nz(mRoc) > mOwnROC, Rocrank +
n, rocrank); > > > > > > rsirank =
IIf(Nz(mRsi) > mOwnRSI, Rsirank + m, rsirank); > > >
> > > Totrank = rocrank + rsirank; > > >
> > > } > > > > > > ROCn =
ROC(C, 14); > > > > > > RSIn = RSIa(C,
14); > > > > > > Filter = 1; > >
> > > > Buy = Sell = 0; > > > >
> > AddColumn(ROCn, "ROCn",1.2); > > > > >
> AddColumn(RSIn, "RSIn",1.2); > > > > > >
AddColumn(mRoc, "MROC", 1.2); > > > > > >
AddColumn(ROCrank, "ROCRank", 1.0); > > > > >
> AddColumn(RSIrank, "rsirank",1.0); > > > >
> > AddColumn(Totrank, "Totrank", 1.0); > > >
> > > > > > > > > // To check
the sorting, run on a watchlist, then click once on > >
the date column, > > > > > > // Then shift
click on one of the indicators, ie, RSIn, and you > >
will see the > > > > > > // ordinal values
in order. > > > > > > >
------------------------------------ > > > > Please
note that this group is for discussion between users only. >
> > > To get support from AmiBroker please send an e-mail
directly to > SUPPORT {at} > > amibroker.com >
> > > For NEW RELEASE ANNOUNCEMENTS and other news always
check DEVLOG: > > http://www.amibroker.com/devlog/ > >
> > For other support material please check also: >
> http://www.amibroker.com/support.html > >
Yahoo! Groups Links >
> >
__._,_.___
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
__,_._,___
|