[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] Paul Ho: Memory Challenges with Great Ranking Tool

PureBytes Links

Trading Reference Links
Hello,
 
Well, it is technically possible to create sum of ordinal rankings,
by running subsequent backtests (can be automatic via optimization)
and reading/writing files generated in multiple passes.
Problem is it is neither elegant or particularly fast solution.
 
Summing up scores leads of course to different results.
 
I am not sure how critical (time-critical) is this for you, but generally speaking
really fast solution would be to write this procedure as a plugin in C/C++.
 
The other solution is using Osaka plugin for that which features 2 dimensional array sorting feature,
but this would require a bit of work.
 
If you do not find any better solution by yourself or with help of other people, I may look into this
later (when I finish the work on 5.15).

Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: Ken Close
Sent: Sunday, July 06, 2008 11:12 PM
Subject: RE: [amibroker] Paul Ho: Memory Challenges with Great Ranking Tool

Tomasz:
 
Of course you are right, it does do bar for bar basis.  I confirmed with examination of output.html after changing n =1 setting (sorry).
 
However, I do not think the rawbacktester can do what I want, unless you speak to my need/desire for summing ordinal rankings of multiple indicators.
 
In the version of the code (which is inefficient and sorts n*n), I calculate ordinal rankings of RSI14 then, separately, create ordinal rankings of ROC and sum them together, then sort on the sum.
 
To try and see if the rawbacktester could produce the same result, I used the following procedure (twice to check).
 
I created a PositionScore = RSI14 + (ROC( 14 ) + 1000); and compared the sorting of the rawbacktester to the sorting of the sum of the ordinals, and the two are not the same.  Sorting by the sum of the ordinals of the individual indicators is what I must have and it does not seem like the backtester can do it.
 
I fully expect I am still missing something in the rawbacktester but, if so, could you point a little deeper and tell me if the rawbacktester can somehow be manipulated to use its fast sort capability to do what I want, or could you tell me how to make the code below more efficient for larger populated Watchlists.
 
Sorry to drag you through this multiple times.
 
Thanks,
 
Ken
 

From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Tomasz Janeczko
Sent: Sunday, July 06, 2008 3:20 PM
To: amibroker@xxxxxxxxxxxxxxx
Subject: Re: [amibroker] Paul Ho: Memory Challenges with Great Ranking Tool

Hello,
 
No, look again. The code I provided gives the sort is ON BAR BY BAR basis.

Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: Ken Close
Sent: Sunday, July 06, 2008 9:08 PM
Subject: RE: [amibroker] Paul Ho: Memory Challenges with Great Ranking Tool

Tomasz:
 
Thanks for all the help you give to so many people, me included.
 
However, while I did as you suggested with the custombacktester, and looked into the output file it produces, I am at a loss to know how to use the data it contains.  It is not all of the data that I need.
 
I want the ordinal ranking of multiple indicators, add them all together, per bar and per symbol, and use the final sum, of the ORDINAL ranks, as the ranking value for all symbols.
 
This output represents what I want (but it is only for two indicators).  I want to turn this into my "recipe" which will have approximately 8 to 10 indicators.
 
 
I ran the custom backtest, opened the output.html file, and see that the symbols are sorted by the ranking value and it is indeed an ordinal value.  But, the sort is done only once (probably as a lastbar basis) and Paul Ho sorting algorithm gives me ordinal values for each bar for each symbol (displayed above using a lastbar basis).
 
You say Paul's code is inefficient, and maybe it is because it sorts all symbols by all bars.  Can you suggest a change to the specific code that would do what I want, but more efficiently?
 
Again, thanks for all that you do.
 
Ken

From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Tomasz Janeczko
Sent: Sunday, July 06, 2008 1:39 PM
To: amibroker@xxxxxxxxxxxxxxx
Subject: Re: [amibroker] Paul Ho: Memory Challenges with Great Ranking Tool

Hello,
 
The code is inefficient because it repeats the sorting N*N times where N is number of symbols, while
only N times is enough.
 
Ranking is a process that is done during first pass of backtest. It is implemented efficiently. 
We can use this built-in process easily using custom backtest procedure as shown here:
 
Note that this formula will not produce output in AA directly. Instead it will produce a HTML
file (output.html) that you can later import to AA using AA, File->Import
 
Also please be warned that produced files are huge and attempt to load such big HTML file
into Internet Explorer instead will easily hang IE.
 

PositionScore = ROC( C, 14 ) + 1000; // WHAT YOU WANT TO RANK

SetOption("MaxOpenPositions", 10 );
SetBacktestMode( backtestRegularRaw );
Buy=1;
Sell=0;
SetCustomBacktestProc("");
if( Status("action")==actionPortfolio )
{
  bo = GetBacktesterObject();

  bo.PreProcess();

  dt = DateTime();

  fh = fopen("output.html", "w" );

  fputs("<TABLE><TR><TH>Symbol</TH><TH>Date/Time</TH><TH>Rank</TH></TR>\n", fh );

  for( i = 0; i < BarCount; i++ )
  {
    k = 1;  
    for( sig = bo.GetFirstSignal( i ); sig; sig = bo.GetNextSignal( i ) )
    {
       Line = "<TR><TD>" + sig.Symbol + "</TD><TD>" +
              DateTimeToStr( dt[ i ] ) + "</TD><TD>" + k + "</TD></TR>\n";
       fputs( Line, fh );
       k++;
    }
   }

  bo.PostProcess();

  fputs( "</TABLE>", fh );
  fclose( fh );
}


Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: Ken Close
Sent: Sunday, July 06, 2008 5:35 PM
Subject: [amibroker] Paul Ho: Memory Challenges with Great Ranking Tool

Paul:
 
my initial euphoria has turned somewhat downward as I attempt to apply the code below (just two indicators) to larger Watchlists.  You sounded (from other messages) like someone who knows the ins and outs of memory management with AB, and perhaps can comment on how to keep the code below from "bogging down".
 
In spite of my many years with AB and its array processing, my mind still has a problem wrapping around what this code is doing and why (and whether) larger populated Watchlists will ever be able to work.
 
I initially tested against the DJ-30 (30 symbols) and all went well, fairly quickly, perhaps 10-15 seconds.
 
I then tried the NDX (100 symbols) and things went more slowly but finished.  I noticed the symbols appearing in the AA window more slowly.
 
I have not been able to nor wanted to wait for the SP-500, as the symbols appear more and more slowly and the est time counter was saying something like 1 1/2 hours to complete 500 symbols.
 
I was assuming that the code had to collect or process all symbols before it could make comparisons among them---this is probably false or else why would processed symbols start to appear in the AA window while it is still accessing symbols. 
 
What suggestions can you make, given your understanding of the code and AB, that would minimize the processing of large member watchlists?
 
Can adding a SetBarsRequired in the right place limit the number of lookback bars that are processed, and thus speed up execution?
 
As the number of indicators I wish to process into a "Total Rank" score increases, I imagine that executing this code will get slower and slower and may not be possible at all.  Would you agree?
 
Thanks for any added help.
 
Ken

From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Ken Close
Sent: Saturday, July 05, 2008 10:47 AM
To: amibroker@xxxxxxxxxxxxxxx
Subject: [amibroker] What a Great Ranking Tool

Paul Ho has come up with a supurb ranking tool.  I have expanded it to two indicators.  Feel free to expand the code structure to any number of indicators.
 
Possible next step: stick the Tot_Rank values into the OI field for the symbols, then Plot the Ranks for a visual representation of "where the symbol is over time".
 
The possibilities are endless (or at least enlarged because of Paul's code idea).  Thanks Paul for your creative input.
 
Ken
 

//  Ranking_Alt01.afl    KSC    07/05/2008

//  Original code by Paul Ho, Amibroker list 07/05/2008

//  Modifications and expansions by Ken Close 07/05/2008

 

//  Will ordinal rank every symbol in watchlist for every bar.

 

 

mOwnROC = ROC(C, 14);

mOwnRSI = RSIa(C, 14);

mRoc = 0;

mRSI = 0;

list = CategoryGetSymbols(categoryWatchlist, 16);

ROCcount[0] = rocrank[0] = 0;

RSIcount[0] = RSIrank[0] = 0;

for(i = 0; (sym = StrExtract(list, i)) != ""; i++)

  {

   SetForeign(sym);

   mRoc = ROC(C, 14);

   mRSI = RSIa(C, 14);

   RestorePriceArrays();

   n = !IsNull(mRoc);

   m = !IsNull(mRSI);

   roccount += n;

   rsicount += m;

   rocrank = IIf(Nz(mRoc) > mOwnROC, Rocrank + n, rocrank);

   rsirank = IIf(Nz(mRsi) > mOwnRSI, Rsirank + m, rsirank);

   Totrank = rocrank + rsirank;

  }

ROCn = ROC(C, 14);

RSIn = RSIa(C, 14);

Filter = 1;

Buy = Sell = 0;

AddColumn(ROCn, "ROCn",1.2);

AddColumn(RSIn, "RSIn",1.2);

AddColumn(mRoc, "MROC", 1.2);

AddColumn(ROCrank, "ROCRank", 1.0);

AddColumn(RSIrank, "rsirank",1.0);

AddColumn(Totrank, "Totrank", 1.0);

 

//  To check the sorting, run on a watchlist, then click once on the date column,

//  Then shift click on one of the indicators, ie, RSIn, and you will see the

//  ordinal values in order.

 

 

 

__._,_.___

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html



Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___