[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: [amibroker] Re: More on Database Structure and Local Storage



PureBytes Links

Trading Reference Links

Tomasz:

This is a great suggestion.  I will work on this further.

I am so very impressed with the speed of the rawbacktester.

After getting my FT db to local storage finally, I was able to repeat two tests.

Writing OI field using N^2 logic code snippit by Paul Ho.
Time to complete 426 symbols using combination of two indicators = 7min 25sec.
All sums of ordinal values written successfully to OI.

Test using custom backtester, writing directly to OI field, code again courtesy of Paul Ho (listed below for interest)
Time to complete same 426 symbols using combination of two indicators = 1min 42sec.
All sums of ordinal values written successfully to OI.

Interestingly, after retrieving the OI values and writing them to csv file, combining both results into one csv file (Test1 and Test2), two things were obvious: the values written to the OI field were absolutely different integers from both tests, and, the sort order of the OI field contents put both lists in exactly the same order!

Ticker Date/Time Price           OI Ticker Date/Time Price OI
AMGN 7/3/2008 50.84 2 AMGN 7/3/2008 50.84 4
SDS 7/3/2008 68.61 8 SDS 7/3/2008 68.61 10
JBL 7/3/2008 16.29 16 JBL 7/3/2008 16.29 16
NBR 7/3/2008 48.71 20 NBR 7/3/2008 48.71 18
KG 7/3/2008 10.58 30 KG 7/3/2008 10.58 17
BAX 7/3/2008 64.12 34 BAX 7/3/2008 64.12 43
EDS 7/3/2008 24.7 36 EDS 7/3/2008 24.7 52
GENZ 7/3/2008 73.03 37 GENZ 7/3/2008 73.03 39

With your suggestion of the artificial ticker symbol and the apparent lightning speed of the custombacktester (used for ranking and writing), my hopes are alive again that I can do my application within FT, without writing to external files, and at speeds that may allow me a reasonable time to complete the full database.

Wow, the happiness of great results coming together.

Many thanks for your help.

Ken

For those interested in the code written for this test by Paul Ho, here it is.  Tomasz has said several times that the ranking done in the custombacktester is fast and he certainly has understated the case here,

run = Optimize("run", 1, 1, 2, 1);

function ranking(Ordinal)

{

    switch(Ordinal)

    {

    case 1:

        res = ROC(C, 14) + 1000;

        break;

    case 2:

        res = RSI(14);

        break;

    default:

        res = EMA(C, 50);

        break;

    }

    return res;

}

PositionScore = ranking(run); // WHAT YOU WANT TO RANK

 

SetOption("MaxOpenPositions", 50 ); //AB only keeps 2* maxpos top rank signals

SetBacktestMode( backtestRegularRaw );

Buy=1;

Sell= Short = Cover = 0;

ab = CreateObject("broker.Application");

target = ab.Stocks(Name());

target.isDirty = True;

SetCustomBacktestProc("");

if( Status("action")==actionPortfolio )

{

    bo = GetBacktesterObject();

    bo.PreProcess();

    dt = DateTime();

//  fh = fopen("output.csv", "w"); uncomment these if you want to save it to a csv file as well

//  fputs("Symbol,Date,Rank\n", fh);

    for( i = 0; i < BarCount; i++ )

    {

        k = 1;  

        for( sig = bo.GetFirstSignal( i ); sig; sig = bo.GetNextSignal( i ) )

        {

//         Line = sig.Symbol + "," + DateTimeToStr(dt[i]) + "," + k + "\n";

//         fputs(Line, fh);

           target = ab.Stocks(sig.Symbol);

           qt = target.Quotations(DateTimeToStr(dt[i]));

           if(qt)

           {

               if(run == 1)

                   qt.OpenInt = k;

               else

                   qt.OpenInt += k;

           }

           target.isDirty = True;

       k++;

        }

    }

 

    bo.PostProcess();

//  if(fh)fclose(fh);

 



-----Original Message-----
From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Tomasz Janeczko
Sent: Tuesday, July 08, 2008 1:16 PM
To: amibroker@xxxxxxxxxxxxxxx
Subject: Re: [amibroker] Re: More on Database Structure and Local Storage

Ken,

Instead of writing to the very same symbol OI field, you may consider writing to separate 'artificial' symbol (like composite).
That way you will have both: direct access to FT and ability to store what you need.

The only difference in the code is that instead of using actual symbol, you use artificial one (say with tilde in front).

Instead of this:
      for( sig = bo.GetFirstSignal( i ); sig; sig = bo.GetNextSignal( i ) )
      {
         target = ab.Stocks(sig.Symbol);
         qt = target.Quotations(i);
         if(qt)
         {


You would need this:

      for( sig = bo.GetFirstSignal( i ); sig; sig = bo.GetNextSignal( i ) )
      {
         target = ab.Stocks.Add("~" + sig.Symbol); // create artificial symbol
         target.DataSource = 1; // use only local DB
         qts = target.Quotations;
         qt = qts.Add(DateTimeToStr(dt[i]));
         if(qt)
         {


Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: "Ken Close" <ken45140@xxxxxxxxx>
To: <amibroker@xxxxxxxxxxxxxxx>
Sent: Tuesday, July 08, 2008 6:59 PM
Subject: RE: [amibroker] Re: More on Database Structure and Local Storage


> Barry:
>
> FT has its own datafiles and AB reads these files via the FastTrack PlugIn.
> I want to write a ranking score into the OI field of each symbol, as the
> calculation of the ranking symbol takes a long time to do itself.
>
> As I have learned, you copy to the local database by doing a Scan over the
> entire external database with local set as the source; then symbol contents
> are copied.  That is all fine, and I could do my calc and stick it into OI
> field, but then the next time I need to update the local database, the OI
> fields will be overwritten.
>
> Sort of circular.
>
> Ken
>
> -----Original Message-----
> From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf
> Of Barry Scarborough
> Sent: Tuesday, July 08, 2008 8:12 AM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: [amibroker] Re: More on Database Structure and Local Storage
>
> There is a parameter for each symbol, Symbol > Information > Use only local
> database, yes or no. But that will only keep the data from being back filled
> if set to yes and you may not want to do that. If I read your note right you
> want to import the data into AB and then keep it updated from some data
> source, maybe FT. If so the parameter has to be set to no.
>
> How did you copy or import the FT data into AB? AB is kind of fussy about
> what you put in their database. If you import the data incorrectly it will
> blow up every time you try to use the symbol.
>
> What do you want to write into the OI field? When are you trying to write
> it?
>
> Barry
>
> --- In amibroker@xxxxxxxxxxxxxxx, "Ken Close" <ken45140@xxx> wrote:
>>
>> Can someone explain to me how I make the FastTrack database a local
> one? 
>> 
>> Background: I have had a FT db for years, with local storage
> disabled?  I
>> want to write to the OI field and preserve the written data. I
> checked
>> "Enable Local Storage" but nothing different happened.
>> 
>> Next, I tried to scan all symbols with a simple formula (I
> remembered having
>> to do that years ago with a TC2000 database that came on a CD--I
> thought the
>> same kind of step was needed.).   Amibroker got hung on a single
> symbol and
>> I had to End Program.  AB would lock if I even clicked on this same
> symbol.
>> 
>> Next, I created a new FastTrack database with a different name and
> copied
>> over the appropriate files to the new DB folder. I enabled Local
> Storage
>> from the start.  All looked good.  Except that writes to the OI
> field in the
>> new DB would not save.  I tried the scan all symbols trick again,
> and again
>> it hung on the same symbol, certainly suggesting my FastTrack data
> itself is
>> corrupt on that one symbol. (Special updates with the FastTrack data
>> designed to correct problems did not seem to change anything.)
>> 
>> My question is how do I get the FastTrack database to be local and
> to allow
>> writes to the OI field to stick? 
>> My second question (as I assume I have to have the data in the
> Amibroker
>> folders and not just in the FastTrack folders for the AB DB to be
> local), is
>> how do I update the nightly downloads of new data which FastTrack
> will stick
>> in their own file/folder?
>> 
>> Thomasz, can you help?
>> 
>> Thanks,
>> 
>> Ken
>>
>
>
>
> ------------------------------------
>
> Please note that this group is for discussion between users only.
>
> To get support from AmiBroker please send an e-mail directly to SUPPORT {at}
> amibroker.com
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> For other support material please check also:
> http://www.amibroker.com/support.html
> Yahoo! Groups Links
>
>
>
>
> ------------------------------------
>
> Please note that this group is for discussion between users only.
>
> To get support from AmiBroker please send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> For other support material please check also:
> http://www.amibroker.com/support.html
> Yahoo! Groups Links
>
>
>

------------------------------------

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:amibroker-digest@xxxxxxxxxxxxxxx
    mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

__._,_.___

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html




Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___