[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] Re: Adventures in improving Ami Backtest speed



PureBytes Links

Trading Reference Links

Hello,
 
Yes this is actually the same calculation, but written in more readable and portable way.
 
Previous code was in fact one liner.
  int ret = year+year/4-year/100+year/400+"-bed=pen+mad."[month]+day;
 
The
 
"-bed=pen+mad."[month]
 
bit is indexing in-line char array instead of explicit array as defined in your example.
It makes code shorter but less readable and less portable
(may not work on double-byte or some other encodings). Well.. the beauty(?) of C :-)

Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: Paul Ho
Sent: Wednesday, July 16, 2008 9:27 AM
Subject: RE: [amibroker] Re: Adventures in improving Ami Backtest speed

My day of week code for the "hard stuff" is as follows only 3 lines
int dayofweek(int y, int m, int d) /* 0 = Sunday */
{
     static int t[] = {0, 3, 2, 5, 0, 3, 5, 1, 4, 6, 2, 4};
     y -= m < 3;
     return (y + y/4 - y/100 + y/400 + t[m-1] + d) % 7;
}
courtesy to Tomohiko Sakamoto

From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Tomasz Janeczko
Sent: Wednesday, 16 July 2008 4:48 PM
To: amibroker@xxxxxxxxxxxxxxx
Subject: Re: [amibroker] Re: Adventures in improving Ami Backtest speed

Hello,

Thank you very much for posting.

Note that mathematically your RoundToPenny does different thing
than Prec. Your function performs rounding (adds 0.5) while prec does
truncation. Also when applied to negative numbers RoundToPenny may lead to incorrect results,
for -12 input it will give -11.99 result. It is non-issue if you apply to positive-only price data.

As for your Dayofweek code, it seems like some part got damaged as these two lines
do not compile and some variables are missing:

int ret = year+year/4-year/100+year/400+"-
bed=pen+mad."[month]+day;

Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: "dloyer123" <dloyer123@xxxxxxcom>
To: <amibroker@xxxxxxxxxps.com>
Sent: Wednesday, July 16, 2008 4:49 AM
Subject: [amibroker] Re: Adventures in improving Ami Backtest speed

> Here the the dll code for fast DayOfWeek(), RoundPenny() and
> IsOptionsWeek().
>
> I release this code to the public domain. Anyone may do anything
> they want with it....
>
> There is a dll with this code in the file area. Look for my name.
>
>
> // round to a penny
> AmiVar RoundPenny(int NumArgs, AmiVar *ArgsTable) {
> AmiVar result = gSite.AllocArrayResult();
> int nSize = gSite.GetArraySize();
> float *SrcArray = ArgsTable[0].array;
> int j = SkipEmptyValues(nSize,SrcArray,result.array);
>
> // round each to a pennny
> for (int i = j; i < nSize; i++) {
> float val = SrcArray[i]*100+0.5f;
> int ival = (int) val;
> val = ival/100.0f;
> result.array[i] = val;
> }
> return result;
> }
>
> // day of week logic, runs over array
> void dow(float *in, float *out, int cnt) {
> int j = SkipEmptyValues(cnt,in, out);
> long lastday = 0;
> long lastret = 0;
> for (int i = j; i < cnt; i++) {
> long val = (long) in[i];
> if (val == lastday) out[i] = (float) lastret;
> else if (val == lastday+1) {
> lastret++; lastret %= 7;
> lastday = val;
> out[i] = (float) lastret;
> } else {
> // we have to calc it the hard way
> lastday = val;
> int day = val % 100;
> val /= 100;
> int month = val % 100;
> val /= 100;
> int year = val + 1900;
> if (month < 3) year--;
> int ret = year+year/4-year/100+year/400+"-
> bed=pen+mad."[month]+day;
> // ret++; // now convert to sun = 0;
> ret %= 7;
> lastret = ret;
> out[i] = (float) ret;
> }
> }
> }
>
>
>
> // fast day of week
> // pass in datenum
> AmiVar FDayOfWeek(int NumArgs, AmiVar *ArgsTable) {
> AmiVar result = gSite.AllocArrayResult();
> int nSize = gSite.GetArraySize();
>
> //FILE *log = fopen("log","w");
> //fprintf(log,"fdayofweek elements:%d\n", nSize);
> //fclose(log);
> float *SrcArray = ArgsTable[0].array;
> dow(SrcArray, result.array, nSize);
> return result;
> }
>
> void iow(float *dt, float *dow, float *out, int cnt) {
> int j = SkipEmptyValues(cnt,dt,out);
> int ldt = 0;
> float lval = 0.0f;
>
> for (int i = j; i < cnt; i++) {
> int idtnum = (int) dt[i];
> if (idtnum == ldt) out[i] = lval;
> else {
> int idow = (int) dow[i];
> int iday = idtnum % 100;
> int df = (12-idow)%7 + iday;
> if (df >= 15 && df <= 21) lval = 1.0f;
> else lval = 0.0f;
> out[i] = lval;
> ldt = idtnum;
> }
> }
> }
>
>
> // calc is options week
> // pass datenum, dow
> //df = (12-expwd)%7 + Day();
> //expisow = df >= 15 && df <= 21;
> AmiVar IsOptionsWeek(int NumArgs, AmiVar *ArgsTable) {
> AmiVar result = gSite.AllocArrayResult();
> int nSize = gSite.GetArraySize();
> float *dt = ArgsTable[0].array;
> float *dow = ArgsTable[1].array;
> iow(dt,dow,result.array,nSize);
> return result;
> }
>
> --- In amibroker@xxxxxxxxxps.com, "Tomasz Janeczko" <groups@xxx>
> wrote:
>>
>> Hello,
>>
>> Thank you for your feedback. It is quite interesting what you
>> are saying about DayOfWeek() and Prec().
>>
>> As for DayOfWeek() it is true that no optimizations were
>> done in this function and it simply calls C runtime mktime() for
> dates >= 1970
>> and Windows OLE date functions for earlier dates that are
>> not particularly fast and does so for every bar, so
>> it could be speeded up especially if operating on intraday data
>> when day changes infrequently.
>>
>> I assume that as far as DayOfWeek is considered you are just calling
>> it once and save the result in variable.
>>
>> As for Prec(): Prec() uses floor() C-runtime function plus one
> multiplication
>> and one division per bar. In general case it is the optimum choice,
> however
>> more efficient ways can be found in specialized cases when for
> example
>> you are truncating fractional part only.
>>
>>
>> Best regards,
>> Tomasz Janeczko
>> amibroker.com
>> ----- Original Message -----
>> From: "dloyer123" <dloyer123@x..>
>> To: <amibroker@xxxxxxxxxps.com>
>> Sent: Tuesday, July 15, 2008 6:59 PM
>> Subject: [amibroker] Adventures in improving Ami Backtest speed
>>
>>
>> > Greetings
>> >
>> > I just wanted to share my results in improving AmiBroker backtest
>> > performance.
>> >
>> > I use 5 min intraday data over a 1,000 symbol database and
>> > Amibroker's new walkforward support. However each walkforward
> step
>> > takes a long time probably due to the large size of the data
> set.
>> >
>> > I have looked into ways to improve the run time performance.
> Here
>> > are some of the things I tried:
>> >
>> > 1) Buy new computer. This had the best results. Run time per
> pass
>> > dropped greatly. A new core 2 processor with the highest clock
> rate
>> > I could find was about twice as fast as my old laptop.
>> >
>> > 2) Use "Check AFL" to find functions that are slow and write them
> in
>> > C. I found that the DayOfWeek() function is very slow in
> comparison
>> > to other operations, so is Prec(). I moved these to C and
> reduced my
>> > run time a good bit. My DayOfWeek() function exploits the fact
> that
>> > each bar of a day is the same day of the week and the first bar
> of
>> > the day is probably the day after the last. It avoids finding
> the
>> > actual day on each bar. This saved more time than any of the
> other
>> > optimizations.
>> >
>> > 3) Rewrite the whole system in C. Sadly this only saved a about
> 10%,
>> > not enough to justify the risk of bugs and the trouble of
> maintaining
>> > the code. I went to no great lengths to optimize the code, but
> it
>> > goes to show how little overhead AFL adds and how highly
> optimized
>> > the AFL code is already.
>> >
>> > 4) Write a C route to cache results that do not depend on the
> value
>> > being optimized. Since many of the calculations dont change with
>> > each optimization step or only depend on a single optimization
>> > parameter, they dont need to be recalculated with each
> optimization
>> > step. I had high hopes for this approach and spent some time on
> it.
>> > I was able to get cache hit rates up over 99% on a walkforward
> test,
>> > but ran into memory management problems. I could overcome the
>> > problems, but the results where disappointing. It only saved
> about
>> > 20% of the time per run. I suspect that there is enough overhead
> in
>> > setting up the stock arrays that avoiding some calculations did
> not
>> > make much difference. Also, it becomes hard to track what each
>> > calculation depends on.
>> >
>> > It was interesting that many of the values could be compressed
> using
>> > simple repeat coding. 200MB was enough the cache all of the
> values
>> > that did not depend on any optimization parm and are re-used many
>> > times for each optimization pass.
>> >
>> > 5) Improvements to AmiBroker. The new optimizer was a big help,
> so
>> > was the new support for QuickALF. These required no changes to
> the
>> > code and had a large improvement and made walkforward testing
> much
>> > more practical. The new optimizer allowed me to replace my own
>> > search code that I had hacked using AFL.
>> >
>> > 6) Multi core support - Sadly I have not found a practical way to
> put
>> > the other cores on my system to work. To work with the new
>> > optimization framework, it looks like this is best done within
> the
>> > ami code itself.
>> >
>> > 7) Really exotic stuff - I played with the CUDA api a bit. Very
> cool
>> > stuff. It allows of the 64 or so cores in the graphics
> processor.
>> > These are each able to perform one single precision floating
> point
>> > operation per cycle. To work, I would have to preload the
> database
>> > into the graphics card memory and find a way to avoid any per
> symbol
>> > overhead in amibroker. This would just shove the
> buy/sell/buyprice,
>> > etc arrays back to ami to process the trade list. If I could
> find a
>> > way to avoid any overhead in setting up open/high/low/close
> arrays in
>> > amibroker, I would do it.
>> >
>> > Fresh out of other ideas.
>> >
>> >
>> >
>> > ------------------------------------
>> >
>> > Please note that this group is for discussion between users only.
>> >
>> > To get support from AmiBroker please send an e-mail directly to
>> > SUPPORT {at} amibroker.com
>> >
>> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
>> > http://www.amibroker.com/devlog/
>> >
>> > For other support material please check also:
>> > http://www.amibroker.com/support.html
>> > Yahoo! Groups Links
>> >
>> >
>> >
>>
>
>
>
> ------------------------------------
>
> Please note that this group is for discussion between users only.
>
> To get support from AmiBroker please send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> For other support material please check also:
> http://www.amibroker.com/support.html
> Yahoo! Groups Links
>
>
>

__._,_.___

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html




Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___