|
PureBytes Links
Trading Reference Links
|
Yep, Paul's version is better and every bit as fast.
Sorry about loosing the formatting in the cut and paste. It is much
harder to read without proper indenting.
--- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@xxx>
wrote:
>
> Hello,
>
> Yes this is actually the same calculation, but written in more
readable and portable way.
>
> Previous code was in fact one liner.
> int ret = year+year/4-year/100+year/400+"-bed=pen+mad."[month]
+day;
>
>
> The
>
> "-bed=pen+mad."[month]
>
> bit is indexing in-line char array instead of explicit array as
defined in your example.
> It makes code shorter but less readable and less portable
> (may not work on double-byte or some other encodings). Well.. the
beauty(?) of C :-)
>
> Best regards,
> Tomasz Janeczko
> amibroker.com
> ----- Original Message -----
> From: Paul Ho
> To: amibroker@xxxxxxxxxxxxxxx
> Sent: Wednesday, July 16, 2008 9:27 AM
> Subject: RE: [amibroker] Re: Adventures in improving Ami Backtest
speed
>
>
> My day of week code for the "hard stuff" is as follows only 3
lines
> int dayofweek(int y, int m, int d) /* 0 = Sunday */
> {
> static int t[] = {0, 3, 2, 5, 0, 3, 5, 1, 4, 6, 2, 4};
> y -= m < 3;
> return (y + y/4 - y/100 + y/400 + t[m-1] + d) % 7;
> }
> courtesy to Tomohiko Sakamoto
>
> --------------------------------------------------------------------
--------
> From: amibroker@xxxxxxxxxxxxxxx
[mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Tomasz Janeczko
> Sent: Wednesday, 16 July 2008 4:48 PM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: Re: [amibroker] Re: Adventures in improving Ami
Backtest speed
>
>
> Hello,
>
> Thank you very much for posting.
>
> Note that mathematically your RoundToPenny does different thing
> than Prec. Your function performs rounding (adds 0.5) while
prec does
> truncation. Also when applied to negative numbers RoundToPenny
may lead to incorrect results,
> for -12 input it will give -11.99 result. It is non-issue if
you apply to positive-only price data.
>
> As for your Dayofweek code, it seems like some part got damaged
as these two lines
> do not compile and some variables are missing:
>
> int ret = year+year/4-year/100+year/400+"-
> bed=pen+mad."[month]+day;
>
> Best regards,
> Tomasz Janeczko
> amibroker.com
> ----- Original Message -----
> From: "dloyer123" <dloyer123@xxx>
> To: <amibroker@xxxxxxxxxxxxxxx>
> Sent: Wednesday, July 16, 2008 4:49 AM
> Subject: [amibroker] Re: Adventures in improving Ami Backtest
speed
>
> > Here the the dll code for fast DayOfWeek(), RoundPenny() and
> > IsOptionsWeek().
> >
> > I release this code to the public domain. Anyone may do
anything
> > they want with it....
> >
> > There is a dll with this code in the file area. Look for my
name.
> >
> >
> > // round to a penny
> > AmiVar RoundPenny(int NumArgs, AmiVar *ArgsTable) {
> > AmiVar result = gSite.AllocArrayResult();
> > int nSize = gSite.GetArraySize();
> > float *SrcArray = ArgsTable[0].array;
> > int j = SkipEmptyValues(nSize,SrcArray,result.array);
> >
> > // round each to a pennny
> > for (int i = j; i < nSize; i++) {
> > float val = SrcArray[i]*100+0.5f;
> > int ival = (int) val;
> > val = ival/100.0f;
> > result.array[i] = val;
> > }
> > return result;
> > }
> >
> > // day of week logic, runs over array
> > void dow(float *in, float *out, int cnt) {
> > int j = SkipEmptyValues(cnt,in, out);
> > long lastday = 0;
> > long lastret = 0;
> > for (int i = j; i < cnt; i++) {
> > long val = (long) in[i];
> > if (val == lastday) out[i] = (float) lastret;
> > else if (val == lastday+1) {
> > lastret++; lastret %= 7;
> > lastday = val;
> > out[i] = (float) lastret;
> > } else {
> > // we have to calc it the hard way
> > lastday = val;
> > int day = val % 100;
> > val /= 100;
> > int month = val % 100;
> > val /= 100;
> > int year = val + 1900;
> > if (month < 3) year--;
> > int ret = year+year/4-year/100+year/400+"-
> > bed=pen+mad."[month]+day;
> > // ret++; // now convert to sun = 0;
> > ret %= 7;
> > lastret = ret;
> > out[i] = (float) ret;
> > }
> > }
> > }
> >
> >
> >
> > // fast day of week
> > // pass in datenum
> > AmiVar FDayOfWeek(int NumArgs, AmiVar *ArgsTable) {
> > AmiVar result = gSite.AllocArrayResult();
> > int nSize = gSite.GetArraySize();
> >
> > //FILE *log = fopen("log","w");
> > //fprintf(log,"fdayofweek elements:%d\n", nSize);
> > //fclose(log);
> > float *SrcArray = ArgsTable[0].array;
> > dow(SrcArray, result.array, nSize);
> > return result;
> > }
> >
> > void iow(float *dt, float *dow, float *out, int cnt) {
> > int j = SkipEmptyValues(cnt,dt,out);
> > int ldt = 0;
> > float lval = 0.0f;
> >
> > for (int i = j; i < cnt; i++) {
> > int idtnum = (int) dt[i];
> > if (idtnum == ldt) out[i] = lval;
> > else {
> > int idow = (int) dow[i];
> > int iday = idtnum % 100;
> > int df = (12-idow)%7 + iday;
> > if (df >= 15 && df <= 21) lval = 1.0f;
> > else lval = 0.0f;
> > out[i] = lval;
> > ldt = idtnum;
> > }
> > }
> > }
> >
> >
> > // calc is options week
> > // pass datenum, dow
> > //df = (12-expwd)%7 + Day();
> > //expisow = df >= 15 && df <= 21;
> > AmiVar IsOptionsWeek(int NumArgs, AmiVar *ArgsTable) {
> > AmiVar result = gSite.AllocArrayResult();
> > int nSize = gSite.GetArraySize();
> > float *dt = ArgsTable[0].array;
> > float *dow = ArgsTable[1].array;
> > iow(dt,dow,result.array,nSize);
> > return result;
> > }
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@>
> > wrote:
> >>
> >> Hello,
> >>
> >> Thank you for your feedback. It is quite interesting what
you
> >> are saying about DayOfWeek() and Prec().
> >>
> >> As for DayOfWeek() it is true that no optimizations were
> >> done in this function and it simply calls C runtime mktime()
for
> > dates >= 1970
> >> and Windows OLE date functions for earlier dates that are
> >> not particularly fast and does so for every bar, so
> >> it could be speeded up especially if operating on intraday
data
> >> when day changes infrequently.
> >>
> >> I assume that as far as DayOfWeek is considered you are just
calling
> >> it once and save the result in variable.
> >>
> >> As for Prec(): Prec() uses floor() C-runtime function plus
one
> > multiplication
> >> and one division per bar. In general case it is the optimum
choice,
> > however
> >> more efficient ways can be found in specialized cases when
for
> > example
> >> you are truncating fractional part only.
> >>
> >>
> >> Best regards,
> >> Tomasz Janeczko
> >> amibroker.com
> >> ----- Original Message -----
> >> From: "dloyer123" <dloyer123@>
> >> To: <amibroker@xxxxxxxxxxxxxxx>
> >> Sent: Tuesday, July 15, 2008 6:59 PM
> >> Subject: [amibroker] Adventures in improving Ami Backtest
speed
> >>
> >>
> >> > Greetings
> >> >
> >> > I just wanted to share my results in improving AmiBroker
backtest
> >> > performance.
> >> >
> >> > I use 5 min intraday data over a 1,000 symbol database and
> >> > Amibroker's new walkforward support. However each
walkforward
> > step
> >> > takes a long time probably due to the large size of the
data
> > set.
> >> >
> >> > I have looked into ways to improve the run time
performance.
> > Here
> >> > are some of the things I tried:
> >> >
> >> > 1) Buy new computer. This had the best results. Run time
per
> > pass
> >> > dropped greatly. A new core 2 processor with the highest
clock
> > rate
> >> > I could find was about twice as fast as my old laptop.
> >> >
> >> > 2) Use "Check AFL" to find functions that are slow and
write them
> > in
> >> > C. I found that the DayOfWeek() function is very slow in
> > comparison
> >> > to other operations, so is Prec(). I moved these to C and
> > reduced my
> >> > run time a good bit. My DayOfWeek() function exploits the
fact
> > that
> >> > each bar of a day is the same day of the week and the
first bar
> > of
> >> > the day is probably the day after the last. It avoids
finding
> > the
> >> > actual day on each bar. This saved more time than any of
the
> > other
> >> > optimizations.
> >> >
> >> > 3) Rewrite the whole system in C. Sadly this only saved a
about
> > 10%,
> >> > not enough to justify the risk of bugs and the trouble of
> > maintaining
> >> > the code. I went to no great lengths to optimize the code,
but
> > it
> >> > goes to show how little overhead AFL adds and how highly
> > optimized
> >> > the AFL code is already.
> >> >
> >> > 4) Write a C route to cache results that do not depend on
the
> > value
> >> > being optimized. Since many of the calculations dont
change with
> >> > each optimization step or only depend on a single
optimization
> >> > parameter, they dont need to be recalculated with each
> > optimization
> >> > step. I had high hopes for this approach and spent some
time on
> > it.
> >> > I was able to get cache hit rates up over 99% on a
walkforward
> > test,
> >> > but ran into memory management problems. I could overcome
the
> >> > problems, but the results where disappointing. It only
saved
> > about
> >> > 20% of the time per run. I suspect that there is enough
overhead
> > in
> >> > setting up the stock arrays that avoiding some
calculations did
> > not
> >> > make much difference. Also, it becomes hard to track what
each
> >> > calculation depends on.
> >> >
> >> > It was interesting that many of the values could be
compressed
> > using
> >> > simple repeat coding. 200MB was enough the cache all of
the
> > values
> >> > that did not depend on any optimization parm and are re-
used many
> >> > times for each optimization pass.
> >> >
> >> > 5) Improvements to AmiBroker. The new optimizer was a big
help,
> > so
> >> > was the new support for QuickALF. These required no
changes to
> > the
> >> > code and had a large improvement and made walkforward
testing
> > much
> >> > more practical. The new optimizer allowed me to replace my
own
> >> > search code that I had hacked using AFL.
> >> >
> >> > 6) Multi core support - Sadly I have not found a practical
way to
> > put
> >> > the other cores on my system to work. To work with the new
> >> > optimization framework, it looks like this is best done
within
> > the
> >> > ami code itself.
> >> >
> >> > 7) Really exotic stuff - I played with the CUDA api a bit.
Very
> > cool
> >> > stuff. It allows of the 64 or so cores in the graphics
> > processor.
> >> > These are each able to perform one single precision
floating
> > point
> >> > operation per cycle. To work, I would have to preload the
> > database
> >> > into the graphics card memory and find a way to avoid any
per
> > symbol
> >> > overhead in amibroker. This would just shove the
> > buy/sell/buyprice,
> >> > etc arrays back to ami to process the trade list. If I
could
> > find a
> >> > way to avoid any overhead in setting up
open/high/low/close
> > arrays in
> >> > amibroker, I would do it.
> >> >
> >> > Fresh out of other ideas.
> >> >
> >> >
> >> >
> >> > ------------------------------------
> >> >
> >> > Please note that this group is for discussion between
users only.
> >> >
> >> > To get support from AmiBroker please send an e-mail
directly to
> >> > SUPPORT {at} amibroker.com
> >> >
> >> > For NEW RELEASE ANNOUNCEMENTS and other news always check
DEVLOG:
> >> > http://www.amibroker.com/devlog/
> >> >
> >> > For other support material please check also:
> >> > http://www.amibroker.com/support.html
> >> > Yahoo! Groups Links
> >> >
> >> >
> >> >
> >>
> >
> >
> >
> > ------------------------------------
> >
> > Please note that this group is for discussion between users
only.
> >
> > To get support from AmiBroker please send an e-mail directly
to
> > SUPPORT {at} amibroker.com
> >
> > For NEW RELEASE ANNOUNCEMENTS and other news always check
DEVLOG:
> > http://www.amibroker.com/devlog/
> >
> > For other support material please check also:
> > http://www.amibroker.com/support.html
> > Yahoo! Groups Links
> >
> >
> >
>
------------------------------------
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|