Re: [amibroker] Adventures in improving Ami Backtest speed, AmiBroker Email List Archive

----- Original Message -----
From: dloyer123
Date: Tuesday, July 15, 2008 12:59 pm
Subject: [amibroker] Adventures in improving Ami Backtest speed
To: amibroker@xxxxxxxxxxxxxxx

> Greetings
>
> I just wanted to share my results in improving AmiBroker
> backtest
> performance.
>
> I use 5 min intraday data over a 1,000 symbol database and
> Amibroker's new walkforward support. However each walkforward
> step
> takes a long time probably due to the large size of the data
> set.
>
> I have looked into ways to improve the run time performance.
> Here
> are some of the things I tried:
>
> 1) Buy new computer. This had the best results. Run time per
> pass
> dropped greatly. A new core 2 processor with the highest clock
> rate
> I could find was about twice as fast as my old laptop.
>
> 2) Use "Check AFL" to find functions that are slow and write
> them in
> C. I found that the DayOfWeek() function is very slow in
> comparison
> to other operations, so is Prec(). I moved these to C and
> reduced my
> run time a good bit. My DayOfWeek() function exploits the fact
> that
> each bar of a day is the same day of the week and the first bar
> of
> the day is probably the day after the last. It avoids finding
> the
> actual day on each bar. This saved more time than any of the
> other
> optimizations.
>
> 3) Rewrite the whole system in C. Sadly this only saved a about
> 10%,
> not enough to justify the risk of bugs and the trouble of
> maintaining
> the code. I went to no great lengths to optimize the code, but
> it
> goes to show how little overhead AFL adds and how highly
> optimized
> the AFL code is already.
>
> 4) Write a C route to cache results that do not depend on the
> value
> being optimized. Since many of the calculations dont change
> with
> each optimization step or only depend on a single optimization
> parameter, they dont need to be recalculated with each
> optimization
> step. I had high hopes for this approach and spent some time on
> it.
> I was able to get cache hit rates up over 99% on a walkforward
> test,
> but ran into memory management problems. I could overcome the
> problems, but the results where disappointing. It only saved
> about
> 20% of the time per run. I suspect that there is enough
> overhead in
> setting up the stock arrays that avoiding some calculations did
> not
> make much difference. Also, it becomes hard to track what each
> calculation depends on.
>
> It was interesting that many of the values could be compressed
> using
> simple repeat coding. 200MB was enough the cache all of the
> values
> that did not depend on any optimization parm and are re-used
> many
> times for each optimization pass.
>
> 5) Improvements to AmiBroker. The new optimizer was a big help,
> so
> was the new support for QuickALF. These required no changes to
> the
> code and had a large improvement and made walkforward testing
> much
> more practical. The new optimizer allowed me to replace my own
> search code that I had hacked using AFL.
>
> 6) Multi core support - Sadly I have not found a practical way
> to put
> the other cores on my system to work. To work with the new
> optimization framework, it looks like this is best done within
> the
> ami code itself.
>
> 7) Really exotic stuff - I played with the CUDA api a bit. Very
> cool
> stuff. It allows of the 64 or so cores in the graphics
> processor.
> These are each able to perform one single precision floating
> point
> operation per cycle. To work, I would have to preload the
> database
> into the graphics card memory and find a way to avoid any per
> symbol
> overhead in amibroker. This would just shove the
> buy/sell/buyprice,
> etc arrays back to ami to process the trade list. If I could
> find a
> way to avoid any overhead in setting up open/high/low/close
> arrays in
> amibroker, I would do it.
>
> Fresh out of other ideas.
>
>
>