[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] Freakishly fast backtest using 64 cores



PureBytes Links

Trading Reference Links

Title: Re: [amibroker] Freakishly fast backtest using 64 cores
Yes, I'm puzzled too. 
 
Looping should depend upon the speed of the CPU (or so I thought).  Why would putting the data in a different memory store increase looping speed? 
 
I noticed that AB runs its second backtest much faster than the first -- I assumed because somehow it caches all of the files on the first run (although these are big files, and how it does that is a mystery to me).
 
In any case, completely fascinating!
 
cs
 
----- Original Message -----
From: Herman
Sent: Tuesday, August 05, 2008 5:09 AM
Subject: Re: [amibroker] Freakishly fast backtest using 64 cores

This is amazing... But, i am sorry to say, i don't understand what you are doing...


can you put this in layman's terms?


best regards,

herman



Monday, August 4, 2008, 7:18:45 PM, you wrote:


> Greetings,


> I ported part of my AFL backtest code to a plugin, that takes 

> advantage of the graphics math cores on the video card that are 

> normally used for 3d graphics.  


> I was able to get a several thousand fold performance improvement 

> over AFL code alone.


> My goal was to reduce the 25 seconds AFL code alone uses for a single 

> portfolio level back test to less than 1 second, allowing multi day 

> optimization and walkforward runs to complete in a more reasonable 

> time, and also just to see how fast I could get it to run.


> The backtest runs over 1 year of 5 minute bars for about 1000 

> symbols.  1 year of data normally takes 25 seconds for AmiBroker 

> alone, or 18 seconds for 6 months of data.  A typical optimization 

> run takes hundreds of these passes per walk forward step, taking 

> hours.


> Using the Nvidia CUDA API, running on my mid range video card.  It 

> was much faster.  Much, much, much faster.  How fast?


> It reduced the run time from 25s to... 4.4ms.  That is more than 

> 200/s!  


> I didnt believe the timing when I saw it at first.  So, I put 1,000 

> runs in a loop and sure enough, it ran 1,000 iterations in about 4 

> 1/2 seconds.  This far exceeded my gaol or expectations.


> The resulting trade list matches that obtained by the AFL version of 

> this code. 


> I estimate that it is processing 32GB of bar data/sec.

>   

> Getting this to work at peak performance was tricky.  Most of what I 

> have learned about code optimization does not apply.  


> It uses AmiBroker to load the symbol data and perform calculations 

> that do not depend on the optimization parameters.  Once loaded into 

> video memory, repeated passes can be made with different parameters, 

> avoiding any overhead. 


> For non backtest/optimization runs, the code just evaluates one 

> symbol and passes the data back to AmiBroker buy/sell/short/cover 

> arrays, making it easy to test, validate and visualize the trades.  

> There is very little performance gain in this case.  


> There are problems, however.  To run optimizations at peak speed, I 

> can not use AmiBroker to calculate the optimization goal function.  

> So, I am in the process of writing code to match signals and 

> calculate the portfolio fitness function.  Once I do this, I will be 

> able to perform full optimizations and walk forwards at 3 orders of 

> magnitude faster than is possible with AmiBroker alone.


> Also, this is not general purpose code.  Changing the system code 

> means changing a dll written in C.  However, there is no reason that 

> this could not be made more general. 


> I have made some prototypes of "Cuda" versions of basic AFL 

> functions.  The idea is to queue the function calls into a definition 

> executed by a micro kernel running on the graphics cores.  The result 

> would be the ability to use the full power of the graphics cores by 

> modifying AFL code to use Cuda aware versions with no changes to C 

> code. It would be an interesting, but big project.  




> ------------------------------------


> Please note that this group is for discussion between users only.


> To get support from AmiBroker please send an e-mail directly to 

> SUPPORT {at} amibroker.com


> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:

> http://www.amibroker.com/devlog/


> For other support material please check also:

> http://www.amibroker.com/support.html

> Yahoo! Groups Links


> <*> To visit your group on the web, go to:

>     http://groups.yahoo.com/group/amibroker/


> <*> Your email settings:

>     Individual Email | Traditional


> <*> To change settings online go to:

>     http://groups.yahoo.com/group/amibroker/join

>     (Yahoo! ID required)


> <*> To change settings via email:

>     mailto:amibroker-digest@xxxxxxxxxxxxxxx 

>     mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx


> <*> To unsubscribe from this group, send an email to:

>     amibroker-unsubscribe@xxxxxxxxxxxxxxx


> <*> Your use of Yahoo! Groups is subject to:

>     http://docs.yahoo.com/info/terms/



__._,_.___

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html




Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___