|
PureBytes Links
Trading Reference Links
|
Hello,
Turn on QuickAFL in AA.
Other than that, there is very simple reason why more data leads to performance degradation:
every bar of your data is 40 bytes. Multiply number of bars x 40. You will get memory requirement for data alone
per symbol. Now compare this to CPU cache size. If your CPU has say 2MB on-chip L2 cache it can keep
on-chip only 2MB/40 = 50K bars. Anything more than 50000 bars does not fit onto on-chip cache.
Now since system RAM is 10 times slower than CPU cache, as soon as you reach the size of on-chip CPU
cache you will hit performance drop.
Next boundary is when your system runs out of physical free RAM and starts swaping. This is easily 3 orders of magnitude
slower than RAM.
For this simple reason - keep your databases *small* if you want speed.
Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: "davemabe2000" <davemabe@xxxxxxxxx>
To: <amibroker@xxxxxxxxxxxxxxx>
Sent: Thursday, July 30, 2009 2:07 PM
Subject: [amibroker] Maximizing Backtesting Performance on Large Data Sets
> When I create a database and insert 1.5 years of 5 minute bar data, the optimization I run takes around 2.5 hours.
>
> When I add more data to the database to end up with 8 years of 5 minute bar data, the same optimization takes 4 days+
> (exponentially worse).
>
> Even when I restrict the date range to a 1.5 year period in the larger database, a single backtest is noticeably longer than the
> same backtest with just the 1.5 years in the database.
>
> Is there anything that I can do to improve the performance of the optimization over the larger period?
>
> My plan now is to create a database for each 2 year period and run the optimizations on each individually.
>
> Any other ideas? Any tricks for getting more performance on a large data set?
>
>
>
> ------------------------------------
>
> **** IMPORTANT PLEASE READ ****
> This group is for the discussion between users only.
> This is *NOT* technical support channel.
>
> TO GET TECHNICAL SUPPORT send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> http://www.amibroker.com/feedback/
> (submissions sent via other channels won't be considered)
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> Yahoo! Groups Links
>
>
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|