|
PureBytes Links
Trading Reference Links
|
Hello,
It is perfectly valid question.
First it does not really matter if the process goes through both ends or
sequentially but one core goes though odd and another through even steps,
and at first look it seems like this would give significant speed up.
BUT... in real world things are more ugly that in theory.
I did lots of testing and profiling (measuring time of execution of code on function-level),
and dual thread execution on dual core processor is faster if and only if
each core can execute accessing data only from its own on-chip data cache.
This is unfortunatelly NOT the case for backtesting/optimization.
On-chip caches are usually limited to well below 1MB. Almost every backtest
requires way more than 1MB.
Now what happens if you run code that uses more memory -
BOTH cores need to access on-board (regular) RAM. Both cores do this
through single memory interface that is SHARED between cores and
access one memory that runs at fixed speed (no matter if 1 or 8 cores access
the memory - it can not respond quicker than factory limit and one core is
fast enough to actually need to WAIT for memory).
Now if you run on 2 or more cores, they have to wait for the same, single shared memory
that runs at constant pace, slow enough for one core, not to mention more.
Net result is that if you actually try to run something that needs more than 1MB
of data and does not fit into individual data cache, the performance drops down
to actually single-core. What's more it can run slower because of additional overhead with
thread management.
And it is not imagination or theory. I did actual code profiling and I was surprised to when I tested multi-threaded
code. It works upto 2x faster, on dual core BUT ONLY IF you don't access more than
the size of on-chip per-core data cache. Or your code needs way more calculation than memory access.
If your code does a LOT of memory access (more than 1MB) and does it QUICKLY
(backtesting is extremely memory intensive and AFL scans through mem like crazy)
all advantages of running in multiple cores are gone.
BTW: what I did in this upgrade to speed up the backtest/optimization was to reduce the COUNT
of memory accesses to absolute minimum required. As it turns out even single CPU core was waiting for memory.
Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: "tipequity" <tagroups@xxxxxxxxxxxxx>
To: <amibroker@xxxxxxxxxxxxxxx>
Sent: Friday, October 05, 2007 4:20 AM
Subject: [amibroker] Re: Optimization speed increase in 5.01
> Tomasz, at the risk of sounding stupid, I am gonna run this idea by
> you. Since AB during backtest and optimizations work on a list of
> stocks why not have one cpu (dual core CPUs) work on symbols from top
> of the list and another cpu to work on symbols from the bottom of the
> list. Like buring candles from both end.
>
> Regards
>
> Kam
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@xxx>
> wrote:
>>
>> Hello,
>>
>> If you are running optimizations using new version I would love to
> hear about the timings you get
>> compared with old one.
>> Note that optimization with new version may run even 2 times faster
> (or more),
>> but actual speed increase depends how complex the formula is and
> how often system
>> trades and how large baskets. Speed increases are larger with
> simpler formulas,
>> because AFL execution speed did NOT change. The only things that
> has changed
>> is collection of signals (1st backtest phase) and entire 2nd phase
> of backtest.
>> As it turns out, when backtesting very simple formulas the AFL code
> execution is only less
>> than 20% of total time, the rest is collecting signals and sorting
> them
>> according to score and 2nd phase of the backtest (actual trading
> simulation).
>> These latter areas were the subject of performance tweaking.
>>
>> Best regards,
>> Tomasz Janeczko
>> amibroker.com
>>
>
>
>
>
> Please note that this group is for discussion between users only.
>
> To get support from AmiBroker please send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> For other support material please check also:
> http://www.amibroker.com/support.html
>
> Yahoo! Groups Links
>
>
>
>
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|