|
PureBytes Links
Trading Reference Links
|
Brian,
Just as an aside, Apple announced that the next version of OS X "Snow
Leopard" due in about a year would not have any new UI features as a
goal. The goal of the release will be to effectively support multiple
processors and release the support tools for developers that can make
it a lot easier for utilizing multiple processors. You can bet that
if Apple is making a statement like that, they will be making industry
leading changes to the way things work.
I have a quad 2.8GHz machine right now. I opted to special order a
"downgrade" of their standard 8 core machine (and save $500) because I
did not see that I could get any value added with the extra 4
processors due to the current state of multi-core application
support. I run OS X, AB on a Windows XP VM, A very heavy Java app
from my broker, Safari browser, and a few utilities. I only keep 2
cores busy on average --though it distributes the load around to all
4. AB keeps one core busy 100%. I figure that by the time the
software can really use the extra cores, it will be time for a faster
machine with larger built-in caches anyway. Then I will upgrade to 8
or more cores --likely for the same price I paid for 4 this time.
BR,
Dennis
On Jun 29, 2008, at 10:23 PM, brian_z111 wrote:
>> As far as backward compatibility is concerned . I don't think this
>> is an
>> issue . It is relatively simple to detect how many processors are
>> available
>> and make use of more than one if it exists.
>
> Developers don't seem to be fenced in when it comes to strategies for
> utilizing MCP capabilities - they can take a distributed code
> approach (as you have done in MCO) or they can take a distributed
> load approach (at the level of the OS) or something in between.
>
> If they go all the way with the distributed code approach then
> possibly the capabilities of different chipsets don't equate at the
> lowest level e.g. threading versus fibers?
>
> The overheads of code distribution would be high and for some
> programs the payback wouldn't be there e.g. CAD involves a lot of
> data crunching and demands a distributed approach? On the other hand
> AB is a different kind of program altogether and I don't see that
> Tomasz can use a distributed code approach, maintain it on into the
> future and keep prices around where they are (CAD doesn't come at AB
> prices).
>
> Another software architecture could be a better approach for AB - you
> can still have lightning fast otpimization on the side but under that
> scenario AB would make a quantum leap in overall capabilities rather
> than a quantum leap solely in backtesting/otpimization speed.
>
> If AB did go down that path then the software architecture could be a
> radical depature from what has gone before and therefore the cost of
> maintaining a parallel version for uniprocessors would, IMO, be too
> high (in some cases I think that what has been pulled apart for
> distribution to multicores doesn't necessarily come back together so
> readily, or efficiently, for single core processing).
>
> If I was Tomasz I would be thinking about a radical departure in AB
> architecture and also making the quantum leap to an AB MCP version
> (leave the uniprocessor versions behind completely with support for
> uniprocessing versions ending pretty quickly - somewhere between
> immediately and 2-3 years).
>
> The big advantage of a load distributed model is that it can be done
> piecemeal (to some extent) - it isn't an all or nothing, in one big
> bang, scenario
>
>
> Speculating on a load distributed model:
>
> - when AB starts it 'polls' the OS for system status/CPU
> availability/useage
> - an AB task scheduler 'knows' what the load requirements are for
> each of its distributable loads
> - (optionally) it prompts the user with info about CPU consumption
> and calls for input on AB priorities
> - AB then distributes its load to 'spare' CPU capacity
> - AB would need to make conservative decisions about CPU consumption
> so as not to choke overall needs, as decided by the OS
> - the AB task scheduler would continue to monitor CPU load and make
> adjustments if core (OS) CPU consumption changed
> - the AB task scheduler could prompt users about CPU competition and
> suggest alternatives for manual load shedding.
>
> Example 1:
>
> - AB is started (it is an 8 core machine)
> - no other programs are running (CPU consumption is low)
> - AB is not using any core features (scrolling charts, calculating
> indicators)
> - the user has an I/O plugin installed and wants to run optimization
> - AB, in conjunction with the OS, utilizes 7 cores for I/O (pretty
> fast?)
>
> (plugins could be a pretty effective way to manage AB processes as
> load modules?)
>
>
> Example 2:
>
> - a dual core machine
> - AB is running a backtest, using the 'spare' core, and the test is
> incomplete
> - an exchange opens and the user goes to live RT mode in AB and
> starts running a trading platform for the session
> - AB backtesting progress is preserved and the backtest is halted, by
> the AB task scheduler, to free up the CPU's
> - when the trading session is over AB auto resumes the backtest
>
> Anyway, it appears Tomasz is someway down the track with MCP
> deployment so I dare say he has made his decisions already.
>
> Still, it is interesting to speculate on this subject.
>
> brian_z
>
>
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, Fred Tonetti <ftonetti@xxx> wrote:
>>
>> Ohhh it certainly wasn't my idea to use sessions . It was Paul's
> offhand
>> comment about already doing it that brought light to the subject
> for me .
>>
>>
>>
>> If I had been aware enough to realize this could have been done I
> would have
>> done it years ago .
>>
>>
>>
>> In regards to having multithreading within individual programs .
> yes of
>> course . CAD/CAM programs have been doing this for years .
>>
>>
>>
>> In regards to doing the same thing in IO . Yes of course . MCO was
> the proof
>> of concept if you will . The capability within IO already exists
> now in test
>> mode . I'm not sure what you mean by whether or not it might be a
> smart
>> thing to do but imho it certainly is useful as is MCO to speed up
> exhaustive
>> search.
>>
>>
>>
>> While you may not have the same chip . you will have the same
> underlying
>> instruction set .
>>
>>
>>
>> As far as backward compatibility is concerned . I don't think this
> is an
>> issue . It is relatively simple to detect how many processors are
> available
>> and make use of more than one if it exists.
>>
>>
>>
>> I can't say that I see anything all that wonderful out of CMA-ES
> either but
>> that is of course just one person's opinion based on limited
> testing. The
>> problem with intelligent algorithms in general is that by and large
> most are
>> not all that wonderful in their originally published version for
> the kinds
>> of tasks that they'd be put to in AB . Some have very limited scope
> and
>> others have characteristics and limitations that make them of
> limited
>> usefulness as published . Thus many need to be played with to see
> what makes
>> them better and that playing takes oodles of time just to see if
> some
>> particular one is worthwhile pursuing .
>>
>> _____
>>
>> From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx]
> On Behalf
>> Of brian_z111
>> Sent: Sunday, June 29, 2008 10:44 AM
>> To: amibroker@xxxxxxxxxxxxxxx
>> Subject: [amibroker] Re: Another side to the multi-core support
> argument
>>
>>
>>
>> You're not sorry - you enjoyed it ;-)
>>
>> Exactly as I imagined it - a perfect example of a single AB task
> that
>> lends itself perfectly to, what I would call, a distributed
> approach.
>>
>> I think that is about as simple as AB MCP could get since such
>> symmetry (repetition) of code would be rare, outside of opt?
>>
>> Carefully avoiding any further reading of docs, I was also
> referring
>> to the fact that you must have utilized the OS to 'force' the ABnn
>> sessions (since we are in the accreditation mood I had better add I
>> think someone else kicked in the idea to use 'sessions' but what
> the
>> hell?).
>>
>> You said that MCO was a pretty basic approach (or words to that
>> effect) but it seems to me that utilizing the capabilities of the
> OS
>> to distribute tasks to CPU resources, in various ways, can also be
>> done from within a program without the need for sessions.
>>
>> As things stand at the moment couldn't you go ahead (if you wanted
>> to) and write an I/O plugin that utilizes all available cores WIWO
>> Tomasz changing anything in AB? (not to say it is a smart idea but
>> possible, yes?)
>>
>> As a general comment - not particularly for you - what I am kicking
>> around is the idea that it would be a heck of a task to isolate the
>> bits of AB code that lend themselves to it, chop them up for
>> processing etc and then keep that up over the years (especially
> since
>> we won't all have the same OS/chip and/or number of cores) - I am
>> wondering of a less optimum but more achieveable path might be to
> use
>> the task scheduling capabilities of the OS to produce a more
> generic,
>> maintainable way to use all available cores.
>>
>> The other point I am throwing on the table is - why would any
>> developer maintain backward compatibility for uniprocessors when as
>> of now no one is buyig one and realistically computers only have a
>> life of a few years.
>>
>> brian_z
>>
>> brian_z
>>
>> FTR
>>
>> I am not convinced that CMA-ES is the ants pants of optimization
> but
>> I thought I would sit on the sidelines and let you two guys slug it
>> out (of itself it might be nice and it could be a very good move as
>> part of an overall, yet to be seen AB strategy but I would say it
> has
>> some limitations) .
>>
>> brian_z
>>
>> --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> ps.com, Fred
>> Tonetti <ftonetti@> wrote:
>>>
>>> Nope . Didn't do anything like that .
>>>
>>>
>>>
>>> What I did was simply as you state . where a sequence of code
> lends
>> itself
>>> to being chopped up, shoved through the processor and put back
>> together
>>> again on the other side .
>>>
>>>
>>>
>>> Even this description is not entirely accurate . What really
>> happens is that
>>> virtually the same AFL with different ranges of values for the
>> optimization
>>> statements is sent to multiple AB instances to be worked on by
>> different
>>> cores and then the results combined .
>>>
>>>
>>>
>>> From the doc ( Sorry to semi force you to read this )
>>>
>>>
>>>
>>> o MCO can deal with most forms of optimization statements.
>> MCO will
>>> check each of the Optimize statements in the AFL looking for the
>> one that
>>> has a numeric Min, Max and Increment arguments and needs the most
>> steps to
>>> get from the Min to Max value. Individual AFL's will be cloned
> for
>> each
>>> ABnn session with modifications to this Optimize statement so
> that
>> each core
>>> performs its share of the process. If for example the AFL to be
>> run had the
>>> following Optimize statements .
>>>
>>>
>>>
>>> Len1 = Optimize("Len1", 10, 1, 20, 1);
>>>
>>> Len2 = Optimize("Len2", 50, 1, 100, 1);
>>>
>>>
>>>
>>> . then it is the second Optimize statement that would be altered
>> for each of
>>> the AFL's to be used by the ABnn sessions. If one were using
> three
>> cores
>>> without any /load command line arguments then the resulting
> Optimize
>>> statement for each of the three AFL's would look like what is
> below
>>> effectively splitting the workload as closely as possible across
>> the user
>>> specified cores.
>>>
>>>
>>>
>>> Len2 = Optimize("Len2", 50, 1, 33, 1);
>>>
>>> Len2 = Optimize("Len2", 50, 34, 66, 1);
>>>
>>> Len2 = Optimize("Len2", 50, 67, 100, 1);
>>>
>>>
>>>
>>> _____
>>>
>>> From: amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> ps.com
>> [mailto:amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> ps.com]
>> On Behalf
>>> Of brian_z111
>>> Sent: Sunday, June 29, 2008 4:39 AM
>>> To: amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> ps.com
>>> Subject: [amibroker] Re: Another side to the multi-core support
>> argument
>>>
>>>
>>>
>>> Sorry, I used the term 'symmetric processing' incorrectly (I now
>> see
>>> that it has a formal definition whereas I used it informally).
>>>
>>> What I meant was that I don't think software applications for MCP
>> are
>>> limited only to single tasks where a sequence of code lends
> itself
>> to
>>> being chopped up, shoved through the processor and put back
>> together
>>> again on the other side.
>>>
>>> Couldn't software modularise its tasks/features and present them
> to
>>> the OS as if they are separate programs e.g. downloading RT data,
>> in
>>> AB, and saving to a local database, could be allocated one core,
> by
>>> the OS, and rendering that data into a chart/processing
> indicators
>>> allocated to another - possibly AB could use some smart
>>> prioritisation of its own tasks and work in tandem with the OS to
>>> prioritise access to core CPU's?
>>>
>>> It seems to me we are limiting ourselves with regard to the
>>> possibilities.
>>>
>>> (I haven't read, or tried, Freds MCO stuff - maybe that is what
> he
>>> did there already, in a basic way).
>>>
>>> If and when AB goes ahead with multicore application I have to
>> wonder
>>> why anyone wouldn't want to upgrade - could that be the time to
> end
>>> the unlimited backward compatibility rule (the benefits of
>>> maintaining backward compatibility for a few would no longer be
>> worth
>>> the effort and even detrimental to the many)?
>>>
>>> brian_z
>>>
>>> --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
>> ps.com,
>>> "brian_z111" <brian_z111@> wrote:
>>>>
>>>> Re: what can traders do with multicore processing (MCP)?.
>>>>
>>>> Gee, where are all the Codesters when you need one?
>>>>
>>>> Are you guys holding out on us?
>>>>
>>>> MCP applications aren't limited to symmetric processing are
> they?
>>>>
>>>> http://en.wikipedia
>> <http://en.wikipedia
> <http://en.wikipedia.org/wiki/Computer_multitasking>
>> .org/wiki/Computer_multitasking>
>>> .org/wiki/Computer_multitasking
>>>>
>>>> http://en.wikipedia <http://en.wikipedia
>> <http://en.wikipedia.org/wiki/Multi-core> .org/wiki/Multi-core>
>>> .org/wiki/Multi-core
>>>>
>>>> "The amount of performance gained by the use of a multicore
>>> processor
>>>> depends on the problem being solved and the algorithms used, as
>>> well
>>>> as their implementation in software (Amdahl's law). For so-
>>>> called "embarrassingly parallel" problems, a dual-core
> processor
>>> with
>>>> two cores at 2GHz may perform very nearly as fast as a single
>> core
>>> of
>>>> 4GHz.[1] Other problems though may not yield so much speedup.
>> This
>>>> all assumes however that the software has been designed to take
>>>> advantage of available parallelism. If it hasn't, there will
> not
>> be
>>>> any speedup at all. However, the processor will multitask
> better
>>>> since it can run two programs at once, one on each core."
>>>>
>>>>
>>>> To what extent does it depend on the hardware/OS e.g. are the
>>>> possibilties for AB different on a MSFT system with an AMD chip
>>>> compared to another OS with a different chip - if so how would
> AB
>>>> handle that across a wide range of users - it seems to me the
> AB
>>>> solution will have to be generic rather than specialist?
>>>>
>>>>
>>>> As a layperson, it appears that multi-tasking is a more
> exciting
>>>> prospect, for the majority of traders, than optimising at warp
>>> speeds.
>>>>
>>>> Can't muliple processors handle multi-tasking at speeds unheard
>> of
>>>> before and hence bring some things into the realm of
>> possibilities?
>>>>
>>>>
>>>> How about:
>>>>
>>>> a) different databases running in one instance of AB - maybe
> even
>>>> with the databases in different timeframes/formats/coming from
>>>> different providers and being cross referenced in indicators.
>>>>
>>>> Can MCP allow users,who now run more than one installation of
> AB
>> on
>>> a
>>>> single computer (for whatever reason) to get the same results
>> from
>>>> one installation.
>>>>
>>>>
>>>> b) in the past Tomasz would enable AB's native database for
> extra
>>>> data (historical fields other than OHLCVOI) - as I understand
> it
>>>> because it would weigh AB down with things not used by the
>> majority.
>>>>
>>>> Perhaps MCP will make the weight of an open structure AB
> database
>> a
>>>> featherweight (PremiumData has already expressed an interest in
>>>> synching their database with AB so it is not as if there is no
>>> demand
>>>> for this feature).
>>>>
>>>> c) a task scheduler built into AB - set the tasks at the start
> of
>>> the
>>>> day - if the keyboard/processor is idle AB starts to run the
>> tasks
>>>> (backtesting/optimising).
>>>>
>>>> d) if we want to run our trading platform and AB on real time
>> with
>>>> two monitors on one computer do we have enough grunt to do that
>> now
>>>> OR would MCP allow us to run processing intensive tasks like
> that
>>>> concurrently OR maybe with a minimised AB chart (on tick data)
>>>> floating on top of our trading platform window.
>>>>
>>>> I don't know - I am merely speculating!
>>>>
>>>> Surely someone out there has some better ideas?
>>>>
>>>> brian_z
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
>> ps.com,
>>> "sidhartha70" <sidhartha70@>
>>>> wrote:
>>>>>
>>>>> Hi Tomasz,
>>>>>
>>>>> I've been reading the arguments for and against multi-core
>> support
>>>>> with interest, and I just wanted to add my own voice to it
> and
>>>> mention
>>>>> an angle which hasn't currently been focused on.
>>>>>
>>>>> Since I bought AmiBroker a couple of months back I've been
>> blown
>>>> away
>>>>> by not only what a great piece of software it is, but also by
>> it's
>>>>> value and the fantastic customer support. Your own
>> responsiveness
>>> to
>>>>> customer needs and ability to efficiently add new features
>> leaves
>>> me
>>>>> with no hesitations in recommending AmiBroker to my fellow
>>> traders.
>>>>>
>>>>> However, I use AB in two ways. First for swing trading... for
>>> which
>>>> it
>>>>> is not only awesome, but also works perfectly happily on one
>> core
>>>>> because of the lower overhead in terms of data through put and
>>>>> therefore calculation overhead on that data.
>>>>>
>>>>> I also use AB for higher frequency day trading... and it's
> here
>>> for
>>>> me
>>>>> that multi-core support could really add something. For
> obvious
>>>>> reasons the multi-core argument has so far been almost
>> exclusively
>>>>> focused on optimization. However, for higher frequency
> traders
>>> like
>>>>> myself, working off just one core can be a significant
>> bottleneck.
>>>>>
>>>>> I have an IQ Feed 'tick' database... and some quite complex
>> chart
>>>>> setup's & indictaors. Currently, with chart refresh interval
>> set
>>> to
>>>> 2
>>>>> seconds (I'd like it quikcer) I can see one core of my
> computer
>>>>> working away at about 50%. However, if I increase the update
>>>> interval
>>>>> to zero to bring in every tick, I see useage on one core go
> up
>> to
>>>>> 80-90%. At this point, you can imagine, AB becomes noticeably
>>> clunky
>>>>> and difficult to use. Things get even worse if I start to
> think
>>>> about
>>>>> symbol linking. (for reference I am only looking at a couple
> of
>> e-
>>>> mini
>>>>> symbols)
>>>>> The frustrating aspect is that I have another 7 x 3Ghz cores
>>> sitting
>>>>> there twiddling their fingers while AB is grinding up on one
>> core.
>>>>>
>>>>> So, I do think, for those of us who use AB for higher frequncy
>>>>> trading, there is a very good argument to consider adding
> multi-
>>> core
>>>>> support outside of the optimization argument.
>>>>>
>>>>> Love to hear your (or anyone's!) thoughts on this.
>>>>>
>>>>> Many Thanks
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>> _____
>>>
>>> I am using the free version of SPAMfighter for private users.
>>> It has removed 492 spam emails to date.
>>> Paying users do not have this message in their emails.
>>> Try SPAMfighter <http://www.spamfigh
> <http://www.spamfighter.com/len>
>> ter.com/len> for free now!
>>>
>>
>>
>>
>>
>> _____
>>
>> I am using the free version of SPAMfighter for private users.
>> It has removed 492 spam emails to date.
>> Paying users do not have this message in their emails.
>> Try SPAMfighter <http://www.spamfighter.com/len> for free now!
>>
>
>
>
> ------------------------------------
>
> Please note that this group is for discussion between users only.
>
> To get support from AmiBroker please send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> For other support material please check also:
> http://www.amibroker.com/support.html
> Yahoo! Groups Links
>
>
>
------------------------------------
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|