|
PureBytes Links
Trading Reference Links
|
Here are some links that I found interesting - general interest at
the top (3) and programmers links at the bottom (3):
http://www.gotw.ca/publications/concurrency-ddj.htm
http://zone.ni.com/devzone/cda/tut/p/id/6424
http://www.pcworld.com/article/id,143669-pg,1/article.html
http://www.multicore-association.org/press/news.php
http://xrl.us/immkg
http://xrl.us/immxz
brian_z
--- In amibroker@xxxxxxxxxxxxxxx, "brian_z111" <brian_z111@xxx> wrote:
>
> > As far as backward compatibility is concerned . I don't think
this
> >is an
> > issue . It is relatively simple to detect how many processors are
> >available
> > and make use of more than one if it exists.
>
> Developers don't seem to be fenced in when it comes to strategies
for
> utilizing MCP capabilities - they can take a distributed code
> approach (as you have done in MCO) or they can take a distributed
> load approach (at the level of the OS) or something in between.
>
> If they go all the way with the distributed code approach then
> possibly the capabilities of different chipsets don't equate at the
> lowest level e.g. threading versus fibers?
>
> The overheads of code distribution would be high and for some
> programs the payback wouldn't be there e.g. CAD involves a lot of
> data crunching and demands a distributed approach? On the other
hand
> AB is a different kind of program altogether and I don't see that
> Tomasz can use a distributed code approach, maintain it on into the
> future and keep prices around where they are (CAD doesn't come at
AB
> prices).
>
> Another software architecture could be a better approach for AB -
you
> can still have lightning fast otpimization on the side but under
that
> scenario AB would make a quantum leap in overall capabilities
rather
> than a quantum leap solely in backtesting/otpimization speed.
>
> If AB did go down that path then the software architecture could be
a
> radical depature from what has gone before and therefore the cost
of
> maintaining a parallel version for uniprocessors would, IMO, be too
> high (in some cases I think that what has been pulled apart for
> distribution to multicores doesn't necessarily come back together
so
> readily, or efficiently, for single core processing).
>
> If I was Tomasz I would be thinking about a radical departure in AB
> architecture and also making the quantum leap to an AB MCP version
> (leave the uniprocessor versions behind completely with support for
> uniprocessing versions ending pretty quickly - somewhere between
> immediately and 2-3 years).
>
> The big advantage of a load distributed model is that it can be
done
> piecemeal (to some extent) - it isn't an all or nothing, in one big
> bang, scenario
>
>
> Speculating on a load distributed model:
>
> - when AB starts it 'polls' the OS for system status/CPU
> availability/useage
> - an AB task scheduler 'knows' what the load requirements are for
> each of its distributable loads
> - (optionally) it prompts the user with info about CPU consumption
> and calls for input on AB priorities
> - AB then distributes its load to 'spare' CPU capacity
> - AB would need to make conservative decisions about CPU
consumption
> so as not to choke overall needs, as decided by the OS
> - the AB task scheduler would continue to monitor CPU load and make
> adjustments if core (OS) CPU consumption changed
> - the AB task scheduler could prompt users about CPU competition
and
> suggest alternatives for manual load shedding.
>
> Example 1:
>
> - AB is started (it is an 8 core machine)
> - no other programs are running (CPU consumption is low)
> - AB is not using any core features (scrolling charts, calculating
> indicators)
> - the user has an I/O plugin installed and wants to run optimization
> - AB, in conjunction with the OS, utilizes 7 cores for I/O (pretty
> fast?)
>
> (plugins could be a pretty effective way to manage AB processes as
> load modules?)
>
>
> Example 2:
>
> - a dual core machine
> - AB is running a backtest, using the 'spare' core, and the test is
> incomplete
> - an exchange opens and the user goes to live RT mode in AB and
> starts running a trading platform for the session
> - AB backtesting progress is preserved and the backtest is halted,
by
> the AB task scheduler, to free up the CPU's
> - when the trading session is over AB auto resumes the backtest
>
> Anyway, it appears Tomasz is someway down the track with MCP
> deployment so I dare say he has made his decisions already.
>
> Still, it is interesting to speculate on this subject.
>
> brian_z
>
>
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, Fred Tonetti <ftonetti@> wrote:
> >
> > Ohhh it certainly wasn't my idea to use sessions . It was Paul's
> offhand
> > comment about already doing it that brought light to the subject
> for me .
> >
> >
> >
> > If I had been aware enough to realize this could have been done I
> would have
> > done it years ago .
> >
> >
> >
> > In regards to having multithreading within individual programs .
> yes of
> > course . CAD/CAM programs have been doing this for years .
> >
> >
> >
> > In regards to doing the same thing in IO . Yes of course . MCO
was
> the proof
> > of concept if you will . The capability within IO already exists
> now in test
> > mode . I'm not sure what you mean by whether or not it might be a
> smart
> > thing to do but imho it certainly is useful as is MCO to speed up
> exhaustive
> > search.
> >
> >
> >
> > While you may not have the same chip . you will have the same
> underlying
> > instruction set .
> >
> >
> >
> > As far as backward compatibility is concerned . I don't think
this
> is an
> > issue . It is relatively simple to detect how many processors are
> available
> > and make use of more than one if it exists.
> >
> >
> >
> > I can't say that I see anything all that wonderful out of CMA-ES
> either but
> > that is of course just one person's opinion based on limited
> testing. The
> > problem with intelligent algorithms in general is that by and
large
> most are
> > not all that wonderful in their originally published version for
> the kinds
> > of tasks that they'd be put to in AB . Some have very limited
scope
> and
> > others have characteristics and limitations that make them of
> limited
> > usefulness as published . Thus many need to be played with to see
> what makes
> > them better and that playing takes oodles of time just to see if
> some
> > particular one is worthwhile pursuing .
> >
> > _____
> >
> > From: amibroker@xxxxxxxxxxxxxxx
[mailto:amibroker@xxxxxxxxxxxxxxx]
> On Behalf
> > Of brian_z111
> > Sent: Sunday, June 29, 2008 10:44 AM
> > To: amibroker@xxxxxxxxxxxxxxx
> > Subject: [amibroker] Re: Another side to the multi-core support
> argument
> >
> >
> >
> > You're not sorry - you enjoyed it ;-)
> >
> > Exactly as I imagined it - a perfect example of a single AB task
> that
> > lends itself perfectly to, what I would call, a distributed
> approach.
> >
> > I think that is about as simple as AB MCP could get since such
> > symmetry (repetition) of code would be rare, outside of opt?
> >
> > Carefully avoiding any further reading of docs, I was also
> referring
> > to the fact that you must have utilized the OS to 'force' the
ABnn
> > sessions (since we are in the accreditation mood I had better add
I
> > think someone else kicked in the idea to use 'sessions' but what
> the
> > hell?).
> >
> > You said that MCO was a pretty basic approach (or words to that
> > effect) but it seems to me that utilizing the capabilities of the
> OS
> > to distribute tasks to CPU resources, in various ways, can also
be
> > done from within a program without the need for sessions.
> >
> > As things stand at the moment couldn't you go ahead (if you
wanted
> > to) and write an I/O plugin that utilizes all available cores
WIWO
> > Tomasz changing anything in AB? (not to say it is a smart idea
but
> > possible, yes?)
> >
> > As a general comment - not particularly for you - what I am
kicking
> > around is the idea that it would be a heck of a task to isolate
the
> > bits of AB code that lend themselves to it, chop them up for
> > processing etc and then keep that up over the years (especially
> since
> > we won't all have the same OS/chip and/or number of cores) - I am
> > wondering of a less optimum but more achieveable path might be to
> use
> > the task scheduling capabilities of the OS to produce a more
> generic,
> > maintainable way to use all available cores.
> >
> > The other point I am throwing on the table is - why would any
> > developer maintain backward compatibility for uniprocessors when
as
> > of now no one is buyig one and realistically computers only have
a
> > life of a few years.
> >
> > brian_z
> >
> > brian_z
> >
> > FTR
> >
> > I am not convinced that CMA-ES is the ants pants of optimization
> but
> > I thought I would sit on the sidelines and let you two guys slug
it
> > out (of itself it might be nice and it could be a very good move
as
> > part of an overall, yet to be seen AB strategy but I would say it
> has
> > some limitations) .
> >
> > brian_z
> >
> > --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> ps.com, Fred
> > Tonetti <ftonetti@> wrote:
> > >
> > > Nope . Didn't do anything like that .
> > >
> > >
> > >
> > > What I did was simply as you state . where a sequence of code
> lends
> > itself
> > > to being chopped up, shoved through the processor and put back
> > together
> > > again on the other side .
> > >
> > >
> > >
> > > Even this description is not entirely accurate . What really
> > happens is that
> > > virtually the same AFL with different ranges of values for the
> > optimization
> > > statements is sent to multiple AB instances to be worked on by
> > different
> > > cores and then the results combined .
> > >
> > >
> > >
> > > From the doc ( Sorry to semi force you to read this )
> > >
> > >
> > >
> > > o MCO can deal with most forms of optimization statements.
> > MCO will
> > > check each of the Optimize statements in the AFL looking for
the
> > one that
> > > has a numeric Min, Max and Increment arguments and needs the
most
> > steps to
> > > get from the Min to Max value. Individual AFL's will be cloned
> for
> > each
> > > ABnn session with modifications to this Optimize statement so
> that
> > each core
> > > performs its share of the process. If for example the AFL to be
> > run had the
> > > following Optimize statements .
> > >
> > >
> > >
> > > Len1 = Optimize("Len1", 10, 1, 20, 1);
> > >
> > > Len2 = Optimize("Len2", 50, 1, 100, 1);
> > >
> > >
> > >
> > > . then it is the second Optimize statement that would be
altered
> > for each of
> > > the AFL's to be used by the ABnn sessions. If one were using
> three
> > cores
> > > without any /load command line arguments then the resulting
> Optimize
> > > statement for each of the three AFL's would look like what is
> below
> > > effectively splitting the workload as closely as possible
across
> > the user
> > > specified cores.
> > >
> > >
> > >
> > > Len2 = Optimize("Len2", 50, 1, 33, 1);
> > >
> > > Len2 = Optimize("Len2", 50, 34, 66, 1);
> > >
> > > Len2 = Optimize("Len2", 50, 67, 100, 1);
> > >
> > >
> > >
> > > _____
> > >
> > > From: amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> ps.com
> > [mailto:amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> ps.com]
> > On Behalf
> > > Of brian_z111
> > > Sent: Sunday, June 29, 2008 4:39 AM
> > > To: amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> ps.com
> > > Subject: [amibroker] Re: Another side to the multi-core support
> > argument
> > >
> > >
> > >
> > > Sorry, I used the term 'symmetric processing' incorrectly (I
now
> > see
> > > that it has a formal definition whereas I used it informally).
> > >
> > > What I meant was that I don't think software applications for
MCP
> > are
> > > limited only to single tasks where a sequence of code lends
> itself
> > to
> > > being chopped up, shoved through the processor and put back
> > together
> > > again on the other side.
> > >
> > > Couldn't software modularise its tasks/features and present
them
> to
> > > the OS as if they are separate programs e.g. downloading RT
data,
> > in
> > > AB, and saving to a local database, could be allocated one
core,
> by
> > > the OS, and rendering that data into a chart/processing
> indicators
> > > allocated to another - possibly AB could use some smart
> > > prioritisation of its own tasks and work in tandem with the OS
to
> > > prioritise access to core CPU's?
> > >
> > > It seems to me we are limiting ourselves with regard to the
> > > possibilities.
> > >
> > > (I haven't read, or tried, Freds MCO stuff - maybe that is what
> he
> > > did there already, in a basic way).
> > >
> > > If and when AB goes ahead with multicore application I have to
> > wonder
> > > why anyone wouldn't want to upgrade - could that be the time to
> end
> > > the unlimited backward compatibility rule (the benefits of
> > > maintaining backward compatibility for a few would no longer be
> > worth
> > > the effort and even detrimental to the many)?
> > >
> > > brian_z
> > >
> > > --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> > ps.com,
> > > "brian_z111" <brian_z111@> wrote:
> > > >
> > > > Re: what can traders do with multicore processing (MCP)?.
> > > >
> > > > Gee, where are all the Codesters when you need one?
> > > >
> > > > Are you guys holding out on us?
> > > >
> > > > MCP applications aren't limited to symmetric processing are
> they?
> > > >
> > > > http://en.wikipedia
> > <http://en.wikipedia
> <http://en.wikipedia.org/wiki/Computer_multitasking>
> > .org/wiki/Computer_multitasking>
> > > .org/wiki/Computer_multitasking
> > > >
> > > > http://en.wikipedia <http://en.wikipedia
> > <http://en.wikipedia.org/wiki/Multi-core> .org/wiki/Multi-core>
> > > .org/wiki/Multi-core
> > > >
> > > > "The amount of performance gained by the use of a multicore
> > > processor
> > > > depends on the problem being solved and the algorithms used,
as
> > > well
> > > > as their implementation in software (Amdahl's law). For so-
> > > > called "embarrassingly parallel" problems, a dual-core
> processor
> > > with
> > > > two cores at 2GHz may perform very nearly as fast as a single
> > core
> > > of
> > > > 4GHz.[1] Other problems though may not yield so much speedup.
> > This
> > > > all assumes however that the software has been designed to
take
> > > > advantage of available parallelism. If it hasn't, there will
> not
> > be
> > > > any speedup at all. However, the processor will multitask
> better
> > > > since it can run two programs at once, one on each core."
> > > >
> > > >
> > > > To what extent does it depend on the hardware/OS e.g. are the
> > > > possibilties for AB different on a MSFT system with an AMD
chip
> > > > compared to another OS with a different chip - if so how
would
> AB
> > > > handle that across a wide range of users - it seems to me the
> AB
> > > > solution will have to be generic rather than specialist?
> > > >
> > > >
> > > > As a layperson, it appears that multi-tasking is a more
> exciting
> > > > prospect, for the majority of traders, than optimising at
warp
> > > speeds.
> > > >
> > > > Can't muliple processors handle multi-tasking at speeds
unheard
> > of
> > > > before and hence bring some things into the realm of
> > possibilities?
> > > >
> > > >
> > > > How about:
> > > >
> > > > a) different databases running in one instance of AB - maybe
> even
> > > > with the databases in different timeframes/formats/coming
from
> > > > different providers and being cross referenced in indicators.
> > > >
> > > > Can MCP allow users,who now run more than one installation of
> AB
> > on
> > > a
> > > > single computer (for whatever reason) to get the same results
> > from
> > > > one installation.
> > > >
> > > >
> > > > b) in the past Tomasz would enable AB's native database for
> extra
> > > > data (historical fields other than OHLCVOI) - as I understand
> it
> > > > because it would weigh AB down with things not used by the
> > majority.
> > > >
> > > > Perhaps MCP will make the weight of an open structure AB
> database
> > a
> > > > featherweight (PremiumData has already expressed an interest
in
> > > > synching their database with AB so it is not as if there is
no
> > > demand
> > > > for this feature).
> > > >
> > > > c) a task scheduler built into AB - set the tasks at the
start
> of
> > > the
> > > > day - if the keyboard/processor is idle AB starts to run the
> > tasks
> > > > (backtesting/optimising).
> > > >
> > > > d) if we want to run our trading platform and AB on real time
> > with
> > > > two monitors on one computer do we have enough grunt to do
that
> > now
> > > > OR would MCP allow us to run processing intensive tasks like
> that
> > > > concurrently OR maybe with a minimised AB chart (on tick
data)
> > > > floating on top of our trading platform window.
> > > >
> > > > I don't know - I am merely speculating!
> > > >
> > > > Surely someone out there has some better ideas?
> > > >
> > > > brian_z
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --- In amibroker@xxxxxxxxx <mailto:amibroker%
40yahoogroups.com>
> > ps.com,
> > > "sidhartha70" <sidhartha70@>
> > > > wrote:
> > > > >
> > > > > Hi Tomasz,
> > > > >
> > > > > I've been reading the arguments for and against multi-core
> > support
> > > > > with interest, and I just wanted to add my own voice to it
> and
> > > > mention
> > > > > an angle which hasn't currently been focused on.
> > > > >
> > > > > Since I bought AmiBroker a couple of months back I've been
> > blown
> > > > away
> > > > > by not only what a great piece of software it is, but also
by
> > it's
> > > > > value and the fantastic customer support. Your own
> > responsiveness
> > > to
> > > > > customer needs and ability to efficiently add new features
> > leaves
> > > me
> > > > > with no hesitations in recommending AmiBroker to my fellow
> > > traders.
> > > > >
> > > > > However, I use AB in two ways. First for swing trading...
for
> > > which
> > > > it
> > > > > is not only awesome, but also works perfectly happily on
one
> > core
> > > > > because of the lower overhead in terms of data through put
and
> > > > > therefore calculation overhead on that data.
> > > > >
> > > > > I also use AB for higher frequency day trading... and it's
> here
> > > for
> > > > me
> > > > > that multi-core support could really add something. For
> obvious
> > > > > reasons the multi-core argument has so far been almost
> > exclusively
> > > > > focused on optimization. However, for higher frequency
> traders
> > > like
> > > > > myself, working off just one core can be a significant
> > bottleneck.
> > > > >
> > > > > I have an IQ Feed 'tick' database... and some quite complex
> > chart
> > > > > setup's & indictaors. Currently, with chart refresh
interval
> > set
> > > to
> > > > 2
> > > > > seconds (I'd like it quikcer) I can see one core of my
> computer
> > > > > working away at about 50%. However, if I increase the
update
> > > > interval
> > > > > to zero to bring in every tick, I see useage on one core go
> up
> > to
> > > > > 80-90%. At this point, you can imagine, AB becomes
noticeably
> > > clunky
> > > > > and difficult to use. Things get even worse if I start to
> think
> > > > about
> > > > > symbol linking. (for reference I am only looking at a
couple
> of
> > e-
> > > > mini
> > > > > symbols)
> > > > > The frustrating aspect is that I have another 7 x 3Ghz
cores
> > > sitting
> > > > > there twiddling their fingers while AB is grinding up on
one
> > core.
> > > > >
> > > > > So, I do think, for those of us who use AB for higher
frequncy
> > > > > trading, there is a very good argument to consider adding
> multi-
> > > core
> > > > > support outside of the optimization argument.
> > > > >
> > > > > Love to hear your (or anyone's!) thoughts on this.
> > > > >
> > > > > Many Thanks
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > > _____
> > >
> > > I am using the free version of SPAMfighter for private users.
> > > It has removed 492 spam emails to date.
> > > Paying users do not have this message in their emails.
> > > Try SPAMfighter <http://www.spamfigh
> <http://www.spamfighter.com/len>
> > ter.com/len> for free now!
> > >
> >
> >
> >
> >
> > _____
> >
> > I am using the free version of SPAMfighter for private users.
> > It has removed 492 spam emails to date.
> > Paying users do not have this message in their emails.
> > Try SPAMfighter <http://www.spamfighter.com/len> for free now!
> >
>
------------------------------------
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|