|
PureBytes Links
Trading Reference Links
|
> As far as backward compatibility is concerned . I don't think this
>is an
> issue . It is relatively simple to detect how many processors are
>available
> and make use of more than one if it exists.
Developers don't seem to be fenced in when it comes to strategies for
utilizing MCP capabilities - they can take a distributed code
approach (as you have done in MCO) or they can take a distributed
load approach (at the level of the OS) or something in between.
If they go all the way with the distributed code approach then
possibly the capabilities of different chipsets don't equate at the
lowest level e.g. threading versus fibers?
The overheads of code distribution would be high and for some
programs the payback wouldn't be there e.g. CAD involves a lot of
data crunching and demands a distributed approach? On the other hand
AB is a different kind of program altogether and I don't see that
Tomasz can use a distributed code approach, maintain it on into the
future and keep prices around where they are (CAD doesn't come at AB
prices).
Another software architecture could be a better approach for AB - you
can still have lightning fast otpimization on the side but under that
scenario AB would make a quantum leap in overall capabilities rather
than a quantum leap solely in backtesting/otpimization speed.
If AB did go down that path then the software architecture could be a
radical depature from what has gone before and therefore the cost of
maintaining a parallel version for uniprocessors would, IMO, be too
high (in some cases I think that what has been pulled apart for
distribution to multicores doesn't necessarily come back together so
readily, or efficiently, for single core processing).
If I was Tomasz I would be thinking about a radical departure in AB
architecture and also making the quantum leap to an AB MCP version
(leave the uniprocessor versions behind completely with support for
uniprocessing versions ending pretty quickly - somewhere between
immediately and 2-3 years).
The big advantage of a load distributed model is that it can be done
piecemeal (to some extent) - it isn't an all or nothing, in one big
bang, scenario
Speculating on a load distributed model:
- when AB starts it 'polls' the OS for system status/CPU
availability/useage
- an AB task scheduler 'knows' what the load requirements are for
each of its distributable loads
- (optionally) it prompts the user with info about CPU consumption
and calls for input on AB priorities
- AB then distributes its load to 'spare' CPU capacity
- AB would need to make conservative decisions about CPU consumption
so as not to choke overall needs, as decided by the OS
- the AB task scheduler would continue to monitor CPU load and make
adjustments if core (OS) CPU consumption changed
- the AB task scheduler could prompt users about CPU competition and
suggest alternatives for manual load shedding.
Example 1:
- AB is started (it is an 8 core machine)
- no other programs are running (CPU consumption is low)
- AB is not using any core features (scrolling charts, calculating
indicators)
- the user has an I/O plugin installed and wants to run optimization
- AB, in conjunction with the OS, utilizes 7 cores for I/O (pretty
fast?)
(plugins could be a pretty effective way to manage AB processes as
load modules?)
Example 2:
- a dual core machine
- AB is running a backtest, using the 'spare' core, and the test is
incomplete
- an exchange opens and the user goes to live RT mode in AB and
starts running a trading platform for the session
- AB backtesting progress is preserved and the backtest is halted, by
the AB task scheduler, to free up the CPU's
- when the trading session is over AB auto resumes the backtest
Anyway, it appears Tomasz is someway down the track with MCP
deployment so I dare say he has made his decisions already.
Still, it is interesting to speculate on this subject.
brian_z
--- In amibroker@xxxxxxxxxxxxxxx, Fred Tonetti <ftonetti@xxx> wrote:
>
> Ohhh it certainly wasn't my idea to use sessions . It was Paul's
offhand
> comment about already doing it that brought light to the subject
for me .
>
>
>
> If I had been aware enough to realize this could have been done I
would have
> done it years ago .
>
>
>
> In regards to having multithreading within individual programs .
yes of
> course . CAD/CAM programs have been doing this for years .
>
>
>
> In regards to doing the same thing in IO . Yes of course . MCO was
the proof
> of concept if you will . The capability within IO already exists
now in test
> mode . I'm not sure what you mean by whether or not it might be a
smart
> thing to do but imho it certainly is useful as is MCO to speed up
exhaustive
> search.
>
>
>
> While you may not have the same chip . you will have the same
underlying
> instruction set .
>
>
>
> As far as backward compatibility is concerned . I don't think this
is an
> issue . It is relatively simple to detect how many processors are
available
> and make use of more than one if it exists.
>
>
>
> I can't say that I see anything all that wonderful out of CMA-ES
either but
> that is of course just one person's opinion based on limited
testing. The
> problem with intelligent algorithms in general is that by and large
most are
> not all that wonderful in their originally published version for
the kinds
> of tasks that they'd be put to in AB . Some have very limited scope
and
> others have characteristics and limitations that make them of
limited
> usefulness as published . Thus many need to be played with to see
what makes
> them better and that playing takes oodles of time just to see if
some
> particular one is worthwhile pursuing .
>
> _____
>
> From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx]
On Behalf
> Of brian_z111
> Sent: Sunday, June 29, 2008 10:44 AM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: [amibroker] Re: Another side to the multi-core support
argument
>
>
>
> You're not sorry - you enjoyed it ;-)
>
> Exactly as I imagined it - a perfect example of a single AB task
that
> lends itself perfectly to, what I would call, a distributed
approach.
>
> I think that is about as simple as AB MCP could get since such
> symmetry (repetition) of code would be rare, outside of opt?
>
> Carefully avoiding any further reading of docs, I was also
referring
> to the fact that you must have utilized the OS to 'force' the ABnn
> sessions (since we are in the accreditation mood I had better add I
> think someone else kicked in the idea to use 'sessions' but what
the
> hell?).
>
> You said that MCO was a pretty basic approach (or words to that
> effect) but it seems to me that utilizing the capabilities of the
OS
> to distribute tasks to CPU resources, in various ways, can also be
> done from within a program without the need for sessions.
>
> As things stand at the moment couldn't you go ahead (if you wanted
> to) and write an I/O plugin that utilizes all available cores WIWO
> Tomasz changing anything in AB? (not to say it is a smart idea but
> possible, yes?)
>
> As a general comment - not particularly for you - what I am kicking
> around is the idea that it would be a heck of a task to isolate the
> bits of AB code that lend themselves to it, chop them up for
> processing etc and then keep that up over the years (especially
since
> we won't all have the same OS/chip and/or number of cores) - I am
> wondering of a less optimum but more achieveable path might be to
use
> the task scheduling capabilities of the OS to produce a more
generic,
> maintainable way to use all available cores.
>
> The other point I am throwing on the table is - why would any
> developer maintain backward compatibility for uniprocessors when as
> of now no one is buyig one and realistically computers only have a
> life of a few years.
>
> brian_z
>
> brian_z
>
> FTR
>
> I am not convinced that CMA-ES is the ants pants of optimization
but
> I thought I would sit on the sidelines and let you two guys slug it
> out (of itself it might be nice and it could be a very good move as
> part of an overall, yet to be seen AB strategy but I would say it
has
> some limitations) .
>
> brian_z
>
> --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
ps.com, Fred
> Tonetti <ftonetti@> wrote:
> >
> > Nope . Didn't do anything like that .
> >
> >
> >
> > What I did was simply as you state . where a sequence of code
lends
> itself
> > to being chopped up, shoved through the processor and put back
> together
> > again on the other side .
> >
> >
> >
> > Even this description is not entirely accurate . What really
> happens is that
> > virtually the same AFL with different ranges of values for the
> optimization
> > statements is sent to multiple AB instances to be worked on by
> different
> > cores and then the results combined .
> >
> >
> >
> > From the doc ( Sorry to semi force you to read this )
> >
> >
> >
> > o MCO can deal with most forms of optimization statements.
> MCO will
> > check each of the Optimize statements in the AFL looking for the
> one that
> > has a numeric Min, Max and Increment arguments and needs the most
> steps to
> > get from the Min to Max value. Individual AFL's will be cloned
for
> each
> > ABnn session with modifications to this Optimize statement so
that
> each core
> > performs its share of the process. If for example the AFL to be
> run had the
> > following Optimize statements .
> >
> >
> >
> > Len1 = Optimize("Len1", 10, 1, 20, 1);
> >
> > Len2 = Optimize("Len2", 50, 1, 100, 1);
> >
> >
> >
> > . then it is the second Optimize statement that would be altered
> for each of
> > the AFL's to be used by the ABnn sessions. If one were using
three
> cores
> > without any /load command line arguments then the resulting
Optimize
> > statement for each of the three AFL's would look like what is
below
> > effectively splitting the workload as closely as possible across
> the user
> > specified cores.
> >
> >
> >
> > Len2 = Optimize("Len2", 50, 1, 33, 1);
> >
> > Len2 = Optimize("Len2", 50, 34, 66, 1);
> >
> > Len2 = Optimize("Len2", 50, 67, 100, 1);
> >
> >
> >
> > _____
> >
> > From: amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
ps.com
> [mailto:amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
ps.com]
> On Behalf
> > Of brian_z111
> > Sent: Sunday, June 29, 2008 4:39 AM
> > To: amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
ps.com
> > Subject: [amibroker] Re: Another side to the multi-core support
> argument
> >
> >
> >
> > Sorry, I used the term 'symmetric processing' incorrectly (I now
> see
> > that it has a formal definition whereas I used it informally).
> >
> > What I meant was that I don't think software applications for MCP
> are
> > limited only to single tasks where a sequence of code lends
itself
> to
> > being chopped up, shoved through the processor and put back
> together
> > again on the other side.
> >
> > Couldn't software modularise its tasks/features and present them
to
> > the OS as if they are separate programs e.g. downloading RT data,
> in
> > AB, and saving to a local database, could be allocated one core,
by
> > the OS, and rendering that data into a chart/processing
indicators
> > allocated to another - possibly AB could use some smart
> > prioritisation of its own tasks and work in tandem with the OS to
> > prioritise access to core CPU's?
> >
> > It seems to me we are limiting ourselves with regard to the
> > possibilities.
> >
> > (I haven't read, or tried, Freds MCO stuff - maybe that is what
he
> > did there already, in a basic way).
> >
> > If and when AB goes ahead with multicore application I have to
> wonder
> > why anyone wouldn't want to upgrade - could that be the time to
end
> > the unlimited backward compatibility rule (the benefits of
> > maintaining backward compatibility for a few would no longer be
> worth
> > the effort and even detrimental to the many)?
> >
> > brian_z
> >
> > --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> ps.com,
> > "brian_z111" <brian_z111@> wrote:
> > >
> > > Re: what can traders do with multicore processing (MCP)?.
> > >
> > > Gee, where are all the Codesters when you need one?
> > >
> > > Are you guys holding out on us?
> > >
> > > MCP applications aren't limited to symmetric processing are
they?
> > >
> > > http://en.wikipedia
> <http://en.wikipedia
<http://en.wikipedia.org/wiki/Computer_multitasking>
> .org/wiki/Computer_multitasking>
> > .org/wiki/Computer_multitasking
> > >
> > > http://en.wikipedia <http://en.wikipedia
> <http://en.wikipedia.org/wiki/Multi-core> .org/wiki/Multi-core>
> > .org/wiki/Multi-core
> > >
> > > "The amount of performance gained by the use of a multicore
> > processor
> > > depends on the problem being solved and the algorithms used, as
> > well
> > > as their implementation in software (Amdahl's law). For so-
> > > called "embarrassingly parallel" problems, a dual-core
processor
> > with
> > > two cores at 2GHz may perform very nearly as fast as a single
> core
> > of
> > > 4GHz.[1] Other problems though may not yield so much speedup.
> This
> > > all assumes however that the software has been designed to take
> > > advantage of available parallelism. If it hasn't, there will
not
> be
> > > any speedup at all. However, the processor will multitask
better
> > > since it can run two programs at once, one on each core."
> > >
> > >
> > > To what extent does it depend on the hardware/OS e.g. are the
> > > possibilties for AB different on a MSFT system with an AMD chip
> > > compared to another OS with a different chip - if so how would
AB
> > > handle that across a wide range of users - it seems to me the
AB
> > > solution will have to be generic rather than specialist?
> > >
> > >
> > > As a layperson, it appears that multi-tasking is a more
exciting
> > > prospect, for the majority of traders, than optimising at warp
> > speeds.
> > >
> > > Can't muliple processors handle multi-tasking at speeds unheard
> of
> > > before and hence bring some things into the realm of
> possibilities?
> > >
> > >
> > > How about:
> > >
> > > a) different databases running in one instance of AB - maybe
even
> > > with the databases in different timeframes/formats/coming from
> > > different providers and being cross referenced in indicators.
> > >
> > > Can MCP allow users,who now run more than one installation of
AB
> on
> > a
> > > single computer (for whatever reason) to get the same results
> from
> > > one installation.
> > >
> > >
> > > b) in the past Tomasz would enable AB's native database for
extra
> > > data (historical fields other than OHLCVOI) - as I understand
it
> > > because it would weigh AB down with things not used by the
> majority.
> > >
> > > Perhaps MCP will make the weight of an open structure AB
database
> a
> > > featherweight (PremiumData has already expressed an interest in
> > > synching their database with AB so it is not as if there is no
> > demand
> > > for this feature).
> > >
> > > c) a task scheduler built into AB - set the tasks at the start
of
> > the
> > > day - if the keyboard/processor is idle AB starts to run the
> tasks
> > > (backtesting/optimising).
> > >
> > > d) if we want to run our trading platform and AB on real time
> with
> > > two monitors on one computer do we have enough grunt to do that
> now
> > > OR would MCP allow us to run processing intensive tasks like
that
> > > concurrently OR maybe with a minimised AB chart (on tick data)
> > > floating on top of our trading platform window.
> > >
> > > I don't know - I am merely speculating!
> > >
> > > Surely someone out there has some better ideas?
> > >
> > > brian_z
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> ps.com,
> > "sidhartha70" <sidhartha70@>
> > > wrote:
> > > >
> > > > Hi Tomasz,
> > > >
> > > > I've been reading the arguments for and against multi-core
> support
> > > > with interest, and I just wanted to add my own voice to it
and
> > > mention
> > > > an angle which hasn't currently been focused on.
> > > >
> > > > Since I bought AmiBroker a couple of months back I've been
> blown
> > > away
> > > > by not only what a great piece of software it is, but also by
> it's
> > > > value and the fantastic customer support. Your own
> responsiveness
> > to
> > > > customer needs and ability to efficiently add new features
> leaves
> > me
> > > > with no hesitations in recommending AmiBroker to my fellow
> > traders.
> > > >
> > > > However, I use AB in two ways. First for swing trading... for
> > which
> > > it
> > > > is not only awesome, but also works perfectly happily on one
> core
> > > > because of the lower overhead in terms of data through put and
> > > > therefore calculation overhead on that data.
> > > >
> > > > I also use AB for higher frequency day trading... and it's
here
> > for
> > > me
> > > > that multi-core support could really add something. For
obvious
> > > > reasons the multi-core argument has so far been almost
> exclusively
> > > > focused on optimization. However, for higher frequency
traders
> > like
> > > > myself, working off just one core can be a significant
> bottleneck.
> > > >
> > > > I have an IQ Feed 'tick' database... and some quite complex
> chart
> > > > setup's & indictaors. Currently, with chart refresh interval
> set
> > to
> > > 2
> > > > seconds (I'd like it quikcer) I can see one core of my
computer
> > > > working away at about 50%. However, if I increase the update
> > > interval
> > > > to zero to bring in every tick, I see useage on one core go
up
> to
> > > > 80-90%. At this point, you can imagine, AB becomes noticeably
> > clunky
> > > > and difficult to use. Things get even worse if I start to
think
> > > about
> > > > symbol linking. (for reference I am only looking at a couple
of
> e-
> > > mini
> > > > symbols)
> > > > The frustrating aspect is that I have another 7 x 3Ghz cores
> > sitting
> > > > there twiddling their fingers while AB is grinding up on one
> core.
> > > >
> > > > So, I do think, for those of us who use AB for higher frequncy
> > > > trading, there is a very good argument to consider adding
multi-
> > core
> > > > support outside of the optimization argument.
> > > >
> > > > Love to hear your (or anyone's!) thoughts on this.
> > > >
> > > > Many Thanks
> > > >
> > >
> >
> >
> >
> >
> > _____
> >
> > I am using the free version of SPAMfighter for private users.
> > It has removed 492 spam emails to date.
> > Paying users do not have this message in their emails.
> > Try SPAMfighter <http://www.spamfigh
<http://www.spamfighter.com/len>
> ter.com/len> for free now!
> >
>
>
>
>
> _____
>
> I am using the free version of SPAMfighter for private users.
> It has removed 492 spam emails to date.
> Paying users do not have this message in their emails.
> Try SPAMfighter <http://www.spamfighter.com/len> for free now!
>
------------------------------------
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|