|
PureBytes Links
Trading Reference Links
|
Thanks for that.
That seems to be pointing towards the fact that a generic approach to
suit all OS won't achieve the same performance as software
specifically written for OS X (assuming do they deliver with a giant
step forward).
brian_z
--- In amibroker@xxxxxxxxxxxxxxx, Dennis Brown <see3d@xxx> wrote:
>
> Brian,
>
> Just as an aside, Apple announced that the next version of OS
X "Snow
> Leopard" due in about a year would not have any new UI features as
a
> goal. The goal of the release will be to effectively support
multiple
> processors and release the support tools for developers that can
make
> it a lot easier for utilizing multiple processors. You can bet
that
> if Apple is making a statement like that, they will be making
industry
> leading changes to the way things work.
>
> I have a quad 2.8GHz machine right now. I opted to special order
a
> "downgrade" of their standard 8 core machine (and save $500)
because I
> did not see that I could get any value added with the extra 4
> processors due to the current state of multi-core application
> support. I run OS X, AB on a Windows XP VM, A very heavy Java
app
> from my broker, Safari browser, and a few utilities. I only keep
2
> cores busy on average --though it distributes the load around to
all
> 4. AB keeps one core busy 100%. I figure that by the time the
> software can really use the extra cores, it will be time for a
faster
> machine with larger built-in caches anyway. Then I will upgrade to
8
> or more cores --likely for the same price I paid for 4 this time.
>
> BR,
> Dennis
>
> On Jun 29, 2008, at 10:23 PM, brian_z111 wrote:
>
> >> As far as backward compatibility is concerned . I don't think
this
> >> is an
> >> issue . It is relatively simple to detect how many processors are
> >> available
> >> and make use of more than one if it exists.
> >
> > Developers don't seem to be fenced in when it comes to strategies
for
> > utilizing MCP capabilities - they can take a distributed code
> > approach (as you have done in MCO) or they can take a distributed
> > load approach (at the level of the OS) or something in between.
> >
> > If they go all the way with the distributed code approach then
> > possibly the capabilities of different chipsets don't equate at
the
> > lowest level e.g. threading versus fibers?
> >
> > The overheads of code distribution would be high and for some
> > programs the payback wouldn't be there e.g. CAD involves a lot of
> > data crunching and demands a distributed approach? On the other
hand
> > AB is a different kind of program altogether and I don't see that
> > Tomasz can use a distributed code approach, maintain it on into
the
> > future and keep prices around where they are (CAD doesn't come at
AB
> > prices).
> >
> > Another software architecture could be a better approach for AB -
you
> > can still have lightning fast otpimization on the side but under
that
> > scenario AB would make a quantum leap in overall capabilities
rather
> > than a quantum leap solely in backtesting/otpimization speed.
> >
> > If AB did go down that path then the software architecture could
be a
> > radical depature from what has gone before and therefore the cost
of
> > maintaining a parallel version for uniprocessors would, IMO, be
too
> > high (in some cases I think that what has been pulled apart for
> > distribution to multicores doesn't necessarily come back together
so
> > readily, or efficiently, for single core processing).
> >
> > If I was Tomasz I would be thinking about a radical departure in
AB
> > architecture and also making the quantum leap to an AB MCP version
> > (leave the uniprocessor versions behind completely with support
for
> > uniprocessing versions ending pretty quickly - somewhere between
> > immediately and 2-3 years).
> >
> > The big advantage of a load distributed model is that it can be
done
> > piecemeal (to some extent) - it isn't an all or nothing, in one
big
> > bang, scenario
> >
> >
> > Speculating on a load distributed model:
> >
> > - when AB starts it 'polls' the OS for system status/CPU
> > availability/useage
> > - an AB task scheduler 'knows' what the load requirements are for
> > each of its distributable loads
> > - (optionally) it prompts the user with info about CPU consumption
> > and calls for input on AB priorities
> > - AB then distributes its load to 'spare' CPU capacity
> > - AB would need to make conservative decisions about CPU
consumption
> > so as not to choke overall needs, as decided by the OS
> > - the AB task scheduler would continue to monitor CPU load and
make
> > adjustments if core (OS) CPU consumption changed
> > - the AB task scheduler could prompt users about CPU competition
and
> > suggest alternatives for manual load shedding.
> >
> > Example 1:
> >
> > - AB is started (it is an 8 core machine)
> > - no other programs are running (CPU consumption is low)
> > - AB is not using any core features (scrolling charts, calculating
> > indicators)
> > - the user has an I/O plugin installed and wants to run
optimization
> > - AB, in conjunction with the OS, utilizes 7 cores for I/O (pretty
> > fast?)
> >
> > (plugins could be a pretty effective way to manage AB processes as
> > load modules?)
> >
> >
> > Example 2:
> >
> > - a dual core machine
> > - AB is running a backtest, using the 'spare' core, and the test
is
> > incomplete
> > - an exchange opens and the user goes to live RT mode in AB and
> > starts running a trading platform for the session
> > - AB backtesting progress is preserved and the backtest is
halted, by
> > the AB task scheduler, to free up the CPU's
> > - when the trading session is over AB auto resumes the backtest
> >
> > Anyway, it appears Tomasz is someway down the track with MCP
> > deployment so I dare say he has made his decisions already.
> >
> > Still, it is interesting to speculate on this subject.
> >
> > brian_z
> >
> >
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, Fred Tonetti <ftonetti@> wrote:
> >>
> >> Ohhh it certainly wasn't my idea to use sessions . It was Paul's
> > offhand
> >> comment about already doing it that brought light to the subject
> > for me .
> >>
> >>
> >>
> >> If I had been aware enough to realize this could have been done I
> > would have
> >> done it years ago .
> >>
> >>
> >>
> >> In regards to having multithreading within individual programs .
> > yes of
> >> course . CAD/CAM programs have been doing this for years .
> >>
> >>
> >>
> >> In regards to doing the same thing in IO . Yes of course . MCO
was
> > the proof
> >> of concept if you will . The capability within IO already exists
> > now in test
> >> mode . I'm not sure what you mean by whether or not it might be a
> > smart
> >> thing to do but imho it certainly is useful as is MCO to speed up
> > exhaustive
> >> search.
> >>
> >>
> >>
> >> While you may not have the same chip . you will have the same
> > underlying
> >> instruction set .
> >>
> >>
> >>
> >> As far as backward compatibility is concerned . I don't think
this
> > is an
> >> issue . It is relatively simple to detect how many processors are
> > available
> >> and make use of more than one if it exists.
> >>
> >>
> >>
> >> I can't say that I see anything all that wonderful out of CMA-ES
> > either but
> >> that is of course just one person's opinion based on limited
> > testing. The
> >> problem with intelligent algorithms in general is that by and
large
> > most are
> >> not all that wonderful in their originally published version for
> > the kinds
> >> of tasks that they'd be put to in AB . Some have very limited
scope
> > and
> >> others have characteristics and limitations that make them of
> > limited
> >> usefulness as published . Thus many need to be played with to see
> > what makes
> >> them better and that playing takes oodles of time just to see if
> > some
> >> particular one is worthwhile pursuing .
> >>
> >> _____
> >>
> >> From: amibroker@xxxxxxxxxxxxxxx
[mailto:amibroker@xxxxxxxxxxxxxxx]
> > On Behalf
> >> Of brian_z111
> >> Sent: Sunday, June 29, 2008 10:44 AM
> >> To: amibroker@xxxxxxxxxxxxxxx
> >> Subject: [amibroker] Re: Another side to the multi-core support
> > argument
> >>
> >>
> >>
> >> You're not sorry - you enjoyed it ;-)
> >>
> >> Exactly as I imagined it - a perfect example of a single AB task
> > that
> >> lends itself perfectly to, what I would call, a distributed
> > approach.
> >>
> >> I think that is about as simple as AB MCP could get since such
> >> symmetry (repetition) of code would be rare, outside of opt?
> >>
> >> Carefully avoiding any further reading of docs, I was also
> > referring
> >> to the fact that you must have utilized the OS to 'force' the
ABnn
> >> sessions (since we are in the accreditation mood I had better
add I
> >> think someone else kicked in the idea to use 'sessions' but what
> > the
> >> hell?).
> >>
> >> You said that MCO was a pretty basic approach (or words to that
> >> effect) but it seems to me that utilizing the capabilities of the
> > OS
> >> to distribute tasks to CPU resources, in various ways, can also
be
> >> done from within a program without the need for sessions.
> >>
> >> As things stand at the moment couldn't you go ahead (if you
wanted
> >> to) and write an I/O plugin that utilizes all available cores
WIWO
> >> Tomasz changing anything in AB? (not to say it is a smart idea
but
> >> possible, yes?)
> >>
> >> As a general comment - not particularly for you - what I am
kicking
> >> around is the idea that it would be a heck of a task to isolate
the
> >> bits of AB code that lend themselves to it, chop them up for
> >> processing etc and then keep that up over the years (especially
> > since
> >> we won't all have the same OS/chip and/or number of cores) - I am
> >> wondering of a less optimum but more achieveable path might be to
> > use
> >> the task scheduling capabilities of the OS to produce a more
> > generic,
> >> maintainable way to use all available cores.
> >>
> >> The other point I am throwing on the table is - why would any
> >> developer maintain backward compatibility for uniprocessors when
as
> >> of now no one is buyig one and realistically computers only have
a
> >> life of a few years.
> >>
> >> brian_z
> >>
> >> brian_z
> >>
> >> FTR
> >>
> >> I am not convinced that CMA-ES is the ants pants of optimization
> > but
> >> I thought I would sit on the sidelines and let you two guys slug
it
> >> out (of itself it might be nice and it could be a very good move
as
> >> part of an overall, yet to be seen AB strategy but I would say it
> > has
> >> some limitations) .
> >>
> >> brian_z
> >>
> >> --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> > ps.com, Fred
> >> Tonetti <ftonetti@> wrote:
> >>>
> >>> Nope . Didn't do anything like that .
> >>>
> >>>
> >>>
> >>> What I did was simply as you state . where a sequence of code
> > lends
> >> itself
> >>> to being chopped up, shoved through the processor and put back
> >> together
> >>> again on the other side .
> >>>
> >>>
> >>>
> >>> Even this description is not entirely accurate . What really
> >> happens is that
> >>> virtually the same AFL with different ranges of values for the
> >> optimization
> >>> statements is sent to multiple AB instances to be worked on by
> >> different
> >>> cores and then the results combined .
> >>>
> >>>
> >>>
> >>> From the doc ( Sorry to semi force you to read this )
> >>>
> >>>
> >>>
> >>> o MCO can deal with most forms of optimization statements.
> >> MCO will
> >>> check each of the Optimize statements in the AFL looking for the
> >> one that
> >>> has a numeric Min, Max and Increment arguments and needs the
most
> >> steps to
> >>> get from the Min to Max value. Individual AFL's will be cloned
> > for
> >> each
> >>> ABnn session with modifications to this Optimize statement so
> > that
> >> each core
> >>> performs its share of the process. If for example the AFL to be
> >> run had the
> >>> following Optimize statements .
> >>>
> >>>
> >>>
> >>> Len1 = Optimize("Len1", 10, 1, 20, 1);
> >>>
> >>> Len2 = Optimize("Len2", 50, 1, 100, 1);
> >>>
> >>>
> >>>
> >>> . then it is the second Optimize statement that would be altered
> >> for each of
> >>> the AFL's to be used by the ABnn sessions. If one were using
> > three
> >> cores
> >>> without any /load command line arguments then the resulting
> > Optimize
> >>> statement for each of the three AFL's would look like what is
> > below
> >>> effectively splitting the workload as closely as possible across
> >> the user
> >>> specified cores.
> >>>
> >>>
> >>>
> >>> Len2 = Optimize("Len2", 50, 1, 33, 1);
> >>>
> >>> Len2 = Optimize("Len2", 50, 34, 66, 1);
> >>>
> >>> Len2 = Optimize("Len2", 50, 67, 100, 1);
> >>>
> >>>
> >>>
> >>> _____
> >>>
> >>> From: amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> > ps.com
> >> [mailto:amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> > ps.com]
> >> On Behalf
> >>> Of brian_z111
> >>> Sent: Sunday, June 29, 2008 4:39 AM
> >>> To: amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> > ps.com
> >>> Subject: [amibroker] Re: Another side to the multi-core support
> >> argument
> >>>
> >>>
> >>>
> >>> Sorry, I used the term 'symmetric processing' incorrectly (I now
> >> see
> >>> that it has a formal definition whereas I used it informally).
> >>>
> >>> What I meant was that I don't think software applications for
MCP
> >> are
> >>> limited only to single tasks where a sequence of code lends
> > itself
> >> to
> >>> being chopped up, shoved through the processor and put back
> >> together
> >>> again on the other side.
> >>>
> >>> Couldn't software modularise its tasks/features and present them
> > to
> >>> the OS as if they are separate programs e.g. downloading RT
data,
> >> in
> >>> AB, and saving to a local database, could be allocated one core,
> > by
> >>> the OS, and rendering that data into a chart/processing
> > indicators
> >>> allocated to another - possibly AB could use some smart
> >>> prioritisation of its own tasks and work in tandem with the OS
to
> >>> prioritise access to core CPU's?
> >>>
> >>> It seems to me we are limiting ourselves with regard to the
> >>> possibilities.
> >>>
> >>> (I haven't read, or tried, Freds MCO stuff - maybe that is what
> > he
> >>> did there already, in a basic way).
> >>>
> >>> If and when AB goes ahead with multicore application I have to
> >> wonder
> >>> why anyone wouldn't want to upgrade - could that be the time to
> > end
> >>> the unlimited backward compatibility rule (the benefits of
> >>> maintaining backward compatibility for a few would no longer be
> >> worth
> >>> the effort and even detrimental to the many)?
> >>>
> >>> brian_z
> >>>
> >>> --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> >> ps.com,
> >>> "brian_z111" <brian_z111@> wrote:
> >>>>
> >>>> Re: what can traders do with multicore processing (MCP)?.
> >>>>
> >>>> Gee, where are all the Codesters when you need one?
> >>>>
> >>>> Are you guys holding out on us?
> >>>>
> >>>> MCP applications aren't limited to symmetric processing are
> > they?
> >>>>
> >>>> http://en.wikipedia
> >> <http://en.wikipedia
> > <http://en.wikipedia.org/wiki/Computer_multitasking>
> >> .org/wiki/Computer_multitasking>
> >>> .org/wiki/Computer_multitasking
> >>>>
> >>>> http://en.wikipedia <http://en.wikipedia
> >> <http://en.wikipedia.org/wiki/Multi-core> .org/wiki/Multi-core>
> >>> .org/wiki/Multi-core
> >>>>
> >>>> "The amount of performance gained by the use of a multicore
> >>> processor
> >>>> depends on the problem being solved and the algorithms used, as
> >>> well
> >>>> as their implementation in software (Amdahl's law). For so-
> >>>> called "embarrassingly parallel" problems, a dual-core
> > processor
> >>> with
> >>>> two cores at 2GHz may perform very nearly as fast as a single
> >> core
> >>> of
> >>>> 4GHz.[1] Other problems though may not yield so much speedup.
> >> This
> >>>> all assumes however that the software has been designed to take
> >>>> advantage of available parallelism. If it hasn't, there will
> > not
> >> be
> >>>> any speedup at all. However, the processor will multitask
> > better
> >>>> since it can run two programs at once, one on each core."
> >>>>
> >>>>
> >>>> To what extent does it depend on the hardware/OS e.g. are the
> >>>> possibilties for AB different on a MSFT system with an AMD chip
> >>>> compared to another OS with a different chip - if so how would
> > AB
> >>>> handle that across a wide range of users - it seems to me the
> > AB
> >>>> solution will have to be generic rather than specialist?
> >>>>
> >>>>
> >>>> As a layperson, it appears that multi-tasking is a more
> > exciting
> >>>> prospect, for the majority of traders, than optimising at warp
> >>> speeds.
> >>>>
> >>>> Can't muliple processors handle multi-tasking at speeds unheard
> >> of
> >>>> before and hence bring some things into the realm of
> >> possibilities?
> >>>>
> >>>>
> >>>> How about:
> >>>>
> >>>> a) different databases running in one instance of AB - maybe
> > even
> >>>> with the databases in different timeframes/formats/coming from
> >>>> different providers and being cross referenced in indicators.
> >>>>
> >>>> Can MCP allow users,who now run more than one installation of
> > AB
> >> on
> >>> a
> >>>> single computer (for whatever reason) to get the same results
> >> from
> >>>> one installation.
> >>>>
> >>>>
> >>>> b) in the past Tomasz would enable AB's native database for
> > extra
> >>>> data (historical fields other than OHLCVOI) - as I understand
> > it
> >>>> because it would weigh AB down with things not used by the
> >> majority.
> >>>>
> >>>> Perhaps MCP will make the weight of an open structure AB
> > database
> >> a
> >>>> featherweight (PremiumData has already expressed an interest in
> >>>> synching their database with AB so it is not as if there is no
> >>> demand
> >>>> for this feature).
> >>>>
> >>>> c) a task scheduler built into AB - set the tasks at the start
> > of
> >>> the
> >>>> day - if the keyboard/processor is idle AB starts to run the
> >> tasks
> >>>> (backtesting/optimising).
> >>>>
> >>>> d) if we want to run our trading platform and AB on real time
> >> with
> >>>> two monitors on one computer do we have enough grunt to do that
> >> now
> >>>> OR would MCP allow us to run processing intensive tasks like
> > that
> >>>> concurrently OR maybe with a minimised AB chart (on tick data)
> >>>> floating on top of our trading platform window.
> >>>>
> >>>> I don't know - I am merely speculating!
> >>>>
> >>>> Surely someone out there has some better ideas?
> >>>>
> >>>> brian_z
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> >> ps.com,
> >>> "sidhartha70" <sidhartha70@>
> >>>> wrote:
> >>>>>
> >>>>> Hi Tomasz,
> >>>>>
> >>>>> I've been reading the arguments for and against multi-core
> >> support
> >>>>> with interest, and I just wanted to add my own voice to it
> > and
> >>>> mention
> >>>>> an angle which hasn't currently been focused on.
> >>>>>
> >>>>> Since I bought AmiBroker a couple of months back I've been
> >> blown
> >>>> away
> >>>>> by not only what a great piece of software it is, but also by
> >> it's
> >>>>> value and the fantastic customer support. Your own
> >> responsiveness
> >>> to
> >>>>> customer needs and ability to efficiently add new features
> >> leaves
> >>> me
> >>>>> with no hesitations in recommending AmiBroker to my fellow
> >>> traders.
> >>>>>
> >>>>> However, I use AB in two ways. First for swing trading... for
> >>> which
> >>>> it
> >>>>> is not only awesome, but also works perfectly happily on one
> >> core
> >>>>> because of the lower overhead in terms of data through put and
> >>>>> therefore calculation overhead on that data.
> >>>>>
> >>>>> I also use AB for higher frequency day trading... and it's
> > here
> >>> for
> >>>> me
> >>>>> that multi-core support could really add something. For
> > obvious
> >>>>> reasons the multi-core argument has so far been almost
> >> exclusively
> >>>>> focused on optimization. However, for higher frequency
> > traders
> >>> like
> >>>>> myself, working off just one core can be a significant
> >> bottleneck.
> >>>>>
> >>>>> I have an IQ Feed 'tick' database... and some quite complex
> >> chart
> >>>>> setup's & indictaors. Currently, with chart refresh interval
> >> set
> >>> to
> >>>> 2
> >>>>> seconds (I'd like it quikcer) I can see one core of my
> > computer
> >>>>> working away at about 50%. However, if I increase the update
> >>>> interval
> >>>>> to zero to bring in every tick, I see useage on one core go
> > up
> >> to
> >>>>> 80-90%. At this point, you can imagine, AB becomes noticeably
> >>> clunky
> >>>>> and difficult to use. Things get even worse if I start to
> > think
> >>>> about
> >>>>> symbol linking. (for reference I am only looking at a couple
> > of
> >> e-
> >>>> mini
> >>>>> symbols)
> >>>>> The frustrating aspect is that I have another 7 x 3Ghz cores
> >>> sitting
> >>>>> there twiddling their fingers while AB is grinding up on one
> >> core.
> >>>>>
> >>>>> So, I do think, for those of us who use AB for higher frequncy
> >>>>> trading, there is a very good argument to consider adding
> > multi-
> >>> core
> >>>>> support outside of the optimization argument.
> >>>>>
> >>>>> Love to hear your (or anyone's!) thoughts on this.
> >>>>>
> >>>>> Many Thanks
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>> _____
> >>>
> >>> I am using the free version of SPAMfighter for private users.
> >>> It has removed 492 spam emails to date.
> >>> Paying users do not have this message in their emails.
> >>> Try SPAMfighter <http://www.spamfigh
> > <http://www.spamfighter.com/len>
> >> ter.com/len> for free now!
> >>>
> >>
> >>
> >>
> >>
> >> _____
> >>
> >> I am using the free version of SPAMfighter for private users.
> >> It has removed 492 spam emails to date.
> >> Paying users do not have this message in their emails.
> >> Try SPAMfighter <http://www.spamfighter.com/len> for free now!
> >>
> >
> >
> >
> > ------------------------------------
> >
> > Please note that this group is for discussion between users only.
> >
> > To get support from AmiBroker please send an e-mail directly to
> > SUPPORT {at} amibroker.com
> >
> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> > http://www.amibroker.com/devlog/
> >
> > For other support material please check also:
> > http://www.amibroker.com/support.html
> > Yahoo! Groups Links
> >
> >
> >
>
------------------------------------
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|