|
PureBytes Links
Trading Reference Links
|
TJ,
what's the back of the envelope calculation we should do to work out if our DB data set fits into on board CPU cache...??
For example, if I have an 8MB L2 Cache... how many bars can I fit into that comfortably...?
Many Thanks
--- In amibroker@xxxxxxxxxxxxxxx, Tomasz Janeczko <groups@xxx> wrote:
>
> Hello,
>
> There is no such thing as "no limit". Your HARDWARE IS LIMITED, 3rd
> party data vendor HAS LIMITS on amounts of data sent.
> The limits are there for a reason. To allow smooth REAL TIME OPERATION.
> If you override limits you are shooting yourself in a foot and will face
> problems with real-time operation.
> *DO NOT override limits*
> Real-time database must be able to keep up with data stream counting
> even thousands of ticks per second PER SYMBOL.
> Everyone who thinks that his/her hardware has no limits is simply very,
> very, very wrong.
> The person who thinks so is deeply wrong because he/she does not know
> how the amount of memory affects the performance.
> You guys should run memtest86 program to see what is the difference
> between processing speed of ON-CHIP CPU CACHE
> and normal memory. The difference is TEN FOLD (or larger). There are in
> fact differences as large as 2x or 3x in speed between level-1
> and level-2 caches. If you allow that your data are larger than your CPU
> cache is, you will automatically
> get 10x performance hit because of limited memory thoughput. CPU will
> WAIT for memory. Memory speeds are way behind processor speeds.
>
> If you need want "endless" database use LOCAL DATABASE only without
> external source. Such database working OFF-LINE
> is not subject to real-time constraints.
>
> Best regards,
> Tomasz Janeczko
> amibroker.com
>
> On 2010-03-11 11:19, Magesh wrote:
> >
> >
> > Some how my mails aren't getting through into the group. Hoping to get
> > luckier this time.
> >
> > Thanks to support@xxx, for I got clarity on this mail that #
> > of Bar limitations is meant only for plugin-driven databases.
> > Though that solved my worries, as I use a local database for caching
> > RT data, I still am curious to understand how such scenarios are
> > handled in real-time while using Plugin driven databases
> >
> > In other words, how to preserve my data from getting truncated once
> > the size limit exceeds in a Plugin-Driven database.
> > Any thoughts or suggestions are much appreciated.
> >
> > thanks and regards
> > Magesh
> >
> > On 10/03/10 11:01 AM, Magesh wrote:
> >>
> >> Does this mean a limitation of AB for saving data to infinite without
> >> truncating?
> >>
> >> I am using a real-time stream, Base Time Interval is "Tick" and No.
> >> of Bars=500,000.
> >> Total no of symbols = 10 (all with real-time Tick).
> >> I have a fixed layout, and no frequent change in symbols.
> >>
> >> And with my rough math of a tick for every 2 seconds,
> >> # of bars per trading day = 11700 bars
> >> Max # of days of history (@ 500,000 bars)= 43 days
> >> Max # of days of history (@ 1,500,000 bars) = 128 days
> >>
> >> My questions are:
> >> 1) Is my understandings on the bar count, size per symbol and limit
> >> is correct?
> >>
> >> 2) Is there any ways of overcoming this limit, my reason being, I
> >> would like to hold the historical tick size data for more than a year
> >> (at-least to start with)?
> >>
> >> 3) I think its scalable by altering "Base Time Interval" from "Tick"
> >> to "5/15 seconds" and will allow me to store up to 18 & 60 months
> >> approx. of data respectively. Is this right?
> >>
> >> 4) What's the max integer limit (size of variable) for MaximumNoOfBars?
> >>
> >> 5) Can alternate dBs (MySQL)be used in synch with AB's database, so
> >> that periodic push/flushing of AB's data into an external dBase? (as
> >> I won't be looking or using historical intraday's in realtime or
> >> atleast not so frequently)
> >>
> >> Btw, this is a total scaled down version of my requirement, as my
> >> database is just about a month old and currently I have about 30
> >> symbols and just begun learning AFL and back-testing/explorations.
> >> And with the current bar count setting, I think my database will
> >> start getting truncated anytime sooner (for few of the symbols
> >> updated via RT).
> >>
> >> So I would like to keep my options very open for any kind/type of
> >> suggestions as my current attempt is to have AB retain data and stop
> >> draining LIFO.
> >>
> >> Thanks in Advance,
> >> Magesh
> >>
> >>
> >> P.S.:I couldn't find any updates beyond this chain on this email
> >> thread subject from gmane, so any pointers or suggestions in this
> >> direction will be of great help for me.
> >>
> >>
> >>
> >> On 02/08/08 3:59 PM, Herman wrote:
> >>>
> >>> Hi Tomasz (or anyone who knows),
> >>>
> >>>
> >>> Could you define "advanced users and powerful computers"?
> >>>
> >>>
> >>> Who would use this feature and what what exactly does it do?
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> herman
> >>>
> >>>
> >>>
> >>> Friday, August 1, 2008, 7:21:11 AM, you wrote:
> >>>
> >>>
> >>> > No, it shouldn't. This key is a "hidden extra feature" for advanced
> >>>
> >>> > users having computers powerful enough to handle extra load.
> >>>
> >>> > By default the key does not exist and then the default max is 500000.
> >>>
> >>>
> >>> > Best regards,
> >>>
> >>> > Tomasz Janeczko
> >>>
> >>> > amibroker.com
> >>>
> >>> > ----- Original Message -----
> >>>
> >>> > From: "progster01"
> >>> <progster-sfbTBCYm2/nd5V35PSfx2NBPR1lH4CV8@xxx
> >>> <mailto:progster-sfbTBCYm2/nd5V35PSfx2NBPR1lH4CV8@...>>
> >>>
> >>> > To: <amibroker-hHKSG33TihhbjbujkaE4pw@xxx
> >>> <mailto:amibroker-hHKSG33TihhbjbujkaE4pw@...>>
> >>>
> >>> > Sent: Friday, August 01, 2008 12:55 PM
> >>>
> >>> > Subject: [amibroker] Re: Database settings and the amount of data saved
> >>>
> >>>
> >>>
> >>>
> >>> >>> Make sure you check the path exactly! The key name is
> >>>
> >>> >>> HKEY_CURRENT_USER\Software\TJP\Broker\Settings\MaximumNumber
> >>>
> >>> >>> OfBars
> >>>
> >>> >>> (watch out for line truncations)
> >>>
> >>>
> >>> >>> The key is a DWORD.
> >>>
> >>>
> >>> >> This key does not exist on the machine I'm looking at (which has AB
> >>>
> >>> >> 5.15 installed).
> >>>
> >>>
> >>> >> Obviously I could add it myself, but "should" it already be there?
> >>>
> >>>
> >>> >> IOW, what is the design scenario for the creation of this key?
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> >> ------------------------------------
> >>>
> >>>
> >>> >> Please note that this group is for discussion between users only.
> >>>
> >>>
> >>> >> To get support from AmiBroker please send an e-mail directly to
> >>>
> >>> >> SUPPORT {at} amibroker.com
> >>>
> >>>
> >>> >> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> >>>
> >>> >> http://www.amibroker.com/devlog/
> >>>
> >>>
> >>> >> For other support material please check also:
> >>>
> >>> >> http://www.amibroker.com/support.html
> >>>
> >>> >> Yahoo! Groups Links
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> > ------------------------------------
> >>>
> >>>
> >>> > Please note that this group is for discussion between users only.
> >>>
> >>>
> >>> > To get support from AmiBroker please send an e-mail directly to
> >>>
> >>> > SUPPORT {at} amibroker.com
> >>>
> >>>
> >>> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> >>>
> >>> > http://www.amibroker.com/devlog/
> >>>
> >>>
> >>> > For other support material please check also:
> >>>
> >>> > http://www.amibroker.com/support.html
> >>>
> >>> > Yahoo! Groups Links
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> > (Yahoo! ID required)
> >>>
> >>>
> >>>
> >>> > mailto:amibroker-fullfeatured-hHKSG33TihhbjbujkaE4pw@...
> >>>
> >>>
> >>> <mailto:amibroker-unsubscribe-hHKSG33TihhbjbujkaE4pw@...>
> >>>
> >>>
> >>>
> >>>
> >>>
> >
> >
> >
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
amibroker-digest@xxxxxxxxxxxxxxx
amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|