Re: [amibroker] Re: Database settings and the amount of data saved

[Tomasz Janeczko <groups@xxxxxxxxxxxxx>]

Hello,

There is no such thing as "no limit". Your HARDWARE IS LIMITED, 3rd party data vendor HAS LIMITS on amounts of data sent.
The limits are there for a reason. To allow smooth REAL TIME OPERATION.
If you override limits you are shooting yourself in a foot and will face problems with real-time operation.
*DO NOT override limits*
Real-time database must be able to keep up with data stream counting even thousands of ticks per second PER SYMBOL.
Everyone who thinks that his/her hardware has no limits is simply very, very, very wrong.
The person who thinks so is deeply wrong because he/she does not know how the amount of memory affects the performance.
You guys should run memtest86 program to see what is the difference between processing speed of ON-CHIP CPU CACHE
and normal memory. The difference is TEN FOLD (or larger). There are in fact differences as large as 2x or 3x in speed between level-1
and level-2 caches. If you allow that your data are larger than your CPU cache is, you will automatically
get 10x performance hit because of limited memory thoughput. CPU will WAIT for memory. Memory speeds are way behind processor speeds.

If you need want "endless" database use LOCAL DATABASE only without external source. Such database working OFF-LINE
is not subject to real-time constraints.

Best regards,
Tomasz Janeczko
amibroker.com

On 2010-03-11 11:19, Magesh wrote:

Some how my mails aren't getting through into the group. Hoping to get luckier this time.

Thanks to support@xxxxxxxxxxxxx, for I got clarity on this mail that # of Bar limitations is meant only  for plugin-driven databases.
Though that solved my worries, as I use a local database for caching RT data, I still am curious to understand how such scenarios are handled in real-time while using Plugin driven databases

In other words, how to preserve my data from getting truncated once the size limit exceeds in a Plugin-Driven database.
Any thoughts or suggestions are much appreciated.

thanks and regards
Magesh

On 10/03/10 11:01 AM, Magesh wrote:

Does this mean a limitation of AB for saving data to infinite without truncating?

I am using a real-time stream, Base Time Interval is "Tick" and No. of Bars=500,000.
Total no of symbols = 10 (all with real-time  Tick).
I have a fixed layout, and no frequent change in symbols.

And with my rough math of a tick for every 2 seconds,
# of bars per trading day = 11700 bars
Max # of days of history (@ 500,000 bars)= 43 days
Max # of days of history (@ 1,500,000 bars) = 128 days

My questions are:
1) Is my understandings on the bar count, size per symbol and limit is correct?

2) Is there any ways of overcoming this limit, my reason being, I would like to hold the historical tick size data for more than a year (at-least to start with)?

3) I think its scalable by altering "Base Time Interval" from "Tick" to "5/15 seconds" and will allow me to store up to 18 & 60 months approx. of data respectively. Is this right?

4) What's the max integer limit (size of variable) for MaximumNoOfBars?

5) Can alternate dBs (MySQL)be used in synch with AB's database, so that periodic push/flushing of AB's data into an external dBase? (as I won't be looking or using historical intraday's in realtime or atleast not so frequently)

Btw, this is a total scaled down version of my requirement, as my database is just about a month old and currently I have about 30 symbols and just begun learning AFL and back-testing/explorations. And with the current bar count setting, I think my database will start getting truncated anytime sooner (for few of the symbols updated via RT).

So I would like to keep my options very open for any kind/type of suggestions as my current attempt is to have AB retain data and stop draining LIFO.

Thanks in Advance,
Magesh


P.S.:I couldn't find any updates beyond this chain on this email thread subject from gmane, so any pointers or suggestions in this direction will be of great help for me.



On 02/08/08 3:59 PM, Herman wrote:

Hi Tomasz (or anyone who knows),

Could you define "advanced users and powerful computers"? 

Who would use this feature and what what exactly does it do?

Thanks,

herman

Friday, August 1, 2008, 7:21:11 AM, you wrote:

> No, it shouldn't. This key is a "hidden extra feature" for advanced

> users having computers powerful enough to handle extra load.

> By default the key does not exist and then the default max is 500000.

> Best regards,

> Tomasz Janeczko

> amibroker.com

> ----- Original Message ----- 

> From: "progster01" <progster-sfbTBCYm2/nd5V35PSfx2NBPR1lH4CV8@xxxxxxxxxxxxxxxx>

> To: <amibroker-hHKSG33TihhbjbujkaE4pw@xxxxxxxxxxxxxxxx>

> Sent: Friday, August 01, 2008 12:55 PM

> Subject: [amibroker] Re: Database settings and the amount of data saved

>>> Make sure you check the path exactly! The key name is 

>>> HKEY_CURRENT_USER\Software\TJP\Broker\Settings\MaximumNumber

>>> OfBars

>>> (watch out for line truncations)

>>> The key is a DWORD.

>> This key does not exist on the machine I'm looking at (which has AB

>> 5.15 installed).

>> Obviously I could add it myself, but "should" it already be there?

>> IOW, what is the design scenario for the creation of this key?

>> ------------------------------------

>> Please note that this group is for discussion between users only.

>> To get support from AmiBroker please send an e-mail directly to 

>> SUPPORT {at} amibroker.com

>> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:

>> http://www.amibroker.com/devlog/

>> For other support material please check also:

>> http://www.amibroker.com/support.html

>> Yahoo! Groups Links

> ------------------------------------

> Please note that this group is for discussion between users only.

> To get support from AmiBroker please send an e-mail directly to 

> SUPPORT {at} amibroker.com

> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:

> http://www.amibroker.com/devlog/

> For other support material please check also:

> http://www.amibroker.com/support.html

> Yahoo! Groups Links

> <*> To visit your group on the web, go to:

>     http://groups.yahoo.com/group/amibroker/

> <*> Your email settings:

>     Individual Email | Traditional

> <*> To change settings online go to:

>     http://groups.yahoo.com/group/amibroker/join

>     (Yahoo! ID required)

> <*> To change settings via email:

>     mailto:amibroker-digest-hHKSG33TihhbjbujkaE4pw@xxxxxxxxxxxxxxxx 

>     mailto:amibroker-fullfeatured-hHKSG33TihhbjbujkaE4pw@xxxxxxxxxxxxxxxx

> <*> To unsubscribe from this group, send an email to:

>     amibroker-unsubscribe-hHKSG33TihhbjbujkaE4pw@xxxxxxxxxxxxxxxx

> <*> Your use of Yahoo! Groups is subject to:

>     http://docs.yahoo.com/info/terms/

__._,_.___

**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.

TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com

TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___