|
PureBytes Links
Trading Reference Links
|
Tomasz
Dollars and cents aside, i would like to understand the criteria that
is used in the data that would be purged.
Here is the case that i am trying to understand.
My AB has # o fbars 50000 in base interval 1 minute
The data feed comign in is a tick level data feed.
Questions are
1. Does AB store it as a Tick data internally. The reason is that if
i want to change to see tick data later on, could i see it for the
timeframe without it going to the plugin to get the tick data
2. AB tells me 50k bars = 104 days with 1562 kb per symbol. So does
it use FIFO to purge the bar that are >104 days. This means that i
can backup my database files every 104 days to get some older data if
i needed.
Seede
--- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@xxx>
wrote:
>
> That's special price for AmiBroker users :-)
>
> Best regards,
> Tomasz Janeczko
> amibroker.com
> ----- Original Message -----
> From: Don Lindberg
> To: amibroker@xxxxxxxxxxxxxxx
> Sent: Thursday, September 13, 2007 6:37 AM
> Subject: RE: [AmiBroker] Re: Is 500,000 the maximum number of
bars allowable in a database?
>
>
> Tomasz,
>
> Your to cheap. Programming consultants are in the $80 - $100 hr
range!
>
>
>
> Donald F Lindberg
>
>
> --------------------------------------------------------------------
----------
>
> From: amibroker@xxxxxxxxxxxxxxx
[mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of eric tao
> Sent: Wednesday, September 12, 2007 8:55 PM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: [amibroker] Re: Is 500,000 the maximum number of bars
allowable in a database?
>
>
>
> $60/hours, Good to know :)
>
> --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@>
wrote:
> >
> > > This isn't even a major change. It just removes the direct
> > > relationship of per symbol data on disk to the amount of
symbol data
> > > Amibroker processes.
> >
> > I will tell you something: you have NO idea, absolutely NO
IDEA, if
> it is major change or not.
> > As far as AmiBroker internals are considered your guesses are as
> accurate as forecasting the value of SP500 in December 2017.
> >
> > The one you think isn't major is in fact big (requires in fact
LOTS
> of changes) and no-one except 0.5% of AB user base would benefit
> > from it.
> >
> > Are you willing to pay for custom development? $60/hour is the
rate.
> > It will take about 10 days of development and testing. Are you
> willing to pay $4800 to develop it for you?
> > If so, no problem, I will do that for $4800.
> >
> > Best regards,
> > Tomasz Janeczko
> > amibroker.com
> > ----- Original Message -----
> > From: "scourt2000" <stevehite@>
> > To: <amibroker@xxxxxxxxxxxxxxx>
> > Sent: Wednesday, September 12, 2007 4:43 PM
> > Subject: [amibroker] Re: Is 500,000 the maximum number of bars
> allowable in a database?
> >
> >
> > >
> > > Tomasz,
> > >
> > > Just because I store large amounts of data in one file (aka, a
> > > symbol), it doesn't mean that Amibroker has to process all of
that
> > > data. Take the example of sqlite, the free database stores
all of its
> > > information in one file but users don't process all of that
data just
> > > because it's in one physical file on a disk. The stock data in
> > > Amibroker is a simple database.
> > >
> > > Please address this point:
> > >
> > > 1. Amibroker should never delete old stock/futures data just
because
> > > new data is coming in. All that is needed is for Amibroker to
limit
> > > the amount of data it is willing to pull in historically from
some
> > > specified start point in the past. Whether that is a certain
number
> > > of bars or a specific date range, that's fine. Like I alluded
to, if
> > > I have 5 years worth of something like ES tick data as a
continuous
> > > contract in one file and I want to backtest (or view) a date
range in
> > > that data, then I should have some way in Amibroker of
specifying a
> > > start date and end date for the data segment to be pulled in
for
> > > processing.
> > >
> > > Here's what Amibroker currently 'says': "I'm going to use all
of the
> > > data you have for a symbol. If you have too much, then I am
going to
> > > blow out the PC's memory. Game over."
> > >
> > > Here's what Amibroker should do: If there's too much data for
a
> > > symbol, let the user specify a date range for testing/viewing
or so
> > > many bars leading up to the present bar for testing/viewing.
The
> > > actual physical size of that data on disk remains intact,
just not all
> > > available for processing due to memory limitations.
> > >
> > > This isn't even a major change. It just removes the direct
> > > relationship of per symbol data on disk to the amount of
symbol data
> > > Amibroker processes.
> > >
> > > And like I said previously, there is a workaround for this. I
have to
> > > chop up my data into separate, artificial symbol names. Where
does
> > > this put the burden of the workload? On me, of course. What is
> > > software supposed to do? It's supposed to take the burden of
the
> > > workload off of the user as much as possible.
> > >
> > > Steve
> > >
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@>
wrote:
> > >>
> > >> Hello,
> > >>
> > >> Why don't you move to Metastock that has 65000 bar limit ?
> > >>
> > >> The truth is that AmiBroker is the most capable in this area
because
> > >> it stores way more data than other softwares. Also, by
changing the
> > > registry
> > >> limit you can set it to any value you need. The limit is not
the
> > > software but the hardware
> > >>
> > >> It is funny when people make statements like yours without
taking
> > > calculator and
> > >> actually checking how much memory you need to store that.
> > >>
> > >> 20 million bars - OK, but did you actually count how much
memory you
> > > need for that?
> > >> Each quote is 64 bytes, now 20 million of bars is 1.2GB of
memory
> > > per symbol.
> > >>
> > >> You may say "but you don't need to load it at once". But
> > > unfortunatelly you do need that.
> > >> How you can backtest without loading the data ?
> > >> And each AFL array in such case would need 4 * 20 million =
80 MB of
> > > RAM.
> > >> Usually user formulas use hundreds of such arrays.
> > >>
> > >> Wise old words are: "You can't eat your cake and have it
too".
> > >>
> > >> If you want more data stored - the price for that is
increased
> > > memory requirements
> > >> and more CPU power needed to process it.
> > >>
> > >> Now you can say: you can process one bar at a time - yes you
can -
> > > but you need
> > >> to store intermediate results anyway for next iteration and
that
> > > means that memory requirements
> > >> would not be even one byte less than they are. Plus changing
> > > execution model to 1-bar at a time will
> > >> mean (number-of-bars)-times slow down.
> > >>
> > >> So you would end up with system that is slow, slow and even
more
> slow.
> > >>
> > >> Best regards,
> > >> Tomasz Janeczko
> > >> amibroker.com
> > >> ----- Original Message -----
> > >> From: "scourt2000" <stevehite@>
> > >> To: <amibroker@xxxxxxxxxxxxxxx>
> > >> Sent: Wednesday, September 12, 2007 2:00 PM
> > >> Subject: [amibroker] Re: Is 500,000 the maximum number of
bars
> > > allowable in a database?
> > >>
> > >>
> > >> >
> > >> > And this is my biggest pet peeve about Amibroker:
> > >> >
> > >> > If I have the disk space available, Amibroker should never,
> ever, EVER
> > >> > just silently blow away my historical data because of some
> arbitrary
> > >> > system limit like 500,000 bars or 1,000,000. If I want 20
million
> > >> > bars historical and I have the disk space to store it, then
> fine. If
> > >> > Amibroker needs far less to function well, then that's
fine too.
> > >> >
> > >> > The solution is NOT to delete old data. The solution is to
> have a way
> > >> > for Amibroker to limit the number of past bars (for
calculation
> > >> > purposes) or even have me set a beginning and end date in
an
> > >> > historical range WITHOUT destroying that historical data
(i.e., the
> > >> > present day should not have to be the last bar that I view
or
> use for
> > >> > backtesting).
> > >> >
> > >> > Currently, the only solution is to set some registry value
to
> go over
> > >> > 500,000, but that in itself is not solution. Because of
> Amibroker's
> > >> > implementation, you'll get your memory stressed out big
time and,
> > >> > again, if your new limit gets reached, Amibroker will
remove your
> > >> > oldest data to make room for new data.
> > >> >
> > >> > No one ever said that you had to look at stock/futures data
> from the
> > >> > complete past to today. If I had 20 years of one symbol's
> data and
> > >> > that data was far too much to make available at one time
because of
> > >> > memory problems, then it should be segmentable without me
having to
> > >> > split of the data in artificial symbols.
> > >> >
> > >> > Of course there's a workaround (there usually is). You
have to
> split
> > >> > your data up (e.g., tick data) into segments with different
> artificial
> > >> > symbol names (e.g., I use eSignal continuous futures
contracts).
> > >> >
> > >> >
> > >> > --- In amibroker@xxxxxxxxxxxxxxx, Graham <kavemanperth@>
wrote:
> > >> >>
> > >> >> I believe the database limit is set to 500,000 but can be
> changed in
> > >> >> the windows registry. TJ has mentioned this in a previous
post
> here,
> > >> >> somewhere sometime.
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Cheers
> > >> >> Graham Kav
> > >> >> AFL Writing Service
> > >> >> http://www.aflwriting.com
> > >> >>
> > >> >>
> > >> >> On 12/09/2007, Ara Kaloustian <ara1@> wrote:
> > >> >> >
> > >> >> >
> > >> >> > You can have as much as you want, but performance
degrades.
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Please note that this group is for discussion between
users only.
> > >> >
> > >> > To get support from AmiBroker please send an e-mail
directly to
> > >> > SUPPORT {at} amibroker.com
> > >> >
> > >> > For NEW RELEASE ANNOUNCEMENTS and other news always check
DEVLOG:
> > >> > http://www.amibroker.com/devlog/
> > >> >
> > >> > For other support material please check also:
> > >> > http://www.amibroker.com/support.html
> > >> >
> > >> > Yahoo! Groups Links
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> > >
> > >
> > >
> > >
> > > Please note that this group is for discussion between users
only.
> > >
> > > To get support from AmiBroker please send an e-mail directly
to
> > > SUPPORT {at} amibroker.com
> > >
> > > For NEW RELEASE ANNOUNCEMENTS and other news always check
DEVLOG:
> > > http://www.amibroker.com/devlog/
> > >
> > > For other support material please check also:
> > > http://www.amibroker.com/support.html
> > >
> > > Yahoo! Groups Links
> > >
> > >
> > >
> > >
> > >
> >
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|