|
PureBytes Links
Trading Reference Links
|
Tomasz,
Just because I store large amounts of data in one file (aka, a
symbol), it doesn't mean that Amibroker has to process all of that
data. Take the example of sqlite, the free database stores all of its
information in one file but users don't process all of that data just
because it's in one physical file on a disk. The stock data in
Amibroker is a simple database.
Please address this point:
1. Amibroker should never delete old stock/futures data just because
new data is coming in. All that is needed is for Amibroker to limit
the amount of data it is willing to pull in historically from some
specified start point in the past. Whether that is a certain number
of bars or a specific date range, that's fine. Like I alluded to, if
I have 5 years worth of something like ES tick data as a continuous
contract in one file and I want to backtest (or view) a date range in
that data, then I should have some way in Amibroker of specifying a
start date and end date for the data segment to be pulled in for
processing.
Here's what Amibroker currently 'says': "I'm going to use all of the
data you have for a symbol. If you have too much, then I am going to
blow out the PC's memory. Game over."
Here's what Amibroker should do: If there's too much data for a
symbol, let the user specify a date range for testing/viewing or so
many bars leading up to the present bar for testing/viewing. The
actual physical size of that data on disk remains intact, just not all
available for processing due to memory limitations.
This isn't even a major change. It just removes the direct
relationship of per symbol data on disk to the amount of symbol data
Amibroker processes.
And like I said previously, there is a workaround for this. I have to
chop up my data into separate, artificial symbol names. Where does
this put the burden of the workload? On me, of course. What is
software supposed to do? It's supposed to take the burden of the
workload off of the user as much as possible.
Steve
--- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@xxx> wrote:
>
> Hello,
>
> Why don't you move to Metastock that has 65000 bar limit ?
>
> The truth is that AmiBroker is the most capable in this area because
> it stores way more data than other softwares. Also, by changing the
registry
> limit you can set it to any value you need. The limit is not the
software but the hardware
>
> It is funny when people make statements like yours without taking
calculator and
> actually checking how much memory you need to store that.
>
> 20 million bars - OK, but did you actually count how much memory you
need for that?
> Each quote is 64 bytes, now 20 million of bars is 1.2GB of memory
per symbol.
>
> You may say "but you don't need to load it at once". But
unfortunatelly you do need that.
> How you can backtest without loading the data ?
> And each AFL array in such case would need 4 * 20 million = 80 MB of
RAM.
> Usually user formulas use hundreds of such arrays.
>
> Wise old words are: "You can't eat your cake and have it too".
>
> If you want more data stored - the price for that is increased
memory requirements
> and more CPU power needed to process it.
>
> Now you can say: you can process one bar at a time - yes you can -
but you need
> to store intermediate results anyway for next iteration and that
means that memory requirements
> would not be even one byte less than they are. Plus changing
execution model to 1-bar at a time will
> mean (number-of-bars)-times slow down.
>
> So you would end up with system that is slow, slow and even more slow.
>
> Best regards,
> Tomasz Janeczko
> amibroker.com
> ----- Original Message -----
> From: "scourt2000" <stevehite@xxx>
> To: <amibroker@xxxxxxxxxxxxxxx>
> Sent: Wednesday, September 12, 2007 2:00 PM
> Subject: [amibroker] Re: Is 500,000 the maximum number of bars
allowable in a database?
>
>
> >
> > And this is my biggest pet peeve about Amibroker:
> >
> > If I have the disk space available, Amibroker should never, ever, EVER
> > just silently blow away my historical data because of some arbitrary
> > system limit like 500,000 bars or 1,000,000. If I want 20 million
> > bars historical and I have the disk space to store it, then fine. If
> > Amibroker needs far less to function well, then that's fine too.
> >
> > The solution is NOT to delete old data. The solution is to have a way
> > for Amibroker to limit the number of past bars (for calculation
> > purposes) or even have me set a beginning and end date in an
> > historical range WITHOUT destroying that historical data (i.e., the
> > present day should not have to be the last bar that I view or use for
> > backtesting).
> >
> > Currently, the only solution is to set some registry value to go over
> > 500,000, but that in itself is not solution. Because of Amibroker's
> > implementation, you'll get your memory stressed out big time and,
> > again, if your new limit gets reached, Amibroker will remove your
> > oldest data to make room for new data.
> >
> > No one ever said that you had to look at stock/futures data from the
> > complete past to today. If I had 20 years of one symbol's data and
> > that data was far too much to make available at one time because of
> > memory problems, then it should be segmentable without me having to
> > split of the data in artificial symbols.
> >
> > Of course there's a workaround (there usually is). You have to split
> > your data up (e.g., tick data) into segments with different artificial
> > symbol names (e.g., I use eSignal continuous futures contracts).
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, Graham <kavemanperth@> wrote:
> >>
> >> I believe the database limit is set to 500,000 but can be changed in
> >> the windows registry. TJ has mentioned this in a previous post here,
> >> somewhere sometime.
> >>
> >>
> >> --
> >> Cheers
> >> Graham Kav
> >> AFL Writing Service
> >> http://www.aflwriting.com
> >>
> >>
> >> On 12/09/2007, Ara Kaloustian <ara1@> wrote:
> >> >
> >> >
> >> > You can have as much as you want, but performance degrades.
> >> >
> >>
> >
> >
> >
> >
> > Please note that this group is for discussion between users only.
> >
> > To get support from AmiBroker please send an e-mail directly to
> > SUPPORT {at} amibroker.com
> >
> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> > http://www.amibroker.com/devlog/
> >
> > For other support material please check also:
> > http://www.amibroker.com/support.html
> >
> > Yahoo! Groups Links
> >
> >
> >
> >
> >
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|