A technique I've been using to deal with large intraday historical
DB's is to store the data in a SQL database, use the ODBC plugin (with
some custom mods) and use a SQL stored procedure to fetch the data and
deliver the bars to AB. Then you can control how much data AB "sees"
(has access to) by changing a variable in some SQL table that your your
stored procedure uses indicating the number of bars/days you want to
have delivered. The system I created for doing all of this is too
complicated, proprietary, and valuable to give out in code, but that's
the general idea....
You need an adjustable layer of abstraction between AB and the data
source, and that's how I handle it. In my case, this layer of
abstraction, handled by the stored procedure in the SQL database, is
also capable of delivering the desired size of the bars... I have SQL
bar building code that creates different tables for different bar
intervals from the 1 minute data, and rather than leaving it up to AB
to assemble the 1m bars into, for instance, 5 minute bars (which
burdens AB significantly for a large 1m database), I use my prebuilt
QuotesMin5 SQL table. That simple thing allows me to work with 5 times
the history with 1/5th the memory/CPU requirements, as long as I don't
need to look into bars smaller than 5 minutes.
That kind of thing.... So maybe that sparks some ideas for y'all.
Certainly not an easy solution for the masses.... but the determined
few will always find a way.
-Longstrangest
One man's dream for AB: If only AB itself would use multithreading/multiprocessing
to assemble bars on time intervals different than the database (or
using multithreading/multiprocessing for *anything* for that
matter).... Then maybe the other 3 cores on my quad core CPU would get
some use. Still scratching my head and wondering why AB maxes out one
core constantly, and never uses the other processors, in this day and
age where the only significant performance improvements we've seen in
CPU power has been achieved with multiple CPU cores on a single die.
Until AB becomes multiprocessing-aware, I'll be forced to write
code that duplicates AB functionality, cave-man style, like bar
building, that I can run in my own separate thread. IMO, there's no
better platform than AB.... but if there's one single monstrous
performance improvement AB can make, it's taking advantage of
multiprocessing.
<drums fingers on desk, taps foot>
Dream on.