|
PureBytes Links
Trading Reference Links
|
Hello Keith
Partitioning data into smaller periods consiste of 2 steps
1. exporting data from ... to a certain period. There are some afl lying around that would do the exporting to an ascii file. all you need is an extra if statement to export data only if it is within your desired data range and change the ticker name to one that will differentiate between different date ranges. you might also do some intergrity checking while you're exporting to make sure all bars exported are in chronlogical order (some time corrupted database arent).
Next is to report the data into a new database This is best acomplished with a script. you can also add imported tickers to a certain watchlist in the orders of data ranges.
Thats it.
When you backtest, you just move from one watchlist to the next as you move forward onto the next range. This can also be asisted with writing a script.
I hope that helps
/Paul.
--- In amibroker@xxxxxxxxxxxxxxx, Keith McCombs <kmccombs@xxx> wrote:
>
> Longstrangest --
> Thank you for your reply.
> I'm not about to attack SQL in the near future. However, its nice that
> I am not all alone with this problem.
>
> And your post below gives me some encouragement to try to attack the
> problem. I also like your suggestion for reducing AB overhead by making
> additional files with longer time periods.
>
> I'm thinking of perhaps a C++ program to produce the smaller ASCII
> files, both date limited period modified, for importation into AB.
> Notice that I said 'thinking', not 'writing', at least not yet.
>
> -- Keith
>
> longstrangest wrote:
> >
> >
> > A technique I've been using to deal with large intraday historical
> > DB's is to store the data in a SQL database, use the ODBC plugin (with
> > some custom mods) and use a SQL stored procedure to fetch the data and
> > deliver the bars to AB. Then you can control how much data AB "sees"
> > (has access to) by changing a variable in some SQL table that your
> > your stored procedure uses indicating the number of bars/days you want
> > to have delivered. The system I created for doing all of this is too
> > complicated, proprietary, and valuable to give out in code, but that's
> > the general idea....
> >
> > You need an adjustable layer of abstraction between AB and the data
> > source, and that's how I handle it. In my case, this layer of
> > abstraction, handled by the stored procedure in the SQL database, is
> > also capable of delivering the desired size of the bars... I have SQL
> > bar building code that creates different tables for different bar
> > intervals from the 1 minute data, and rather than leaving it up to AB
> > to assemble the 1m bars into, for instance, 5 minute bars (which
> > burdens AB significantly for a large 1m database), I use my prebuilt
> > QuotesMin5 SQL table. That simple thing allows me to work with 5 times
> > the history with 1/5th the memory/CPU requirements, as long as I don't
> > need to look into bars smaller than 5 minutes.
> > That kind of thing.... So maybe that sparks some ideas for y'all.
> > Certainly not an easy solution for the masses.... but the determined
> > few will always find a way.
> >
> > -Longstrangest
> >
> > One man's dream for AB: If only AB itself would use
> > multithreading/multiprocessing to assemble bars on time intervals
> > different than the database (or using multithreading/multiprocessing
> > for *anything* for that matter).... Then maybe the other 3 cores on my
> > quad core CPU would get some use. Still scratching my head and
> > wondering why AB maxes out one core constantly, and never uses the
> > other processors, in this day and age where the only significant
> > performance improvements we've seen in CPU power has been achieved
> > with multiple CPU cores on a single die. Until AB becomes
> > multiprocessing-aware, I'll be forced to write code that duplicates AB
> > functionality, cave-man style, like bar building, that I can run in my
> > own separate thread. IMO, there's no better platform than AB.... but
> > if there's one single monstrous performance improvement AB can make,
> > it's taking advantage of multiprocessing.
> > <drums fingers on desk, taps foot>
> > Dream on.
> >
> >
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
amibroker-digest@xxxxxxxxxxxxxxx
amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|