I have done exactly this recently; but with only about 5years of intraday 1min data. In the end once the 5year database got loaded (the initial database creation took a while) i started using that database instead of the five 1year databases and used walkforward settings to set the backtest periods to 1year intervals. Backtesting did not seem to be slow for me on the bigger database so it worked out well.
This saved me a bit of time since i dont need to switch between databases etc.
BTW, you mind revealing where you bought your data from and how much it costs; i am myself contemplating buying some futures 1min data from http://disktrading. is99.com/ disktrading/ #data1. The cost seem pretty reasonable and i will have 7 more years of intraday data than i currently have.
thanks
-gariki
--- In amibroker@xxxxxxxxx ps.com, "Mike" <sfclimbers@ ...> wrote:
>
> Kieth,
>
> I will admit that I have not used it. But, wouldn't a single database using QuickAFL solve the problem for you?
>
> Mike
>
> --- In amibroker@xxxxxxxxx ps.com, Keith McCombs <kmccombs@> wrote:
> >
> > I recently purchased 1min historical data, in ASCII format, for system
> > development and backtesting. Some of it goes back many years and
> > therefor the files can be enormous. I have NO problem with how much
> > disk space they take up. However, I do mind how much time is used in
> > back testing.
> >
> >
One of my thoughts is to somehow break the data up into smaller time
> > periods, for example years with some overlap of November and December.
> > So I might have one data base that goes from 11/1/1998 to 12/31/1999 and
> > the next from 11/1/1999 to 12/31/2000, etc. This would allow/require me
> > to test one year at a time, which might also help me keep my "back
> > testing" separate from "forward testing" I might also consider even
> > smaller time periods such as quarters or months or weeks.
> >
> > Has anyone else thought about, or better yet, done this?
> >
> > Is there any efficient (time wise) way to implement this. I don't care
> > about disk space; it's cheap.
> >
> > Your thoughts and suggestions are appreciated.
> > -- Keith
> >
>