I recently purchased 1min historical data, in ASCII format, for system development and backtesting. Some of it goes back many years and therefor the files can be enormous. I have NO problem with how much disk space they take up. However, I do mind how much time is used in back testing.
One of my thoughts is to somehow break the data up into smaller time periods, for example years with some overlap of November and December. So I might have one data base that goes from 11/1/1998 to 12/31/1999 and the next from 11/1/1999 to 12/31/2000, etc. This would allow/require me to test one year at a time, which might also help me keep my "back testing" separate from "forward testing" I might also consider even smaller time periods such as quarters or months or weeks.
Has anyone else thought about, or better yet, done this?
Is there any efficient (time wise) way to implement this. I don't care about disk space; it's cheap.
Your thoughts and suggestions are appreciated.
-- Keith