Re: [amibroker] Re: Backfilling with IB problems, AmiBroker Email List Archive

On Fri, Apr 24, 2009 at 8:31 PM, Keith McCombs <kmccombs@xxxxxxxxxxxx> wrote:

Ozzy --
Ok, I guess I should feel lucky that I'm not alone here.
Did you get your historical data from PiTrading.com or from some other source? If some other, may I ask who?
In any case, what did you think of its quality?

BTW, on closer inspection I discover that data between 2:59pm and 3:59pm is missing for many days prior 4/6/09 (displayed either by AB or IB directly). This may be a problem unique to my system (I'm running Win2000, and I know that it had problems dealing with the change in the dates for DST in the US).

Thank you.
-- Keith

BTW2, while trying to debug a trading system I was working on, I became suspicious of some large spikes up midday 10/10/08 in the IB intraday data for SPY. I compared it with a friend's TradeStation data -- no spikes there. I have another friend who is going to check his WLP data to see if it has the spikes.

Just because two (or more) data sources have different data, does that necessarily mean that one of them is wrong? I don't know. Perhaps their data is from two different sources, one of which has access to a different 'type' of data, for instance large block sales that are not part of the normal retail market?

I would appreciate someone, who knows better than I, commenting on this (and what if anything I can or should do about it).

ozzyapeman wrote:

I am running the same versions you are, and only a couple of times in 3 months of testing was I ever able to get a 180 backfill from IB. Once in a while I can get 30 days (though even those often get throttled).

I have resorted to just buying some historical data from a vendor, and then relying on IB to keep the symbols up to date with new data every day - only have to backfill a day or two, which is no problem.

--- In amibroker@xxxxxxxxxxxxxxx, Keith McCombs <kmccombs@xxx> wrote:
>
> 30 day backfill seems to work Ok.
> Has anybody tried a 180day backfill for 1 minute data?
> Success? or Not?
> Thank you.
> -- Keith
>
> Keith McCombs wrote:
> >
> >
> > This is probably a cockpit problem, and its driving me bonkers.
> >
> > I just started using IB TWS (893.9) intraday data with AB5.20 and IB
> > plugin 1.8.2. I only have 8 equities in the data base. First, tried to
> > backfill all at once, with 180 days of data.
> >
> > Soon gave up on that and decided to concentrate on only SPY. I know
> > that I can't get the full 180days at once because TWS will throttle back
> > and so AB waits for 2 minutes and then tries to get the rest.
> >
> > Well the first run at, AB gets data from 8/5/08 to 11/21/08, then waits
> > 2 minutes and try again. I'm expecting that it will then start at
> > 11/22/08 and begin filling forward. NOPE, now I have only 9/18/08 to
> > 4/21/08 with most days ending at 2:59 instead of 3:59. Two minutes
> > later, start again, and now we have only 11/13/08 to 4/21/08, but with a
> > gap from 11/21/08 to 1/5/09. The longer it runs the worse it gets.
> >
> > Oops -- just checked and Number of bars was 32000. But I distinctly
> > remember setting it to 130000 for 270 days worth. So I try again, after
> > setting Number of bars back to 130000, and double checking it.
> > Now I have SPY with data from 9/26/08 9:30am to 1/6/09 4:00pm and AB
> > gives message saying it is done with SPY. I then try AAPL it eventually
> > finished with data from 8/28/08 9:30am to 3/6/09 3:59pm; and then from
> > 3/9/09 to 3/20/09 all data starts at 9:30am and stops at 2:59pm (that's
> > right someone stole the last hour of trading for 10 trading days (bet
> > that has something to do with DST starting on 3/8/09). No more data
> > after 3/20/09.
> >
> > Besides the above major problems I have some other concerns, namely:
> > 1. When and why should I use Save Database?
> > 2. Colored status box for CONN, etc. is sometimes blue and sometimes
> > green, meaning what?
> > 3. When I hover my cursor over that status boxes it sometimes displays
> > a message and sometimes not, and when it does it's usually for less time
> > than it takes me to read the message.
> > 4. With number bars set for 270 days, I'm hoping that I can get 180
> > days back now and that with time, I can have a maximum of 270 days of
> > data. Can I then increase Number of days to say 360 and keep adding to
> > the data base, or does data base get erased every time I change the
> > Number of Bars value?
> >
> > I hope there is some one who is not as overwhelmed by all this as I am.
> > -- Keith
> >
> >
>