|
PureBytes Links
Trading Reference Links
|
Historical data especially intraday is hard to find,
even these days.
Nobody wants to sell it because primarily of the
contractual obligations with RT datafeed providers.
But some sources are:
Stocks: NYSE TAQ database: raw L1 tick stock data
( not clean / scrubbed ), covers NYSE, NASDAQ, AMEX and
regional exchanges, starting from 1993 and on. Huge and
expensive. Monthly updates. No tech support. if you
are serious about historical testing of stocks systems
get this database, but be ready to scrub and scrub and scrub
and compile your own bar data. Priced high, available from
1993. By providing this database NYSE took out any
prospective competitors on the market.
Used it myself.
Omega: free for Omega users, refresh data,
omz database on there FTP server for bmi and signal, fs.
raw data for stocks and futures, many days are missing,
bad ticks. This databases have been the only data source
for many traders since about 1996. Be ready for long
downloads ( cable modem or better recommended ),
long paste in time( about 12 hours per 500 symbols, per month
per P400/64 ). Historical data has been Achilles hill for Omega
for a while. OK you bot TS spent 2500 bucks, you want to test this intraday
system you bot but where can you get data from? I am sure
a lot of TS users will back me up that that omz database was the
only data source to turn to. Had Omega been a bit smarter they would have
had their own real ticker plant by now and selling tick data left and right
making
probably more money than selling their software. Hopefully they finally got
the clue with this historibank which bring us to:
Historybank: data only available for two weeks to fill them gaps you
got in them charts. So if you got them gaps, download them fast.
Problems include: raw data, bad tick, minor gaps and most of all
long, long downloads. Connection dropouts are common.
It looks like they are still using the omz format
so the paste in time is long. I recommend at least cable modem or higher
and downloading every night to fill any gaps.
Used it myself.
Others:
TickData: futures tick data in their own format with their own app.
Bad ticks are common, so is simply bad data.
Genesis: same as above, different format.
Those don't carry stock data, been promising for years...
Those two have been at it for a while, one claiming their format
is better than the other. The truth is their data isn't that impressive.
Tried it, dealt with it.
CQG ascii database: the best overall intraday database.
Data is clean and but a bit pricy. Large files.
Tried it.
that's about it. The recommendations are:
if you are serious about designing a tradable system for stocks,
have money and programming skills, then get NYSE TAQ.
Scrub it, massage it, compile the bar data and you'll get the best
data there is.
if you dont want to scrub and have money and just want clean historical
data then get CQG ascii.
if you don't have money but want any tick data and can scrub it, grab
omega's database.
Now here is the advice on your intraday database management:
the best overall way of handling your data is this ( if you are serious
and up to it ):
1. understand that RT data and historical data are two different animals.
historical data is used for system testing whereas real time data is
used for trading the systems you tested on historical data.
Those two should not be mixed in most cases.
RT data is dirty in most cases ( except FS )and will likely to stay that
way.
Historical data, in comparison needs to be as clean as possible.
Now, some will say here that no, you need to test on dirty data.
That's not the optimal approach. It used to be but not anymore.
Given the growing of the markets and increased ticks counts
there are more bad ticks than in the past.
The best approach is to test
your system on good data, right a module to scrub rt data on
fly and then trade your system on scrubbed rt data.
This dictates the best way to manage your database:
2. Don't try to collect too much tick data in your server.
Instead: Create ASCII database by exporting your bar data
from TS server. Write a clean and dump routine in EL to
do that. Keep appending to that database as you collect
RT data on a monthly, quarterly basis. That way you
have ascii dbase, which is independent of your platform,
it's clean and ready to be used for historical testing.
3. Once the system is tested on your ascii data, then trade
it on your RT data. Make sure you use data scrubbing
function, that you have to write in EL. After appending
ascii data to your database, delete tick data from your
server. Leave just enough to get your system going.
Keeping your intraday data in ascii format will prevent you
from tweaking tick data in TS server, losing it altogether,
managing your portfolio, etc.
4. Keep building your ascii database thru the years.
Even if you change datafeed providers you wont have
to recompile your tick file in TS. That makes your data
independent of the datafeed provider format, TS formats,
or even if you switch to another trading software. most
of them read ascii data. Best bar interval to use for
the intraday ascii database is 1min. Files are going to be
huge but nothing you can do about that.
good luck.
Bilo.
Omega: if you are reading this here then read on
1.whatever format you pick for your next tradestation.com
ticker plant, for god's sake, CLEAN YOUR DATA!!!
filter out those late trades, i mean how hard is it, every
freaking tick that comes in from the exchange has a flag on it saying:
"I am a good tick, take me, please"
"I am a late tick, delete me, please"
"I am a bad tick, kill me, please"
"I am a block trade reported late, avoid me please"
"I am a digit shift, flush me, please"
If you can't then pressure your datafeed provider to
filter those or pass those flags to your server.
Look at FS, they've been doing it for years.
How hard is it? If you clean data on the fly, you got it,
do you understand the words that are coming out of
my keyboard??? KILL BAD TICKS.
2. Fill those gaps, too. Write a little routine in your TS server
that will alert traders that the datafeed is out!!!. At least
give us that much. Then allow alert traders at the end of the day
at least to fill those gaps. FILL THOSE GAPS.
3. Get into the intraday data selling business and become the best.
Sell, clean, gapless tick data or at least 1 min bar data and
you'll make more money than you are making selling your
software. CLEAN and GAPLESS are the words. Look at
CQG and FS, it's doable. You been shying away from data
but that was a mistake...
Can't you understand that without clean data your TS is just an
expensive toy and nobody is going to do it for you.
PROVIDE CLEAN INTRADAY DATA. make mo money.
4. Look what's happening in the EOD data providing business.
Reuters, the biggest EOD provider cant handle it anymore.
I mean, it's like saying, Omega, get a clue, get in the business.
Get in bed with, CSI or something, i mean they are down the road from you.
Wheel and deal, make people happy by delivering the best that
you can, not just what you can. and for gods sake don't go in
bed with d. they don't know what the hell they are doing,
and nobody knows where they pull their data from. some "new high tech"
ticker
plant somewhere in Hillberry, CA. PROVIDE CLEAN EOD DATA.
Omega, your TS is almost good, but you got no good data.
Your historybank is a joke.
GOOD TS + GOOD DATA = success.
a gun but no ammo.
|