|
PureBytes Links
Trading Reference Links
|
There have been several posts recently regarding intraday tick data. Many
of the questions/issues deal with how to get missing data. But, I have not
read the discussion about what the data is used for. I assume that the
intended use is paper based chart analysis and/or system development and
testing. In my experience the tick data obtained from from BMI (and I
suspect most other data providers) is not suitable for those purposes.
Let me explain my view based on my experiences and observations. I've been
receiving BMI satellite data for four years. I watch, monitor, and
intraday trade full time. Some of the problems I've had are:
I - Power outages with obvious loss of data
II - Hung computer system - forcing reboots and loss of data
III - BMI not getting data from the exchange - resulting in loss of data
IV - Satellite blocked by bad storms (Tornado the NWS called it) -
resulting in loss of data
V - Defective Receiver - 2 days to replace
VI - Satellite receiver toggling between (LOCK and UNLOCK) every few
seconds for 2 days. I never did verify if this caused a loss of data but I
expect that it did.
Recently I have begun to develop systems for intraday trading the SP. But
any attempt to run these systems against a month or two of data always
produces invalid results due to trades taken or trades skipped, due to the
missing data.
An interesting observation I've had is that the BMI data one receives from
Satellite contains more ticks than BMI data received by cable (and I
suspect FM). One indicator I developed uses the tick count, as a surrogate
for trade volume. When This indicator was run on a friends system, which
is fed from BMI cable, we got different results. In tracking down the
source of the problem I compared the tick file from cable to the tick file
from satellite. Fully 80% of the 3 min bars had different tick counts and
2% of the price bars had different price Highs or Lows. I've been told
this is due to the cable providers use of bandwidth. His primary business
is TV and data gets a smaller piece of his bandwidth, which sometimes means
the data doesn't come through. I can't vouch for the technical correctness
of this but the data files ARE DIFFERENT.
If you are getting data from a friend or fellow RTer to fill in gaps in
your data, you might want to consider its source, Satellite, Cable, or FM.
By the way does any body know which way Omega gets the data they put on the
web?
I've had these problems getting data from the Omega web site:
I - Data in wrong format to be read by my Omega product (Support said, Oh
well)
II - The entire day's file missing - Oh well! - again.
III - For daily data, not tick data, I've found multiple days missing per
month! 2 days missing is approx 10% missing data (Oh well again.)
I've had these problems getting data from the Omega Historical CD:
I - All of the SP data is absent. (Support said we just put out what Dial
Data gives us)
I had hoped for more from Dial Data, but I suppose Omega got a price break
on the data for the CD. Like the saying goes "you get what you pay for".
I have not spent much effort in trying to resolve the Omega data issues.I
just decided that Omega is not in the data business, and I can't expect
them to be sensitive to data issues.
Another issue to be concerned about is what to do with bad ticks, how to
know if it is a bad tick? If you look at a chart and one price bar goes to
zero then it is a sure thing that is a bad tick, but what about more subtle
"bad ticks", the ones that are still in the day's range. Just enough to
"trigger" an order. In a system test it just produces a trade that
should'nt be there. But it happens in live trading too. More than once I've
had a price alarm go off indicating that a stop order was just hit, opening
a position for me. I immediately call in a stop loss order which is filled
and I take my "Loss", only to find out later that the original order is
still not filled because the market never got to the stop price! Sure,
that "Loss" turned into a profit but it points out some of the difficulty
in getting tick data that can reliably be used for system testing. If you
watch a market full time it does not take long to see "bad ticks". It is
easy enough to delete or edit the bad tick, but how do you deal with bad
ticks in those markets where you collect the data but don't personally
watch them all day?
If I continue to develop systems I will need a source of intraday tick data
I can trust. I've wondered about data from Tick Data, but I'm highly
suspect theirs will not be much better than what I have collected from BMI.
If Tick Data does the appropriate data correction, and verification
against the exchange's official record it would be worth the price.
In the bottom analysis I want to Trade, not manage data. It is clear to me
that system testing must have good clean, accurate data. It is also clear
that getting that quality of data for intraday ticks is not an easy matter
to accomplish. I wonder if anyone has compared intraday tick data between
multiple sources looking for differences. Are the differences significant
or minor? How would such differences effect system testing? I would like to
here anyone's experiences with these data issues.
At this time I am most interested in the data from CQG. As I understand
it, they correct as many bad ticks during the day as they can, and then at
night after verifying their data against the exchange's record they
retransmit all data to assure that what you have is clean, accurate, and
complete. This will be good for trading, especially in markets you can't
watch full time. But it doesn't help in system development, their data is
in proprietary format and cannot be accessed by other systems.
Can anyone verify my about view of CQG's data?
Tom Quinn
|