|
PureBytes Links
Trading Reference Links
|
In addition I'd say that the problem of cleaning tick data is especially
tricky. As much as I would like to, I don't currently develop systems on
data that's been cleaned because that's not the way my sever will get it and
my system will see it (TS4/TS2K). I design my systems to run on data the
way my system will see it. For RT stocks that means I can't use the highs
and lows of the bar because often they're distorted. For some futures, I
can do it because the data that comes from the exchanges (NY is the
exception) is already pretty clean.
You have to back test on data that is representative of the data you'll get
real time. That means, storing (ie exporting) the data and then running an
algorithm to clean tick data is not a good approach. It's not good because
it permanently distorts the actual tick data. This is acceptable if you
plan to use the same tick cleaning algorithm forever but I like to make
improvements as time goes. This means that historical data will get cleaned
(better) or differently which may affect system performance.
The problem with having someone else clean it is that they may change the
cleaning algorithm without telling anyone. This could affect system
performance in the future. Would a company like Omega tell you when
they've changed their tick cleaning algorithm? And would there be enough
history cleaned with the new algorithm for you to backtest your system?
Worse yet, they may glue the historical data cleaned with old to the current
data cleaned with the new algorithm. Nothing is representative of anything
at this point. But you don't know this. We all know that small changes in
the data can lead to changes in the system results. In other words, data
from BMI will often not generate the same results as data from DTN given the
same system because the data is different (not as many ticks, etc. -- which
is essentially what a tick cleaning algorithm does -- throws out the bad
ones or adjusts them).
What is needed is a platform that allows a small, modular, and customizable
(C++) tick cleaning algorithm that sits between the server and any part of
the program that requests data. Tick data is stored as is from the data
vendor but cleaned when charted or run with your system. When you want to
improve the how your ticks get cleaned, you replace the cleaning module and
rebacktest your systems.
B.
-----Original Message-----
From: Bob Fulks [mailto:bfulks@xxxxxxxxxxxx]
Sent: Saturday, September 01, 2001 12:16 PM
To: gary@xxxxxxxxx
Cc: omega-list@xxxxxxxxxx
Subject: Re: NeoTicker (was Re: EL to VB translator)
At 10:44 PM -0400 8/31/01, Gary Yakhnes wrote:
>TS2K=TS PRO + Third-Party Data Support
Your premise is an interesting one. I think it is more than just that.
I have not tried TSPro yet. (I have learned from experience that no
Omega product is worth even looking at for the first 12 to 18 months
after it's release.)
I also have concerns with their data-on-demand model, the product
quality, and the requirement to use their brokerage services..
Data-on-demand would be ideal if it really worked. No one likes
having to maintain their own data but there is now no choice if you
want lots of accurate historical data for backtesting. An ideal model
would let you collect real-time data for trading, store the data on
your machine locally, fix bad ticks as they occur, and be able to
download years of accurate historical data and store it on your
machine for system development. The ability to download intraday data
to fill holes is essential. The present combination of
DynaStore/DynaLoader plus QFeed is close to this model and works very
well.
A stock has 390 OHLC 1-minute bars in a day that would be easy to
download. Assuming 10 bytes per bar this would be 4000 bytes per day
per symbol. 100 symbols would require 400K bytes which is a few
seconds on a high speed connection. Similarly, 100 days of one symbol
would require a few seconds.
Bad ticks - I cannot see how they can avoid bad ticks screwing up the
data. When we have the data on our workstation we can at least fix a
bad tick. It is unrealistic to think they will fix them in a timely
manner for the many thousands of issues. QCharts has this problem.
The only viable model is to have the data on your machine where you
can fix any bad ticks yourself.
Server Loading - It seems fundamental that any model that requires
server capacity to do work on demand will bog down eventually. To
avoid slow performance during peak usage periods, such as when the
market opens, the capacity has to be probably ten times the average
load and management usually does not supply that kind of excess
capacity. This was a major issue with QCharts. But if you stored the
past data on your machine, the load on the servers would be very much
less and more constant. If you looked at some new symbol, you would
have to download all the data required for that new chart but these
requests would occur more randomly.
Quality Issues - I also have serious concerns about the quality of
Omega software. it seems to be getting worse with each new product.
TS4.0 is pretty good but it took them until about Build 16 until it
was even usable. TS2000i was poorer with lots of annoying bugs but by
Service Pack 5 it was usable. And TSPro ??? the trend is bad.
I have heard that they took out the divide-by-zero checks and code
that runs fine on TS2000i does not run on TSPro. The management
obviously has no appreciation for the need for quality so I wonder
whether they will ever be able to fix this. I am concerned that
software errors will cost me money in a bad trade. Building quality
software is no longer an art. The processes are now well known and
there are a lot of modern development tools to test software. There
is no excuse for shipping such buggy software today.
Brokerage Selection - TRAD seems to have the idea that people will
select their brokerage services based upon TSPro. People select a
broker based upon lots of different reasons. The user interface is
only one factor. Having a customer service person available who
answers the phone promptly, financial stability of the firm,
availability of mutual funds, etc., etc.. are all important factors
to some people. Brokerage services are rapidly becoming a commodity
and commission revenue is approaching zero. Before long, we will be
placing orders directly with the ECN's. Why anyone would want to
start a brokerage business today is beyond me. Ideally, TRAD will
work with other firms to provide a direct order entry linkage into
their order entry systems as well.
That way they would then have hundreds or thousands of salespeople
promoting their product. I would think it would be easy to establish
a more stable revenue stream based upon monthly fees for combined
TSPro/real-time data combination and fees from brokerage houses using
TSPro as a front-end for their order entry systems.
My thoughts.
Bob Fulks
|