|
PureBytes Links
Trading Reference Links
|
a few minor problems along that line are:
- often bid and ask are crossed
- some legal trades do happen outside of bid and ask
in fast market trades sometimes happen way above
the ask and way below the bid...
however, there are workarounds to those problems.
i told omega long time ago and repeat again:
if they don't want to THOROUGHLY scrub data within the server
( not only giant bad ticks but also smaller outside trades )
or don't have the technology or the capacity to accomplish that then
there is actually a much better solution to the problem...
- supply raw ( dirty ) data to the end user in real time.
- have a button on symbol format form called "enable tick filtering"
- supply bad tick tags to the end user in a separate data packet
- a typical user has what? about 10 charts running in TS, so instead
of filtering all data within the server, filter data in the user platform before
it is sent to a chart, that way you are using the user computer recourses,
not server recourses... and it's not much of user recourses either...
- the only catch is that they have to feed bad ticks with tags
and id those on the end user machine ie in TS platform... and they have to maintain
clean historical database ( by running clean-server routine against the
bad tick repository at night ) that's all.
- it is much easer to implement compared to supplying bid and ask
data, filtering based on bid / ask or doing filtering within the server...
SO THAT IS THE CURRENT BEST ALGORITHM FOR BAD
TICK FILTERING:
- do not filter data within the server in real time
- instead supply raw data to end user as usual in real time
- all bad ticks that come from the exchanges tagged, drop em all into
bad tick repository.
- bad ticks with tags will be send to end user along with good data
and filtered based on those tags using user resources, before data
is used in a chart. the fact that the user has the ultimate choice
on whether to enable filtering or not ( by checking that button )
the vendor is not responsible for missing or corrupt data if that
button is check ( should not be a problem anyway )
- at night run the server maintenance and either delete all tagged ticks
that are in bad tick repository or back insert them into the data at
correct times.
as far as bid ask data, they already pass bid and ask data for L2 boxes...
but i think the reason they don't give you bid ask bar data is
because they are not willing to store and maintain it as bar data
on their M3 server...
even if they did than coding a bid ask filter is a real clusterf...ck
i have done that, this was the hardest coding i have done in EL so far...
it will have to be coded as a DLL, AND coded real well otherwise
funny things may happen...
anyhow, they'll never pull it off and it's not needed...
TS is not there yet...
your system is as good as your data... if data is crap who cares
about how fast you can deliver it or how much of it you can store???
i'd rather wait an extra 5 seconds for clean data than
get shitty data in .1 of a second...
TS people assume that we are stupid and would not understand
the concept of an outside trade... and thus they can claim that their
data is clean... in reality it ain't that clean!!!
bad tick can easily trigger an unnecessary trade...and that's a liability
that they think we won't figure out...
M3 people understand bad data, and they do pass those bad tick
tags ( if available from the data vendor )
to the end user to handle, but they hide the issue and
their solution is not solid from what i can gather...
bilo.
ps. the new high-tech platform that's in the works supposedly takes care
of the issue... will have bad tick repository and on the fly tick filtering,
bid ask data for the available issues... and super fast access.
that is my hope. when that happens, say good by to good old Omega.
if TS keeps at this pace the autoexecution won't save their ass...
custom autoexecution has already entered main stream...
----- Original Message -----
From: "Gary Yakhnes" <gary@xxxxxxxxx>
To: <omega-list@xxxxxxxxxx>
Sent: Wednesday, February 27, 2002 8:21 PM
Subject: Re: Bad ticks...
> I think it would be a nice TS feature to have EL access to Bid/Ask values and ability to plot Bid or Ask prices on the chart along
(or instead) with Last Price. This would allow one to apply analysis to Bid/Ask values instead of Last Price, eliminating most of
the issues caused by bad ticks without applying any bad tick filtering. Using Bid/Ask values instead of realistic Last Price could
add some error for large-spread issues but will most likely not affect system results for
> low-spread high-liquidity issues in my opinion. I am just wondering why this functionality is not part of TS or maybe I am
missing something here...?
>
> Thanks,
> Gary
>
> > Subject:
> > Re: Bad ticks...
> > Date:
> > Wed, 27 Feb 2002 13:48:54 -0500
> > From:
> > "Bilo Selhi"
> > To:
> > , "Omega Technical Support" ,
> > "Tom Nielsen" , "Omega List"
> >
> >
> > ...
> > outside trades are mostly trades that are reported a few minutes 1-3 late,
> > mostly block trades, sometimes corrections to previous trades,
> > rarely straight outside trades ( after 1/16 spread removed ).
> > that's why they mostly show up on 1-3 min time frames.
> >
> > those ticks could easily be filtered out by data vendors or in servers
> > because such trades come tagged as late trades.
> > they will trip your system if not handled properly, it adds considerably
> > to noise in price series.
> > none of the data vendors i know filter those, they are just too lazy to do it, plus
> > it would slow their servers considerably. they just don't care...
> > however some system trading firms are well aware of the problem
> > and do filter those out.
> >
> > naz equity data, new your futures data, amex data are contaminated with
> > those bad ticks especially on active symbols. just look at 1 min csco data.
> >
> > you can solve the problem by coding a special filter to monitor probable spikes
> > in high frequency data and filter those out which is a difficult task,
> > or you can step up to lower resolution data such as 5 minutes where
> > those spikes are hardly visible.
> >
> > aside from outside trader you can have other larger bad ticks caused
> > by digit shifts, missing or "0" prices, missing data, other data delivery errors.
> > these are easier to filter because they are highly visible. Tradestation
> > does filter those out, manually :-) , but you can not rely on them either, they miss those
> > frequently.
> >
> > ***bad tick filtering is #1, i repeat #1 feature that the data server should have...
> > ( in view of trading system development )
> > now let's look at the M3 server and see if they understand it
> > http://www.uniserv.com/gen_success_stories.htm
> > check out the server that TS uses, there is no mention of bad tick filtering
> > in their general server description...
> > http://www.uniserv.com/gen_features.htm
> > and the finally, about how M3 handles bad ticks is here, if we
> > dig a bit deeper...
> > http://www.mktstream.com/wwwthreads/showthreaded.pl?Cat=&Board=GenSDK&Number=79&Search=tru
> > e&Forum=All_Forums&Words=filtering&Match=Entire%20Phrase&Searchpage=0&Limit=25&Old=allpost
> > s&Main=73
> > and
> > http://www.mktstream.com/wwwthreads/showthreaded.pl?Cat=&Board=GenSDK&Number=76&Search=tru
> > e&Forum=All_Forums&Words=filtering&Match=Entire%20Phrase&Searchpage=0&Limit=25&Old=allpost
> > s&Main=50
> > pretty much sais it all: data filtering is left to the third party developer ( such as TS )
> > , it can't be
> > relied on, M3 needs help with filtering, filtering takes up space and server time +
> > betcha no one is willing to help because it's issue of a substantial edge in
> > the overall game... ( if i have clean data and you have bad data then you will have more
> > errors in signals than me, hence i will not tell you how to filter it out :-)
> >
> > bad tick filtering is a problem in general is a problem with either the
> > exchange, server or delivery.
> > bilo.
> > ps. since most third party users are data vendors and
> > discretionary traders and
> > not actual traders / system developers they
> > careless about issue of clean data...
> > they are not on THE level yet...
> >
> >
> > > Tom
> > >
> > >
> > >
> >
>
>
|