|
PureBytes Links
Trading Reference Links
|
There was a very useful bad tick filter posted to this list last year by
Randy Tareilo.
I am attaching the two post again below.
Bob Fulks
-----------------------------
Resent-Date: Sat, 6 Sep 1997 11:13:42 -0700 (PDT)
Date: Sat, 06 Sep 1997 13:02:54 -0700
From: Randy Tareilo <taracent@xxxxxxxxx>
Reply-To: taracent@xxxxxxxxx
Organization: Tarac Enterprises
MIME-Version: 1.0
To: omega-list@xxxxxxxxxx
CC: vibri@xxxxxxxxx
Subject: Re: Bad ticks
Resent-From: omega-list@xxxxxxxxxx
X-Mailing-List: <omega-list@xxxxxxxxxx> archive/latest/9400
X-Loop: omega-list@xxxxxxxxxx
Precedence: list
Resent-Sender: omega-list-request@xxxxxxxxxx
Patrick Mikula wrote:
>
> These questions are for real time traders using mechanical trading
> systems. How do you deal with bad ticks in real time? Wont bad ticks
> corrupt a real time mechanical trading trading system?
Hi Patrick,
Good Questions! Yes, bad ticks can destroy any mechanical trading
system, either intraday, daily, weekly, etc depending upon your data.
If the data you are collecting is not "corrected" by the data provider
or the data feed, then either a) you need to correct yourself, or b)
make your systems work correctly using bad data.
While you might do a lot of a), or your data vendor might do that, you
really must build your systems to work on bad data. If you are using
intraday data, this is a must. The worst possible system you might have
is one that fire's off a buy/sell recommendation, you take the correct
action, then you look at your chart or data feed and the whole issue is
a bad tick, not a true buy/sell recommendation.
There are several methods to filter bad ticks out of your data. The
best method is to do a backward statsical analysis of your data to
determine the maximum change between ticks of data. If the maximum true
change is for a stock like DELL of 3/4 point, then any tick that is
outside that range can be determined as a bad tick and not used by your
system. TS has a built in function in the Server system to highlight
data that is outside a "normal range" of data. This notice is brought
to your attention in one of the Servers data windows but you still must
edit the data field itself to correct or delete that data item.
If you are not using tick data, but lets say one minute or five minute
data, a similar trick can be used to preprocess the the data before you
use it in a system. Again, I like the idea that if the data point in
question is outside a range of expected values for that field, you
should exclude it from consideration.
Lets say a case in point, you are using five minute bars and the stock,
DELL, has a normal maximum move at five minute bars of not more than one
and 1/2 points. If you have built a system using only the highs of the
bar, you should preprocess the data with a range check before deciding
to use that data field. Look at the code below to see such an example
of self correcting data systems.
{code to determine new highs for a five minute bar look back over 20
bars)
inputs: medianrange(1.5);
vars: vhigh(0), vhigh20(0);
if high < high[1] + medianrange then (check for out of range high)
vhigh = high (high is ok, use as a normal high)
else begin
(we must decide what to use as a substitue for the bad tick,
should we use the open, close, or low plus medianrange for the true
high. anyone of these could also be incorrect as it could also be a
one tick five minute bar. Solution one could be to use the last
correct high. Solution two could be if the open and close are not
equal to the false high then use the higher of both of them.
A third solution is to use a combination
of both solution one and two which I shall do.
if close <> high and (close not the same as the high)
close > open then (close is the highest of the two points)
vhigh = close
else
if open <> high and (open not the same as the high)
open >= close then (open is equal or higher than the close)
vhigh = open
else
vhigh = vhigh[1]; (we do have a problem, bad
data, use last correct high)
vhigh20 = highest(vhigh,20); (find the highest high for
the last 20 bars)
etc, etc, etc for finding the corrected low of the bar
I believe that in writing and using intraday system, you must program
around the difficults of bad data and systems that do not self adjust.
If you don't, then your system will be UNTRADEABLE because if is taking
all of these bad commands to buy/sell. A bad system is worst than no
system because you might want to trade a bad system and fail.
Good Luck and Good System Trading
Randy Tareilo
Date: Sun, 05 Oct 1997 17:49:42 -0700
From: Randy Tareilo <taracent@xxxxxxxxx>
Organization: Tarac Enterprises
MIME-Version: 1.0
To: Brooks Rimes <72401.651@xxxxxxxxxxxxxx>
CC: ati-list@xxxxxxxxxx, omega-list@xxxxxxxxxx
Subject: Re: System Testing
Sender: owner-ati-list@xxxxxxxxxxxxxxxxxxxx
Precedence: bulk
Reply-To: ati-list@xxxxxxxxxx
Hi Brooks,
You wrote:
>
> <Likewise, if a 10 minute chart is the only method of building a successful
> system, then the true system testing
> must stay with 10 minute charts and only use the high,open,low, and close
> of the bar.>
>
> Even if you trade off a 3 minute chart, isn't it true that you want the
> entry off the 3 minute chart, but need to monitor and exit off what is
> essentially a 1 tick chart?
No, my understanding of how TS builds it's bars is that on a 3 minute chart
the bar is building and collecting the information for your trading system.
At the end of the bar (ie, close of the 3 minutes), you now have a full bar
that will run thru your code in your TS system. To establish your position,
you have some code to buy or sell based upon some type of "knowledge" or
"parameters" or "rules". (ie, buy on a moving average of 20 bars above a
moving average of 60 bars - simple and probabaly worthless!)
Now depending upon your exit method, you can issue a exit to close the
position on a stop limit order, or you can wait for the next 3 minute bar
to finish being built and then have it run thru your code again. Which ever
method you use, it has significate difference on how the system will
operate. If you have a trailing stop limit order or a profit stop order via
your system, a single tick can trigger that event to occur. And with the
level or lack of "bad ticks" in your stock (or whatever, options,
commodities, bonds, etc), you could place a order with your broker which
you really didn't want based upon the system. So after you are out of your
position, and you are studing what the system did for you, you see that
probably you should not have taken the order based upon your rules. Lets
look at a case under consideration and see how it would operate based upon
different rules, type of orders, and how the bar is built.
Suppose I have a long position in the stock INTC (Intel) and I bought the
stock at 95 dollars. My basic rules of my system says to exit the position
if the CLOSE of the bar goes below a trailing exit of the highest high
since I have established the position less 3/8 of a point. That is in my
example, the stock went from 95 to the following
open high low close
09:57 94.75 95.125 94.75 95.00 <== bought on the close - long
here
10:00 95.25 95.375 94.75 95.25
10:03 95.375 95.5 95.375 95.5
10:06 95.5 96 95 95.75
10:09 95.75 95.75 94.75 95.50
While Intel is very active during this time period, all stocks normally
have the largest range during the first hour of the day. So lets see
how my basic system would work on a close of a normal 3 bar high less
3/8 point.
At the 10:00 bar, I close at 95 1/4 and my exit is at 95 3/8 less 3/8 or
at 95. Since I am above the exit, I am still long the position.
At the 10:03 bar, the highest high so far is 95 1/2 less 3/8 point or
exit at 95 1/8, my close is at 95 1/2, so I am still long.
At the 10:06 bar, the highest high so far is 96 less 3/8 point or
my exit at 95 5/8, my close is at 95.75, so I am still long.
At the 10:09 bar, the highest high so far is still 96 less 3/8 point or
my exit at 95 5/8, my close is at 95 1/2, so I have a sell to exit the
position at 95 1/2. So it looks like I had a profit of 95 1/2 less
entry position of 95 to earn 1/2 point for 12 minutes of trading. If
I trade 1,000 shares less commissions of $30.00, then I have a net
profit of 475.00.
But looking at the data, you will see that using that while I was
trading, the range between the high and the low prices expanded from
3/8 point to a point (ie, 96 less 95). Are these real prices, and if
so, how do they affect my exit method if I use a tick method with limit
orders.
With a profit exit of the highest high since the start of the position
less 3/8 point, I would have exited my trade on the next bar after my
entry at 94.75. My profit exit would have been established at the
high at 9:57 (95 1/8) less 3/8 point for a stop at 94 3/4. The low
of the 10:00 bar hit that price and I exited the position.
The difference between those two trades is the difference between trading
on the indiviual tick data and trading on the close of the bar. The tick
data will trigger your exit at the low of the bar, and so any exit that you
program on a 3 minute bar should use the low of the bar if you decide to
trade tick by tick on the 3, 5, 10, or 30 minute bars. It only takes one
tick to create a exit via a minute time frame bar.
I have traded this way and lost money because of this feature of measuring
every indiviual trade in the market place. Some ticks are more important
than other ticks, and certain prices are more important than other prices.
Likewise, at this level, the size of the tick is not taken into account to
determine if it is a "important" tick or price.
A another way of stating this is, if you are so close to the stock that
you are counting the limbs in the tree as it falls, you will fail to
notice that it is falling towards you which is a sure way to be pounded
into the ground. Likewise, this detail inspecting stock tick prices is
not a true indicator of which way the stock is headed.
So, my method of trading is to use the smallest time frame that is reliable
to measure the direction of the stock without getting caught up in the
detail of the stock information.
> What do you do about bad ticks, both in your historical data as well as
> your realtime feed? Is there software to correct it?
There are 6 or 7 ways we can deal with "bad ticks".
1) Ignore them! (more on this later)
2) Program your system to work around them
3) Have the data vendor correct them automatically
4) Hand correct the data yourself
5) Have a Indicator routine to scan and report bad ticks via a report
6) Have the server report out of bounds ticks to rework
7) Have several different data feeds to compare data and to
throw out the bad ones
Likewise, I believe it is wise to have actual "dirty data" that is
provided by your data vendor to do system tesing against. If you
have a million dollar a month winning system, but is becomes a million
dollar month losing system with bad ticks, I want to know about it.
Of the seven items above, I would rank them in this order of how
I perfer to process the data. First, number 2, program around the
data and clean it up on the fly. Second, number 5, have a indicator
to search for bad ticks and to report them via a log file. And last,
number 4, correct the ones that matter and are so far out of sight
that they make no sense to retain in your data file.
A recent item that I have been noticing over the last month or two
is the number of "bad ticks" that are really not bad. Recently, I have
been noticing big blocks of shares trading above or below the most
recent close by one or more points. I believe that these are
mutual funds wanting into / out of a position and willing to
buy / sell a position above / below the market --- JUST TO GET
THE TRADE MADE.
Is this a sign of a top??? If a Fidelity Fund has a million shares
of Intel that it wants to unload without affecting the price 2 to 4
points, how do they do it. They find a willing broker or firm with
deep pockets that will take the order, buy / sell for the company
account, and use the rest of the day / week to unload the position.
When the market is strong, the mutual fund would just do it in the
market without fear of price decay. But in topping or thinning markets,
is this a better method?? If every mutual fund has enough Intel and
are not willing to buy more, who do you sell it to?
>
> <For as much as this affects the systems results of tradestation, the Omega
> Research folks attempt to play back the real ticks of each bar via the
> "bouncing ticks" within a 10 minute chart. So if stop orders are placed
> after a entry, and the stop is hit within the same bar as a profit point
> exit, then you must accept the results as a test with a) a loss, and a test
> with b) a gain.>
>
> Could you take another crack at explaining this. I'm not following,
> especially the bouncing ticks.
If you look in your data server files for any of your tick data, you will
see ticks that arrive at the server with the same time stamp with different
prices. The way that they arrive at the server is not necessary the way
that historically the trading took place. Since none of the exchanges
require seconds on the ticks, then if you have 100 ticks at 10:07, how do
you work thru them to find how they will really affect your system. Omega
has attempted a work around on this problem to address how the ticks would
affect your system trading. If withing your 100 ticks you have 50 at 95 1/2
and the other 50 at 95 3/4, then it would be expected to have the price per
tick bounce between the two prices as the difference between the bid and
ask prices. Since the true sequence is not captured any where, how do you
replay a market using historical data to test a computerize trading
system??? Omega has a unique solution which attempts to address this
problem and to really test your system. It becomes even more difficult when
there are multiple prices with wider range of price at the exact same
minute in time. So can anyone outside of the exchanges provide true tick
data that is reliable (ie, including corrections, cancelations, late
reporting, and broken trades), I believe not. As with any large scale
programming project that is used in the business world, trading for a
living has some unusall aspects that involve trading methods.
(PS, what do you do when your data provider gives you the following data
set for Intel at 10:00 --- Open 95, High 95 1/4, Low 94 3/4, and CLOSE 97
1/8. The close is not within range of the high / low but it is reported as
such by the data provider.
>
> <One simple solution during your trading is to have two data plots in your
> graph, one which is a one minute chart and a ten minute chart
> (hidden). Have you ever done this?
>
Yes, it is messy and takes up a lot of resources, but it can be done.
Another solution is to build ten minute indicators on top of a one minute
chart using one minute data. It again can be done, is a way to clean up
data, and to project prices.
Good Luck and Good Trading
Randy Tareilo
PS: You guys on the ATI list should discuss this to determine if you
have a better solution than what I have discovered. Since mine is from
the school of hard knocks, a better method may be out there???
|