|
PureBytes Links
Trading Reference Links
|
I posted a reply from Ms. Blais on this matter some time ago. But for
those who might have missed it, a copy appears below as a "forward".
What I have not posted before are the following analyses of the problem
which were sent to Melody and which convinced Omega that this really was
a Server bug:
______________________________________________
First, let me dismiss two questions which I am NOT asking (lest they be
answered yet again).
I am NOT asking why a single bad tick seems to corrupt a large block of
consecutive ticks - I understand about the storage scheme involving
blocks of about sixty ticks with all but the first stored as offsets
from the first.
And I am NOT asking how to correct these situations - I have already
successfully corrected far too many.
What I AM asking is that Omega acknowledge that most (all?) of these
problems (sudden, radical jumps or drops in price which continue for a
while at that erroneous level and then abruptly return to correct
values) are in fact due to a Server bug which sometimes corrupts the
initial value of a block in preparing it for storage, and NOT to prices
which are bad as received from a data vendor.
First, here are the observed facts:
1. These problems occurred most recently on three different days last
week and, in all, involved roughly twenty block-initial "bad ticks"
(that is, twenty corrupted blocks, not twenty erroneous prices) for
$INDU. Several cases occurred in which as many as three or four
CONSECUTIVE blocks were corrupted.
2. In NONE of these cases did charts displaying the data in real time
show any problems. (I have on rare occasions in the past seen true bad
ticks, both bad prices and bad time stamps, and they cannot be
overlooked. A bad price produces a large, sharp spike up or down, and a
bad time stamp completely screws up the chart display.)
3. Many others also experienced these problems although, if I recall
correctly, some may have experienced them at slightly different times
and prices.
Various Omega spokesmen have so far insisted that these problems are, in
fact, due to ticks which are bad as received from the data vendor
(Signal in my case - but BMI subscribers reported problems as well).
Here, however, is what these spokesmen are, by implication, asking me to
believe:
1. The reason we did not see the bad ticks in real time is that even
though the Server received and stored these bad ticks, some mysterious
problem in all our computers caused EACH and EVERY bad tick to be lost
somewhere between the Server and the charting module.
2. Even though the chances AGAINST this happening are nearly 60 to 1 for
each individual "bad tick", EVERY ONE of the roughly twenty "bad ticks"
just happened miraculously to arrive at our various computers at the
precise instant when our Servers were ready to enter the initial tick of
a new block.
My final and principal question is: Does the above correctly describe
your official position? If not, then will you please provide an
explanation which accounts for all the above observed characteristics?
Thank you very much.
Carroll Slemaker
_____________________________________________________________
[following is a subsequent e-mail]
Thanks, Melody, for keeping me posted.
Here's an additional observation which has just occurred to me, one
which, it seems, will be much more difficult to explain as the
consequence of a bad tick from the data feed.
Assume that all the following references to prices refer to prices for a
single particular symbol, such as $INDU. Now allow me to define some
notation:
P - a price as received by the Server from the data feed.
Ch - a price as displayed on a 1-tick chart.
ET - a price as displayed in the Server's Edit Tick box.
DB - the value stored in the Server database for a price.
i - the ordinal of a tick from the feed - i.e., P[i] represents the ith
tick received.
j = i mod 60 - where 60 is the assumed number of entries in a database
storage block. Thus, as i counts up, j counts from 0 to 59 and then
recycles to 0 (the start of the next block). The value 60 is not
important; it is not even important that this value be a constant, only
that it be greater than 1.
Understandings (subscripts are all assumed to be j, not i):
DB[0] = P[0]
For j>0, DB[j] = P[j] - DB[0]
In real time, Ch[j] reflects P[j] rather than a value reconstructed from
DB[j] and DB[0].
Now consider an example sequence of received prices. The values shown
represent what we would see IF the Server behaved in accordance with the
above "understandings":
j_ P______ DB_____ Ch_____ ET_____
59 8210.25 xxxx.xx 8210.25 8210.25
00 8810.60 8810.60 8810.60 8810.60 (a bad tick at block start)
01 8210.85 -599.75 8210.85 8210.85 (8210.85 = 8810.60 - 599.75)
...
59 8225.60 -585.00 8225.60 8225.60
00 8224.10 8224.10 8224.10 8224.10
01 8224.90 0000.90 8224.90 8224.90
In other words, we would see a SINGLE spike up to 8810.60 followed by
correct prices; this would appear the same whether observed in real
time or in a reloaded chart.
Now let's assume the user corrects the bad tick. Here is what we should
then see, based still upon the above "understandings":
j_ P______ DB_____ Ch_____ ET_____
59 8210.25 xxxx.xx 8210.25 8210.25
00 8210.60 8210.60 8210.60 8210.60 (bad tick corrected)
01 8210.85 -599.75 7610.85 7610.85 (7610.85 = 8210.60 - 599.75)
...
59 8225.60 -585.00 7625.60 7625.60
00 8224.10 8224.10 8224.10 8224.10
01 8224.90 0000.90 8224.90 8224.90
Note that NEITHER of the above tables portrays what, in fact, we DO
see. Correcting the bad tick now causes all the remaining prices of the
block to be bad, contrary to what we actually observe. Therefore, let's
change the "understandings" slightly.
Let "M" represent the value of P[0] retained in a memory location, as
opposed to the database, and assume that the data corruption occurs
somewhere BETWEEN saving the (correct) value in M and storing it in
DB[0].
Here's what we would see in real time:
j_ P______ DB_____ Ch_____ ET_____
59 8210.25 xxxx.xx 8210.25 8210.25
00 8210.60 8810.60 8210.60 8810.60 (a bad tick store at block start)
01 8210.85 0000.25 8210.85 8810.85
...
59 8225.60 0015.00 8225.60 8825.60
00 8224.10 8224.10 8224.10 8224.10
01 8224.90 0000.90 8224.90 8224.90
Note that all values chart correctly but are bad in ET.
For a reloaded (not real-time) chart we would see, before correction:
j_ DB_____ Ch_____ ET_____
59 xxxx.xx 8210.25 8210.25
00 8810.60 8810.60 8810.60 (a bad tick at block start)
01 0000.25 8810.85 8810.85
...
59 0015.00 8825.60 8825.60
00 8224.10 8224.10 8224.10
01 0000.90 8224.90 8224.90
That is, we would see a jump to the incorrect value, 8810.60, the values
would then REMAIN erroneously high till the end of the block, then they
would drop to the correct level - this is precisely what we do see.
Finally, let's correct the bad DB value and see what happens:
j_ DB_____ Ch_____ ET_____
59 xxxx.xx 8210.25 8210.25
00 8210.60 8210.60 8210.60 (bad tick corrected)
01 0000.25 8210.85 8210.85
...
59 0015.00 8225.60 8225.60
00 8224.10 8224.10 8224.10
01 0000.90 8224.90 8224.90
ALL the prices are now correct, exactly the behavior we observe.
Conclusion: In order for a correction of the initial "bad tick" to
result in correction of all the remaining prices in the same block, the
delta values stored in the database M U S T have been calculated and
stored correctly. But for the deltas to have been calculated correctly,
the correct value for the initial tick M U S T have been available.
Regards,
Carroll SlemakerReturn-Path: <Melody.Blais@xxxxxxxxxxxxxxxxx>
Received: from h1.mail.home.com ([24.0.0.50]) by ha1.rdc2.occa.home.com
(Post.Office MTA v3.5 release 217 ID# 1-1U40000L0S0V35)
with ESMTP id com for <cslemaker1@xxxxxxxxxxxxxxxxxxxxxxxx>;
Tue, 15 Sep 1998 15:26:58 -0700
Received: from mx3.home.com (mx3.home.com [24.0.0.32])
by h1.mail.home.com (8.9.0/8.9.0) with ESMTP id PAA10961
for <cslemaker1@xxxxxxxx>; Tue, 15 Sep 1998 15:26:54 -0700 (PDT)
Received: from omegaxchg.omegaresearch.com (omegaxchg.omegaresearch.com [209.149.196.11])
by mx3.home.com (8.8.5/8.8.5-AtHome) with ESMTP id PAA20893
for <cslemaker1@xxxxxxxx>; Tue, 15 Sep 1998 15:26:50 -0700 (PDT)
Received: by omegaxchg.omegaresearch.com with Internet Mail Service (5.5.2232.9)
id <S7WTY00L>; Tue, 15 Sep 1998 18:26:04 -0400
Message-ID: <5131CB844FF0D111840600805F85967949520D@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
From: Melody Blais <Melody.Blais@xxxxxxxxxxxxxxxxx>
To: "'cslemaker1@xxxxxxxx'" <cslemaker1@xxxxxxxx>
Subject: Bad Data for INDU
Date: Tue, 15 Sep 1998 18:26:04 -0400
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2232.9)
Content-Type: text/plain;
charset="iso-8859-1"
Dear Mr. Slemaker,
After speaking with the engineers who wrote the server and
the charting application, I understand the reason for the price difference
between an active/open chart and the Edit Tick dialog window. When the data
is received from the feed, it is stored in a buffer prior to being written
to the hard drive and simultaneously sent to the charting application.
The prices are sent to the chart without any type of logic
checking. However, when the data that is stored in the buffer is written to
the hard drive, the current price is checked against the first tick of the
day. If the difference between the first tick of the day and the current
tick is greater than 32,767 the data is stored incorrectly. If the price
scale is 1/100 as it is with the INDU, the maximum difference is then 327
since the decimal is moved two places to the left. The problem occurs
because the check should be against the first tick of the block and not
against the first tick of the day. Therefore, in the case of the INDU, if a
price is plus or minus 327 points from the first tick, a bad block of ticks
will be stored on the hard drive. These incorrect values will only be seen,
however, when a chart is created or changed after the data has been stored.
We will try to catch this and correct the data before we
upload the refresh files to the FTP site.
This will not occur in TradeStation 5.0 as the data is
handled and stored using a completely different protocol.
I hope this provides the information you need and addresses
your concerns about this problem being carried over into TradeStation 5.0.
If you have any further questions or comments, please do not hesitate to
contact me.
Sincerely,
Melody A. Blais
Omega Research, Inc.
Customer Support Supervisor
|