|
PureBytes Links
Trading Reference Links
|
This post takes a bit to read, but it may save you significant $$$ some day.
Here is what follows:
1. Preface.
2. Introduction to the Dirty Secret of Trading Infrastructure Failures
3. Types of Infrastructure Failures
4. Causes of Infrastructure Failures
5. Some Ways to Protect Yourself.
- - -
1. Preface. You won’t find this topic discussed much in any books or
articles or seminars.
Why?
As a group, brokers and others who earn their keep by selling securities or
investing and trading services, only attract and retain customer by being
positive – little to no money is made by scaring folks off, although some
modicum of honesty in fine print is required for legal protections.
An individual broker or guru or analyst never knows when s/he might someday
have to work with a colleague or for a company that s/he criticized
publicly.
Candidates for criticism might also someday be customers or partners.
Bottom line, you won’t find much reliable or accurate negative information
about brokerages, trading tools, ECNs, data feed providers - you will only
find positive statements and sometimes neutral or very mild criticisms if
even any of the latter.
Other than chat groups and messages boards, of course. Even there, its
sometimes hard to distinguish between those genuinely trying to be helpful,
and those with less honorable motivations.
2. Introduction to the Dirty Secrets about Trading Infrastructure Failures.
The most hidden major problems of day trading, and ones that brokerages in
particular are very reluctant to publicize, are the frequent failures of
real time trading infrastructures.
Few trading gurus sufficiently warn the rest of us about the frequency,
costliness and seriousness of these pervasive real time trading
infrastructure failures.
The SEC and the exchanges provide no protections for traders and investors
against such, they only provide legal protections for brokers and ECNs.
Even grossly bad IT management and senior management, and gross negligence,
of brokers, ECNs and other real time data providers, are protected. They
have no legal and no regulated obligations to provide anything resembling a
reliable trading infrastructure.
Brokerages blame traders’ losses caused by the brokerages’ own failings, on
the traders’ themselves, saying the traders should protect themselves - - -
never mind that this is often impossible because of the nature of many
infrastructure failures.
Brokerages never, ever publicly accept responsibility for their
underinvestment, incompetence, negligence and mistakes, bad general
management, bad IT management and/or bad IT professionals they may employ,
regarding infrastructure failures. There are no profitable reasons for them
to do so.
Same for data feed providers, and trading and investing software providers.
Many gurus parrot the self-serving interests and excuses of the brokerages.
After all, the brokerages are a major source of referrals to the courses,
seminars, videos and books of these gurus.
If you are an intraday trader, odds are high that once a month or more you
will be impacted by a trading infrastructure failure that is preventable by
good brokerage management.
Over the past four years, I have lost far more trading money because of
these infrastructure failures, than from all other causes combined,
including (but not limited to) bad trades, bad fills, events, not following
my own trading rules, etc.
During my short, 2.5 months of using TS6, it has been remarkably free of
most of these failures, at least ones under the reasonable control of TSS
and TS. But this is hardly typical of many brokerages, software tools and
data feeds. And it is not a sufficiently long enough time sample yet. And
TSS/TS6 is not without infrastructure problems, such as its use of the
replaceable, sometimes unreliable and slow S&P Comstock data feeds.
3. Types of Infrastructure Failures.
Here is a partial list of the kinds of failures that all intraday traders
experience, sooner or later:
- Trading software bugs. So far, TS6 has been kind to me. Hope that TS7
will be at least as well tested.
However, during 2000 and 2001, the market share leading day trading software
that I used before TS6, failed at least once weekly, especially on Mondays,
due to poor testing and hastily applied fixes and patches on weekends. These
bugs cost me many $ thousands in unrealized profits, and some real $ losses.
One trader customer of this firm wrote a published article saying that such
failures cost him over $100,000 annually but that he (very unwisely IMHO)
considered that just another one of his trading costs.
- Broker servers. When I traded at a local broker office for the above
mentioned, accursed competitor, the local broker manager stupidly took
shortcuts on her choices for local Internet Access providers, causing
frequent local office outages and slowness. One failure for a couple hours
on one morning alone, cost me $14,000 in unrealized profits on two trades.
Compounding that, the central servers, networks and customized load
balancing software of the parent broker, were undersized, poorly managed,
and incapable of handling high volume days. So its networks and servers
failed frequently, some for only short time periods, but sometimes for
longer periods. Another problem is that while logged on, its defective
custom server load balancing software would frequently disable a trader’s
connections to its price quote servers, for no externally discernable
reason. Have no idea how much this cost me in unrealized profits and real
losses.
So, guess what? During the very high volume event driven times when traders
needed access the most, the local broker’s and/or central broker’s servers,
etc. would fail.
The particular broker that I used in 2000 and early 2001 combined its
infrastructure problems with a very high degree of central customer support
arrogance, for which it appears to be infamous.
Plus the local incompetent branch office manager turned out to be a con
artist - she lost a lawsuit in court to me, but continues to this day to not
pay the court ordered settlement. Meanwhile, that national brokerage’s
local office has been closed for more than a year – no solace for me,
because does not help me recover any losses caused by the local con artist
nor by the national brokerage.
- ECN failures. Especially ARCA in the past, BRUT more recently. One ARCA
failure in combination with the central broker server failures cost me over
$25,000 in real losses in half a day, in early January 2001. Was in two
profitable shorts with covering stops, when the Fed Reserve announced its
first surprise rate reduction. My stops were skipped, not something
unexpected. Normally that means entering market orders to get out ASAP.
However, my capital was locked up by ARCA and the brokerage, because the
stop orders had been issued but not executed, so I could do nothing until
ARCA came back up, hours later. The brokerage was swamped by phone calls
from me and the many other traders similarly trapped. BTW, neither ARCA nor
the brokerage even apologized, much less offered any compensation.
Where are the class action law firms when you really have a legitimate class
action lawsuit? (Answer: brokerages are their best clients, not traders.)
- Exchange server or network – data feed - failures. Seems to happen at
least twice yearly for the NASDAQ, maybe once yearly for NYSE, not sure
about the other exchanges.
The causes are usually the problems of low bidder network providers, such as
WorldCom, rather than simply spikes in trading volume.
So far, these have not cost me anything, but that has been purely luck.
- Local network provider failures.
Cable modem providers in particular do not seem to understand what the word
“reliability” means for data. AT&T in particular seems to be very slow to
diagnose and repair problems. As a customer, it appears that it does not
have, or does not have sufficient or fast enough, automatic detection of
network failures.
Since I began trading from home a couple years ago, none of the local
network provider’s failures have cost me any actual money – because of what
I’ve learned to do as written later herein – but they have cost me an
unknown amount of lost trading opportunities.
- More insidious, are the slow data feeds that are often hard to detect. We
haven’t had a peak volume period while I’ve been using TS6, so it will be
interesting to me to see how TSS and TS6/7 do during the next one.
4. Causes of Infrastructure Failures. The underlying and largely
preventable causes of the infrastructure failures include:
- Poor IT management and senior management at brokerages. Plus poor IT
professionals hired and kept by latter.
- Poor local brokerage office management compounded by poor controls and
oversight of local broker office managers. Some of these are
pseudo-franchises with resultant frequent and unwise short cuts on local
network and server operating expenditures.
- Poor IT management and senior management at many software firms.
- Apparently poor IT management and senior management at some ECNs, e.g.,
ARCA and BRUT. But even Island and Instinet have their problems.
- Poor testing and change management at software firms, ECNs and brokerages.
- Poor capacity management at brokerages and ECNs. The exchanges seem to do
an excellent job, by comparison. In addition to volume induced outages, more
frequent are the data feed slowdowns and slow order executions.
- Underinvestment in staffing and infrastructure at brokerages especially,
and at many trading software firms.
- Widespread use of WorldCom, which as a former IT exec and customer of
same, I know first hand to have been a very unreliable as a data network
services provider.
- Poor to nonexistent use of fail soft and fail safe processes by brokerages
and their network providers.
- Poor to nonexistent hot site backup facilities by brokerages and their
network providers.
- Poor to nonexistent uptime guarantees, goals – the exchanges, plus many
large banks and retailers, for example, achieve very high uptimes and long
times between failures – and so can well managed brokers, ECN’s, data feed
providers and software firms.
- Nonexistent performance commitments from any organization in the data feed
food chain. Competently managed IT organizations have these with their
internal users. Telephony and data network providers supply these when
asked by their customers, and if negotiated well, can include substantial
rebates in the event of service disruptions or slowdowns.
So, why are brokerages and ECNs and even the exchanges managed differently
with respect to their customers?
Answer: lack of market pressure; lack of regulatory pressure or laws; lack
of competent IT and senior management at many of these firms.
As an IT executive, I was able to lead IT functions to provide extremely
high uptimes with excellent response times, without excessively high costs.
So do many other IT executives and IT functions, in banking, hospitals,
retailing, ISP’s and the like.
So could brokerages and data feed providers to traders.
I know of one software/data feed provider that had zero failures for 24+
consecutive months for the components within its own control, for example.
Its data center has three independent data network feeds through three
different Telco central offices, a large capacity UPS system, plenty of
extra server and bandwidth capacity, backup servers, good change management
practices, etc. Unfortunately the company has other problems preventing it
from being a lot more successful, chiefly marketing and sales.
- Broadband providers still do not get it, regarding reliability and high
speed. Certainly not the cable modem providers that I know anything about.
- Lastly and above all, absolutely no SEC or exchange or other regulatory
organizations care enough about traders, to require at least a minimal
degree of responsibility by any of the members of this real time trading
chain.
5. Some Ways to Protect Yourself.
OK. Here are the important things. Only you can lessen large losses caused
by the inevitable infrastructure failures.
- Stops. If you do not use stops with each of your trades, when detect some
danger, then issue stops or closing trades ASAP.
- Margin. Always, always, always have enough overnight margin left over to
issue hedging trades sufficient to cover open trades going against you, when
there are major failures during fast moving markets.
- Verbal Fall Back. Always have all emergency telephone numbers for your
brokers, software providers, network providers and if appropriate, data feed
providers, on speed dial and clearly identified. Always have both a cell
phone and a land lines based phone ready and at hand. Post these numbers
with clear identification at your trading workspace. Also post coded
note(s) that only you can decipher quickly for your trading account numbers
and associated passwords.
Use the phones to report the trouble(s). Also use them to issue emergency
stops or hedging trades with your broker(s).
- Power. Be sure that you have a large, reliable Uninterruptible Power
Supply (UPS) capable of keeping your PC running for least 20 minutes when
your electrical power fails. Batteries in these systems do wear out, and
must be replaced every three to five years, by the way.
- Backup all files fully each weekly onto reliable backup media. Keep at
least 5 prior backups.
- Records. Keep really good records for each failure. You cannot know when
a failure occurs, just how long it will last, nor how serious it will be in
impacting your trading account. You may need this data initially for
discussions to get credits from the offending provider, or eventually in
arbitration hearings, government agency hearings, lawsuits, etc. A large
failure that is preceded by a bunch of small failures might be used to prove
gross negligence, for example.
- PCs. Have one set up ready to go if the other one fails. One trick is
that you might have to make sure that the detectable PC name is identical on
both boxes.
- Connections. If your location permits it, have two broadband connections,
each from a totally different network provider sharing no cabling, no
servers, nothing in common. If you cannot afford, or choose not to afford,
or cannot technically get a second broadband connection, then at least have
a dial up connection for your trading data feeds. And a cell phone for
emergency voice calls to your broker.
Home broadband connection choices are cable modems; some form of DSL;
satellite. You can also get a dedicated T-1 line or better at very high
cost, or perhaps a shared T-1 in some locations.
Stationary wireless broadband is not yet widely available in the USA. It
also might not yet be technically sound enough to be the primary broadband
connection, for reasons of reliability, true data throughput and security.
ISDN is a semi-broadband choice. It is not nearly as fast as DSL or cable
modems, but ISDN can be a lot faster than dial up. Where available, it is
can be more reliable than DSL or cable modems.
There are other choices, such as Virtual Private Networks, but they tend to
use one of the above physical paths, such as a shared T1 line.
At a minimum, have a dial up back up line. This is best done with two
lines….one for data back up, the second so that you can still telephone your
brokers, software providers, etc.
- Broker, Data Feed, Software. Choose a brokerage, software tool suite and
data feed provider, weighting reliability and speed very high, and trading
commissions next.
Over the years its been kind of amazing to see so many otherwise intelligent
traders, choose brokers and software tools solely on the basis of chance
meetings, rather than by the careful analysis with which they trade. Many
traders spend less time evaluating brokerages, data feeds and software
tools, than when buying a car or PC.
- Change. Change brokerages and software tools if/when the infrastructure
failures are excessive for your needs.
We traders tend to hate changing, after we get used to and understand how to
use a particular software tool. Learning a new tool is time consuming and
can be frustrating, especially the first time a trader changes. But that is
less painful than large losses caused by inexcusable infrastructure
failures, is it not?
Another factor in making changing so difficult, is that the best individual
brokers seem to be as likeable, friendly, warm and caring about us as our
best friends. They are good listeners and always agree with our latest
brainstorms. Good at commiserating. Sympathetic. Congratulatory. Always
tell us how good our trading methods and trades are, no matter how
hair-brained such may actually be. Always have positive attitudes. Always
have some tips. Etc.
But please believe me, your broker is in this game for your commissions
primarily, no matter how much s/he may otherwise value your relationship.
And there are vanishingly few brokers who know beans about technical
analysis and trading.
- Software and Data Feeds. If you can afford it, and if your PC is powerful
enough, use two different software providers having two different data
feeds. That will provide you with trading info if your primary tools or
connections are down. This will also provide you wit speed comparisons, and
perhaps give you a little extra edge when one is faster than the other at
any point in time. This is ideal if you have each of these on different,
separate broadband providers and PCs.
Another alternative if you use trading software that can work with more than
one data feed provider, is to have two different data feeds. This is kind
of a rare situation, and as far as I know, would not work with TS6.
- Complain. Yes, walk with your feet to other providers. But complain,
politely, loudly and often. To the offending providers – beyond just their
customer “service” functions, to their corporate officers and to their board
members. To the exchanges. To the regulatory agencies – Federal and state.
To the media – printed, TV and radio. Etc. When you are sure that you have
been wronged, do not go gently into the night.
Certainly my knowledge, experiences, memory and writing on this topic are
far from complete.
Please help me and yourselves more by adding to the above!
Vince Heiker
Flower Mound, Texas
|