>Engineers agree with Mark Twain: there are lies, damn lies, and
then there is statistics
Err…sorry,
but that is a slightly misplaced quote. Yes, Mark Twain said it and yes it can
be true when statistics are deliberately used to mislead people as in the way
politicians regularly use them. But engineers…? I know you are one, but I
would like to venture a bit of personal experience that may persuade you and
others interested in how statistics are regularly applied in a Real World
application that you may not have considered before now. Perhaps that may then
help you shed a more favourable light on the way stats are applied in the world
of trading.
In my previous
life I was a geologist working for a large mining company. My claim to fame in
that area is that I was on the (small) team that was responsible for one of the
more significant gold discoveries of recent years (for those interested, the
Geita deposit in Tanzania…now
belonging to Anglogold-Ashanti). So, the point here is that I have some
experience in using statistical modelling…namely in the practical
application of “geostatistics”.
Geostats is
nothing fancy…it’s just the name given by geologists to the practice
of determining the size and internal grade distribution of an ore body using
statistical methods. Basically one has a limited data set of samples from drill
holes and uses that to come up with a 3-D model of the ore body and the
distribution of metal within that body…and then to put levels of statistical
confidence in that model. Those levels of confidence determine (usually within
industrial standards) how much more, if any, drilling is required to satisfy
the *mining engineers* that what
you have is not just a mineral deposit but an ore body (the former is just
metal in the ground, whereas the latter can actually be mined economically).
Mining
engineers know that this is only a model. They know that it is not 100% fact (as
the only way to determine that is to actually mine the thing). They know the
statistical confidence levels and therefore they know that there are likely to
be some errors in the model. This means that when the miners actually come to
take the gold out of the ground there will be less in some places and (more
pleasantly) more in others. And yet, look at what happens…capital flows,
mines get built, people get employed, dirt gets shifted, metal comes out of the
ground. And all that hangs on a few statistical inferences made by a bunch of Neanderthal
geologists. Not bad, eh?
So there is no
question of “lying”. To do so would involve not only gross professional
negligence on behalf of the geologist, but it would mean that everything that
else that normally follows would fall apart…usually long after all the
capital has been spent and the people have been employed, i.e. when it’s
far too late. Also, in the case of deliberate scams like Bre-X, the use of geostatistical
“lies” can affect the whole industry.
Now, with all
that said, you might be surprised to find that the principle of stationarity is
relaxed almost to the point of irrelevance in geostatistics. Nature is a
wonderful thing, but it rarely conforms to simple mathematical models and so,
skipping over the jargon, we basically find that we have to make some pretty
sizeable assumptions and generalisations when coming up with the models. Are we
lying when we “bend the rules” so? I don’t think so…we
are not agreeing with Mark Twain at all. He was implying, I think, that
statistics are determined and then manipulated for an ulterior motive. That is
different from honestly recognising, discussing and then trying to work around
the limitations of the practice…
All this has
direct implications to the world of trading. I’ve already gone on too
much so I’ll only say now that you can draw two direct parallels between
geostats and “trading stats” (Tradstats??!!). One is that there is
a known quantity…the drill hole data is dirt already taken out of the
ground and analysed - this compares with the historical data set in trading.
The second is that there is an unknown quantity that you want to estimate, or
model…the very sizeable un-mined bits of the deposit between the drill
holes(!) and the future data in trading. The only problem I can think of there is
to do with continuity. The drill hole data is not spatially continuous in 3D,
whereas the time-series trading data is. Oh well, that’s not relevant…what
I’m trying to say is that you should not confuse statistical modelling
with any sort of “holy grail”. Perfection does not exist when
dealing with models, as any experienced mining engineer will tell you…but
that doesn’t mean in any way that a good model will not help in getting
the job done. And engineering is, after all, about getting the job done.
Hope that’s
helped open your mind a bit?!
From:
Metastockusers@xxxxxxxxxxxxxxx [mailto:Metastockusers@xxxxxxxxxxxxxxx] On Behalf Of jawjahtek
Sent: Friday, July 01, 2005 11:07
PM
To: Metastockusers@xxxxxxxxxxxxxxx
Subject: [Metastockusers] Re: New
Adaptive Tools for Metastock
Random question and comment:
1. Superfragalist, have you tried CSI data
(assuming you use EOD data)?
There is no such thing as a good data provider,
but at least CSI is
honest and up front about continually cleaning
their data AND telling
users what errors were made. If you use intraday
data, I can see how
you have been hosed. The only option that I have
seen is to match the
professional set ups: use multiple intraday
suppliers.
2. While I wish the developers of the new adaptive
tools all of the
luck in the world, I don't believe that ANY
application of
Communications (Signal) theory can be successfully
applied to price
data. In academic terms, these theories require
Stationarity. In
layman's terms, this means that the theories
require a constant range
of frequencies (cycle) and phases (time lag).
Unfortunately, historical
price data tells us nothing about the future
prices' frequency and
phase.
Statistician's definition of Stationarity: a
statistical name for
expressing degrees of invariance in the properties
of random functions;
it refers to the statistical model, and not to the
data. Most commonly
used to indicate invariance in the mean and
variance, but also in the
variance of first differences.
I am an Electrical Engineer. Although EEs use the
concept of
Stationarity, its meaning is slighly different in
engineering.
Engineers agree with Mark Twain: there are lies,
damn lies, and then
there is statistics.
I will try the free trial, but I already know that
the holy grail does
not (and cannot) exist.
jawjahtek
--- In Metastockusers@xxxxxxxxxxxxxxx,
"superfragalist" <jackolso@xxxx>
wrote:
> Well, I've got all those IVs hung off my
wallet also. Reuters at least
> cleans their data, which esignal doesn't do.
In fact, esignal can't
> even adjust for splits.
>
> I didn't know netflix had DVDs on trading.
I've just been trading DVDs
> with them.
>
YAHOO! GROUPS LINKS