|
PureBytes Links
Trading Reference Links
|
At 01:12 PM 3/16/98 -0800, you wrote:
>From: Dennis Holverstott <dennis@xxxxxxxxxx>
>>>
>Another good area for discussion would be some demonstrations of how
>fraudulent vendors can use the built-in trailing stops and profit
>targets to make a losing system look like a winner in backtesting. <<
>
>An excellent suggestion.
>
>We could all learn about the limitations of built-in trailing stops and profit
>targets. Clear examples of how it misleads would be most instructive.
>- Mark Jurik
___________________________________________________________________
March 30, 1998
Experienced users of TradeStation know it has a serious flaw that has been
known by some traders for a long time, in the way the built-in stop features
are applied when trailing stops are run too tightly, or when the built in
"%trailing stop" is set too tight in a historic backtest. Backtest results
end up being dramatically overstated relative to real time results. If those
backtest results are relied upon, people will lose money, and that's the
bottom line.
Given that one of the main product purposes and uses of TradeStation by
traders is to backtest methods and systems BEFORE trading them, in order to
improve the profitability of trading decisions, this flaw is VERY SERIOUS.
It has also gone on for a long time, unaddressed. There's not even a
warning in the users manual about it. I mentioned this in a recent post
hoping it would be fixed in TS 5.0. Dennis H. also mentioned it above and
gave an actual example in his post. I think it was Earl Adamy who also
spotted it in a response to a system that someone posted on the list for
comment. Mark also asked about it recently. So it's hardly a new topic.
I mention it again today for TWO IMPORTANT REASONS:
++++++++++++++++++++++++++++++++++++++++++++++++++
1. I was actually defrauded by this years ago when I was a rookie, by a
vendor who used the bogus trailing stop technique to fraudulently lure me
into a system buy with grossly overstated TradeStation backtest results from
this flaw. We had occasion to shut down and restart TS while trading the
system in real time and voila! suddenly witnessed all the recent trade
numbers change before our very eyes!
A second time, another quite famous vendor (I won't mention his name today)
tried to sucker me with the same technique, but luckily my experience in the
first sting prevented it.
No doubt many new traders have at various times been burned badly by this,
either in a system purchase or by trading a system that backtested well, but
performed miserably in real time, as a direct result of this undocumented,
unreported flaw in TradeStation.
2. You can imagine then how really shocked I was today, when I received a
direct mail piece from Omega Research promoting a "FREE VIDEO [that]reveals
how you can dramatically IMPROVE PROFITS by testing and automating trading
ideas with TradeStation". The flipside headline says "This video could
change your life...". Some of you might have seen this piece very recently.
The shock is on the inside of the brochure dead center where there is a TS
published performance report for "My Favorite System", for the S&P 500.
It's a daily bar system that shows $66,800 in net profits, a win ratio of
57%, profit factor of 3.02 and remarkably only $2,925 in max intraday
drawdown [isn't that great ?] PLUS, a staggering return on account of 2284%.
Enough to get a prospect excited, yes, no? .... SORRY!
The problem is, BELIEVE IT OR NOT, we seem to have the infamous trailing
stop bug at work here, dramatically OVERSTATING BACKTEST PERFORMANCE RESULTS.
There are actually two reasons the results IN THE OMEGA BROCHURE EXAMPLE are
overstated.
The first is that no margin is added into the performance report setup.
Virtually every system vendor I've ever seen does this automatically, that
part is no big deal really. Nevertheless, the result is the "return on
account" [ the ultimate profit number that we all look at] is actually
overstated by a factor of about SIX TIMES!
So the 2284% return is completely out of line when the margin is excluded!
Secondly, the trailing stop bug is no doubt at work here to likely further
overstate the performance results of that 'my favorite system'. How can one
tell that, you ask?
First, "0" average bars in winners and "0" average bars in losers as a dead
give away, although in some cases it isn't always that obvious. How could
they not notice it I asked? Second the average trade size is quite small for
a daily bar system because many trades are likely being closed out during
the entry bar. The result is that all aspects of the performance of "My
favorite System" are erroneous and probably very much overstated as well.
Luckily it's only an Omega product sales brochure. Or unluckily, depending
on how you look at it.
Now an inexperienced trader could easily do a backtest like this (like we
saw in an actual case here recently) get all excited, go out and place the
realtime trades and several in a row would produce dramatically different
results than those set out in the TS generated backtest results. The likely
result would be trading losses, ... maybe he'd lose his shirt for a while
showing all the so-called discipline that a mechanical trader is supposed to
show to make money, right? Wrong! If you think I'm kidding here, or it's
not that serious, think again.
I suspect [and hope] that Omega's own marketing and sales department has
only made an unintentional public error here in one of their own promotional
product brochures.
The question then is, do you think it's possible or likely that customers
and other real traders have made similar more fatal mistakes in testing and
trading with TradeStation as a result of Overstatement of System Results in
backtest?
Remember the main selling proposition of the TradeStation product is: "Test
before you trade". That's supposed to be the main benefit of the product.
New traders need to be aware of the real PITFALLS of system testing in TS.
This is an example of just one. There are others to also watch for that we
can review in future.
PS. I sent a copy of this message to Omega Research. It seems pretty
unlikely to me that Ralph Cruz or the marketing manager would be aware of
such a blatant example of overstating system backtest results in
TradeStation............. right in their very own product brochure, and then
still run with it.
Michael Paauwe
mpaauwe@xxxxxxxxxx
|