|
PureBytes Links
Trading Reference Links
|
Comments below...
| -----Original Message-----
| From: HBernst963@xxxxxxx [mailto:HBernst963@xxxxxxx]
| Sent: Monday, October 12, 1998 6:09 AM
| To: editorial@xxxxxxxxxxxxx; omega-list@xxxxxxxxxx
| Subject: Re: WHY BACKTESTING WORKS
|
|
| Yes, I've read the interview. The question is- is he
| following the same
| strategy that he outlined in his book, or has the human in
| him caused him to
| make changes to the backtested strategy? Since the fund I
| mentioned is 31%
| behind the S&P this year, which is worst year ever for this
| strategy (worst
| prior performance was 18% behind the S&P in 1975-this info
| comes directly from
| his book What works on Wall Street which I have in front of
| me) you have to
| question the strategy.
|
| Futures traders know that when your realtime loss exceeds
| your largest
| historical loss, as in this case, it's time to reexamine
| your system. By the
| way and not to knock O'Shaughnessy, I enjoyed the book and
| his well thought
| out backtested approach. It's just that backtesting and
| assumptions that are
| made by the backtester can amount to curve fitting. I know
| from my own
| backtesting of trading systems how easy it is to be lulled
| into thinking that
| you have just found the holy grail, when the system falls
| apart in realtime.
| Larry Williams tells a story of a great bond system he
| discovered, backtested
| it using over 1000 out of sample trades where it performed
| very well. It
| couldn't make a dime in realtime!
Could some of the backtesting enthusiasts please comment on this Larry
Williams anecdote? Please do not comment on your opinion of Larry Williams,
but on the essence of the story, which to me is:
1. You backtest a system on part of your historical data. It looks good
(whatever your criteria for good is)
2. You test the system on out of sample data. It still looks good.
3. You start trading it. It performs terribly.
Testing out of sample data is the only way I know of to "real time" test a
lot of data without having to wait for real time to go by. There should be
no statistical difference in out of sample results and true real time
results unless something fundamentally changed between the out of sample
data time period and the real time period. This seems to strike at the
fundamental validity of backtesting.
Some possible questions to consider and to generate some discussion on the
list (in no particular order):
Has this scenario happened to others?
What could cause this?
Is there a way to anticipate this?
Is there a step missing before deciding to trade a system in real time?
Does this disprove the validity of backtesting?
Testing on out of sample data is a method to make sure that your system is
not curve fitted to the tested data. Is there something else you should do
to test and eliminate curve fitting?
Howard's experience confirms this story. What about other's experiences?
If you believe in backtesting, how do you account for these experiences?
Thanks for any insights.
Neil
|
| All I'm trying to do in inject a note of caution in the
| reading of backtested
| results from someone who has been burned by it.
|
| Howard Bernstein
|
|