|
PureBytes Links
Trading Reference Links
|
"Gary Fritz" wrote:
>I don't claim to be an expert, but I've never seen a system whose
>performance doesn't drift SOME over time. If a system performs well
>over a long time, I'd consider it fairly robust even if it wasn't
>performing at optimal levels the whole time. I'd be amazed and
>impressed by a system that retained OPTIMAL performance over many
>years with the exact same parameter set. Does such a beast exist?
Gary:
I agree with Mark. I think the semantics here are what you mean
by "optimal". Sure, for any backtest period, a unique set of
system parameters are going to give the highest net profit. So,
yes, at the end of each year (or whatever period), a different
set of parameters could be shown to be "best". Are those parameters
"optimal"? Well, yea, if that's how you define optimal.
However, I define optimal as: the parameters most likely to give
good results in the FUTURE. And, of course, you can't determine
this from simple backtesting. So, I would make two comments:
1) A system whose parameters vary slightly between different
backtest periods is more likely to run successfully in the future.
Which of the "optimal" parameters you apply going forward is a
bit of a crapshoot. Ie, maybe 1997's optimal values will work best
in '00 - not 1999's. The discipline of "walk-forward" testing
provides a methodology for dealing with this, but you are still
betting that the future will behave something like the past.
2) Most commercial systems are designed to be sold, not to be
traded. So, it is in the vendors' interest to make silly claims
like "made $120,000 per contract last year, with 86.4 % successful
trades". Well, sure it did - they just optimized the parameters
over that isolated period to produce those results. Note that
their statement is true in the narrow sense. The dishonesty is
that they fail to tell you that the same parameters would have
crashed and burned in other years, and most likely, in the coming
years. The further dishonesty is to continue running these ads,
basing the reported results on re-optimized parameters, and calling
the product the "N+1 th release". I think that this is what Mark
was objecting to.
Jay Mackro
|