|
PureBytes Links
Trading Reference Links
|
<FONT
color=#0000ff>if optimization makes sense at all, why does it make
sense to optimize on one particular day, but not another? which day would you
choose? if it's useful the first
day, why then not <SPAN
class=788374313-18102003>on the second?
<FONT face="Courier New" color=#0000ff
size=2>
<FONT
face="Courier New" color=#0000ff size=2>note that if you always optimize on all
data before the current day, typically after a while things stabilize and don't
change much. any new behavior has to persist for quite a while for it to
overcome the weight of all that prior history.
<FONT
face="Courier New" color=#0000ff size=2>
<FONT
face="Courier New" color=#0000ff size=2>re forward testing, auto-optimization
(I'll abbreviate AO) makes the entire test a forward test, in the sense
that every day, you're trading out of sample data. you never optimize on future
data, so your optimizations reflect only prior history.
<FONT
face="Courier New" color=#0000ff size=2>
<FONT
face="Courier New" color=#0000ff size=2>
<FONT
face="Courier New" color=#0000ff size=2>if you think about the whole process of
optimization and testing, it goes like this: optimize up to some date, trade
those settings for some period of time after that, optimize up to another later
date, use those settings for some period of time, etc. how do you pick the times
to optimize vs times to test? if it's a valid approach, can't you pick any
optimization period and have the same odds of success?
<FONT
face="Courier New" color=#0000ff size=2>
<FONT
face="Courier New" color=#0000ff size=2>now suppose you wanted to know whether
the entire approach of optimization worked at all. so you test that, by choosing
a series of optimization periods and examining performance with those settings
after that. AO just automates that whole process, while remaining completely
agnostic about which optimization period you choose, by choosing them all, one
day at a time. in a pretty real sense, AO is a backtest of the idea of
optimization. if AO fails to produce good results, why isn't that an
indictment of optimization itself?
<FONT
face="Courier New" color=#0000ff size=2>
<FONT
face="Courier New" color=#0000ff size=2>
<FONT
face="Courier New" color=#0000ff size=2>to me, it keeps boiling down to the same
thing: if past performance predicts future performance, AO should work. if AO
doesn't work, why doesn't that say that past performance doesn't predict future
behavior?
<FONT
face="Courier New" color=#0000ff size=2>
<FONT
face="Courier New" color=#0000ff size=2>this is the reason I glommed
optimization and backtesting together in some senses. what's the use of either
backtesting or optimization if the past doesn't tell us much about the
future?
<FONT
face="Courier New" color=#0000ff size=2>
<FONT
face="Courier New" color=#0000ff size=2>I'm stuck in a mental loop. I keep
saying the same things over and over, as I'm sure you've noticed. either I
believe the conclusions this line of thinking led me to, or I don't. if I do,
logic says I abandon TA completely and look only to fundamentals, industry
understanding, and insider behavior to make investment decisions. if I think
there's a hole in the logic instead, I have to find it, or I can't distinguish
meaningful tests from whatever mistake led me to the conclusion I don't
accept.
<FONT
face="Courier New" color=#0000ff size=2>
<FONT
face="Courier New" color=#0000ff size=2>
<FONT
face="Courier New" color=#0000ff size=2>re your last question about choosing
between optimization results, the case where you optimize over all 3 years,
including the test year, is invalid, or at least it's not a forward test. as we
keep coming back to, of course you do better when you optimize over the period
you're going to test, because your settings accommodate what happened during
that time. only tests on data that wasn't used to generate the settings you're
using are valid, IMO.
<FONT
face="Courier New" color=#0000ff size=2>
<FONT
face="Courier New" color=#0000ff size=2>dave
<BLOCKQUOTE
>Aren't
you mixing the two -- backtest and optimization? The backtest is of
course, absolutely necessary. Probably same goes for a little bit of
optimization. But isn't auto-optimization every day the other end of the
spectrum? While backtest in itself is a form of optimization (with just
one set of parameters), auto-optimization every day comes pretty close to
serious curve-fitting, IMHO.In one of your other posts, I think you
said that you got the best results when you optimized over entire dataset.
Assuming you meant that when you used last X number of days/months as a
forward-test OOS period, I wasn't too surprised to hear that. Why wouldn't
you use all the past data to find optimum parameters if you're going to
try to auto-optimize? Of course, you could then try to find the optimum
period to optimize over, but that too again might be
curve-fitting.I have a question for proponents of back and forward
testing. Assume that we're dealing with 3 years of data, where we use
first two years to optimize and the third year as forward testing period.
Further assume that you get following results -1. When optimized
over first two years, both of those years show profit, but the third year
shows loss.2. When optimized over all three years, all three years show
profit, although less than that showed by first test in first two
years.Which set of parameters would you use going forward in the
fourth year? Or would you rather just dump the strategy? I personally
think that back and forward testing makes sense ONLY if you do the latter,
but I'm not sure that's what happens. But then... I could be
wrong.Jitu--- In amibroker@xxxxxxxxxxxxxxx, "Dave Merrill"
<dmerrill@xxxx> wrote:> I'm willing to do the work I think,
have been trying to already, to the best> of my limited
understanding.> > but it's the concept of what to look for and
how that puzzles me. sometimes> I do get nice backtest numbers, at
least moderately nice. but if I don't> understand why that should
be possible given what's been said about the> dynamics of market
evolution, why should I believe them?> > I keep coming back to
the same question that I'm sure you're all sick to> death of, I
know I am: if auto-optimization isn't very successful, doesn't>
that imply that past performance tells us very little about the future,
and> if that's the case, how do we develop trading systems?>
> dave> > > ...frankly, it's hard to see
how rational> > trading system design is possible
in> > a world like this. or am I just depressed?>
> I like Edison's attitude when looking for the right
material to use> in the light bull. When yet another
"bright" idea (sorry I could not> resist) failed when
tested, he is reported to have said, "We are> making
progress. We now know of 999 things that will not work.">
After trying everything from bamboo to who knows what, he
eventually> found the right material.>
> Genius is 5% inspiration and 95% perspiration.>
> bSend
BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to
suggest@xxxxxxxxxxxxx-----------------------------------------Post
AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: <A
href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check
group FAQ at: <A
href="">http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the <A
href="">Yahoo! Terms of Service.
Yahoo! Groups Sponsor
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
|