|
PureBytes Links
Trading Reference Links
|
Optimizing can overfit if misused. I personally think it's a good
tool to "assess the quality of degrees of freedom" as Eckhardt put it
in the interview I gave you the link to.
--- In amibroker@xxxxxxxxxxxxxxx, "brian.z123" <brian.z123@xxx> wrote:
>
> I'm not discounting your and Fred's testimony.
>
> Is this an acknowledgement that optimising does overfit?
>
>
> BrianB2.
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
> <quanttrader714@> wrote:
> >
> > I've been doing this for a long time and trust me, an OOS test is
> > about an order of magnitude more important than an IS test of the
> same
> > size... sometimes less, but sometimes even more. And if your OOS
> > period includes bearish, bullish and sideways market conditions, an
> > OOS test (and inferences soundly derived from it) are as close to a
> > peep into the future that you'll ever get IMO.
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "brian.z123" <brian.z123@> wrote:
> > >
> > > Fred,
> > >
> > > I haven't studied your I/O work, or any of your other work in
> detail
> > > as yet, but I would wager a bet that new traders could gain more
> > > insight into I/O by *workshopping* your stuff than they can get
> from
> > > 90% of the books and websites etc that are out there.
> > > This is pretty much true of AmiBroker across the board.
> > >
> > > That's a fact that needs to be stated in my book and I am just
> saying
> > > that for you and Tomasz so you both don't have to grapple with
> modesty.
> > >
> > > *****************************************************************
> > > OOS/WF as a peep into the future?
> > >
> > >
> > > I'm sorry Fred but OOS/WF does not constitute a peep into the
> future.
> > >
> > > Last night I debated with myself whether I would rebut your
> arguments
> > > this time around or not and my conclusion was you that would not
> want
> > > me to humour you.
> > >
> > > I will use OOS as the example, as WF is a little more complex
> with
> > > it's multiple, overlapping sample approach.
> > >
> > > OOS is just one sample, no more and no less.
> > > It may be the second one taken but it is still one sample and
> one
> > > sample only.
> > > The point is, that as a look into the future, it is the inferred
> > > behaviour of the (sampling distributed)population that is
> important
> > > and one OOS sample a distributed population does not make.
> > > I am not suggesting that we don't do it, or that I don't do it,
> but
> > > the simple fact is that it is a comfort blanket.
> > >
> > > When we look into the future we can't see a clear picture, we
> can only
> > > see a miasma of probabilities, and we just can not accept that.
> > > It is foreign to our norms and we resist this by denial and
> looking to
> > > find a real time certainty that we can hang our hats on.
> > >
> > > I say that uncertainty is quite embraceable if her features can
> be
> > > vaguely discerned.
> > >
> > > OOS is a not a peep into the future because it is a test sample
> based
> > > on historical data and it provides only one equity curve based
> on that
> > > past history.
> > > Now I don't know about you, but the positive equity curve with a
> high
> > > growth rate is the reason I am playing this game (lucky for me
> it is
> > > good fun at the same time).
> > > Now the chances of that one OOS equity curve being the exact one
> > > received the first time the system is traded are very slim, so
> why is
> > > it so important?
> > >
> > > Well, it's not.
> > >
> > > If any large data sample of trades is walked through using
> random
> > > selection there is an infinite number of possible equity curves,
> any
> > > one of which could be *the one* that the trader gets the first
> time
> > > they trade the system.
> > > It is these equity curves that constitute the range of all
> possible
> > > trading futures.
> > > Which one we are going to get we don't know and never will, but
> we can
> > > know our chances of getting any one of them,especially the bad
> ones.
> > >
> > > I will warrant that traders remember their extreme losses more
> vividly
> > > than their extreme wins.
> > >
> > > When enough trade simulations have been conducted, all the
> equity
> > > curves will fall into a recogniseable distribution; one that
> > > approaches a normal curve.
> > > Unless a trader designs a system that never loses, the equity
> curve
> > > distributions will include some minus 4 or 5 standard deviation
> cases
> > > with a probability for each range.
> > > It doesn't matter if that ruinous curve has a 1/1 billion chance
> of
> > > occurring, it can still occur at anytime,even on the first
> trading
> > > excursion with the system i.e. during the OOS test.
> > > For that reason I say that in some cases we throw out perfectly
> good
> > > trading systems because wild chance sometimes throws up a poor
> > > outcome, from a good system, during our OOS test.
> > > The chances of this happening in the OOS test are exactly the
> same as
> > > happening in the first or any other real time trading scenario.
> > >
> > > If we have do one OOS test and it gets our confidence up, should
> we
> > > then do another one, and another one?
> > > Then we could really start trading the system with confidence>
> > >
> > > The trading commentators I have read don't offer any firm
> criteria for
> > > deciding if a OOS test is acceptable or not.
> > > They generally give a wishy-washy definition e.g. *If the OOOS
> test is
> > > within 50% of the back-test results*.
> > > Why 50% and what does that mean anyway?
> > >
> > > My claim is that we can be definite about it.
> > > If fact we can be just as definite as the margin of error
> provided by
> > > our sampling technique and subsequent statistical evaluation.
> > >
> > > In another post I will provide a simple MCS example, that to my
> mind
> > > demonstrates this quite clearly.
> > >
> > > Anyone still in doubt after that should wait for the *piece de
> > > resistance*, the Random Walk, which ties it all together
> irrefutably
> > > for anyone who is basing their decisions on pure logic.
> > >
> > >
> > > BrianB2
> > >
> > >
> > >
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx, "Fred" <ftonetti@> wrote:
> > > >
> > > > OOS and/or WF Testing is not a concept invented in or for IO
> so that
> > > > particular piece of software is not really the issue per se
> except
> > > that
> > > > IO has facilitated making it considerably easier to perform.
> > > >
> > > > "If one OOS test had a 50% drawdown it doesn't say that much
> about
> > > the
> > > > system. It only says something about that one single OOS test
> of
> > > > however many samples."
> > > >
> > > > It doesn't ? ... It speaks volumes to me ... From my
> perspective,
> > > > decent in sample performance, whether or not one applies MCS
> after
> > > the
> > > > fact, is not the end of system testing, it is only a milestone
> along
> > > > the way to developing a system that MIGHT be tradable ...
> > > >
> > >
> >
>
Content-Description: "AVG certification"
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.14.11/542 - Release Date: 11/20/2006
|