Re: [amibroker] Re: Expectancy - and related--specifically K-rato, AmiBroker Email List Archive

----- Original Message -----
From: brian_z11
Date: Sunday, May 10, 2009 8:58 pm
Subject: [amibroker] Re: Expectancy - and related--specifically K-rato
To: amibroker@xxxxxxxxxxxxxxx

> My understanding of Bob Pardo's book is that he feels
> that the length of the
> out-of-sample period can be determined by a calculation based on
> the length
> of the in-sample period. If that were true, then the ratio
> would be a
> little easier to compute. But, in my experience, there is no
> relationshipbetween the length of the out-of-sample period and
> the length of the
> in-sample period. And gathering the data and performing the
> calculation of
> the ratio for some objective functions would be difficult.
>
> I don't believe that it would be worth the effort but setting
> tbe BT the task of collecting the same number of trades in the
> OOS test as were collected in the IS test is the way to
> standardise and simplify calculating this ratio.
>
> N == the number of closed trades in a test run;
> Where IS(N) == OOS(N) the ratio IS(N bars)/OOS(N bars) will
> always be defined as a probability distribution.
>
> No constant relationsip, between IS and OOS length (in bars) can
> exist because of the (near) random nature of the markets.
>
> Believe that such a relationship does exist, and hence can be
> calculated/exploited, is an artefact of the belief that the
> markets are non random i.e. have a structure that we can
> identify, model and therefore use to make predictions about the
> future of the markets.
>
> Any attempt to improve OOS performance by 'tuning' the length of
> the sample periods is a futile excercise.
>
> The exception to this, as you point out in your book, is that
> the markets do change (they are not quite random .... human
> behaviour leaves a faint trail of non randomness) and so current
> data may be more relevant than non- current data e.g. increased
> use of computers, and access to information, by traders may have
> changed the markets in the last decade or two.
>
> Whether these changes are strong enough, and occur over short
> enough time periods or identifiable time periods, to justify the
> view that we can/must account for this, in our design processes,
> is arguable i.e. it is questionable that we can "synchronise the
> market and our trading system by shortening (varying) the time
> periods used in our walk forward testing", as you suggested in
> your QTS book (P261).
>
> Granted that we are attempting to synchronise our systems to
> market behaviour but this is done within our system rules, not
> by altering the N bars tested.
>
> The fundamental premise of trading is that the system will
> perform, in the future, irrespective of market conditions or the
> time period involved.
>
> The OOS walk through is a test to find out if the patterns we
> identified in the IS data still exist in the future and that the
> code we used to synchronise to that pattern(s) is effective.
>
> The only time that will be time dependent is when the patterns
> are time based e.g. the number of times the price of oil goes up
> on the first day of the month is statistically significant.
>
> Very few exploitable inefficiencies in the markets are time based.
>
>
> On another point:
>
> Theoretically the OOS results should be better/worse than the IS
> tests on a 50/50 basis.
>
> Continual skewing of that ratio to the downside is an indicator
> of curve fitting at the IS stage?
>
>
>
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, Howard B wrote:
> >
> > Greetings all --
> >
> > In-sample results and out-of-sample results can be, and
> usually are, very
> > different in their characteristics.
> >
> > My experience is that the ratio (OOS/(IS+OOS)), where these
> are the
> > in-sample and out-of-sample results, is difficult to compute
> and often even
> > difficult to define. I have not found it to be of value in
> estimating the
> > future performance of the system.
> >
> > My understanding of Bob Pardo's book is that he feels that the
> length of the
> > out-of-sample period can be determined by a calculation based
> on the length
> > of the in-sample period. If that were true, then the ratio
> would be a
> > little easier to compute. But, in my experience, there is no
> relationship> between the length of the out-of-sample period and
> the length of the
> > in-sample period. And gathering the data and performing the
> calculation of
> > the ratio for some objective functions would be difficult.
> >
> > Isn't it the out-of-sample results we are trying to estimate?
> Do we care
> > what the in-sample results look like? If the out-of-sample
> results are
> > terrible, why bother computing the ratio. If the out-of-
> sample results are
> > good, why bother computing the ratio -- how will that
> information be used to
> > improve the system? And if it is used to modify the system,
> then the
> > previously out-of-sample data has become in-sample.
> >
> > Do any of the forum member have examples they can contribute
> where computing
> > the ratio is helpful?
> >
> > Thanks,
> > Howard
> >
>
>
>