|
PureBytes Links
Trading Reference Links
|
<snip> My understanding of Bob Pardo's book is that he feels that the length of the
out-of-sample period can be determined by a calculation based on the length
of the in-sample period. If that were true, then the ratio would be a
little easier to compute. But, in my experience, there is no relationship
between the length of the out-of-sample period and the length of the
in-sample period. And gathering the data and performing the calculation of
the ratio for some objective functions would be difficult.<snip>
I don't believe that it would be worth the effort but setting tbe BT the task of collecting the same number of trades in the OOS test as were collected in the IS test is the way to standardise and simplify calculating this ratio.
N == the number of closed trades in a test run;
Where IS(N) == OOS(N) the ratio IS(N bars)/OOS(N bars) will always be defined as a probability distribution.
No constant relationsip, between IS and OOS length (in bars) can exist because of the (near) random nature of the markets.
Believe that such a relationship does exist, and hence can be calculated/exploited, is an artefact of the belief that the markets are non random i.e. have a structure that we can identify, model and therefore use to make predictions about the future of the markets.
Any attempt to improve OOS performance by 'tuning' the length of the sample periods is a futile excercise.
The exception to this, as you point out in your book, is that the markets do change (they are not quite random .... human behaviour leaves a faint trail of non randomness) and so current data may be more relevant than non- current data e.g. increased use of computers, and access to information, by traders may have changed the markets in the last decade or two.
Whether these changes are strong enough, and occur over short enough time periods or identifiable time periods, to justify the view that we can/must account for this, in our design processes, is arguable i.e. it is questionable that we can "synchronise the market and our trading system by shortening (varying) the time periods used in our walk forward testing", as you suggested in your QTS book (P261).
Granted that we are attempting to synchronise our systems to market behaviour but this is done within our system rules, not by altering the N bars tested.
The fundamental premise of trading is that the system will perform, in the future, irrespective of market conditions or the time period involved.
The OOS walk through is a test to find out if the patterns we identified in the IS data still exist in the future and that the code we used to synchronise to that pattern(s) is effective.
The only time that will be time dependent is when the patterns are time based e.g. the number of times the price of oil goes up on the first day of the month is statistically significant.
Very few exploitable inefficiencies in the markets are time based.
On another point:
Theoretically the OOS results should be better/worse than the IS tests on a 50/50 basis.
Continual skewing of that ratio to the downside is an indicator of curve fitting at the IS stage?
--- In amibroker@xxxxxxxxxxxxxxx, Howard B <howardbandy@xxx> wrote:
>
> Greetings all --
>
> In-sample results and out-of-sample results can be, and usually are, very
> different in their characteristics.
>
> My experience is that the ratio (OOS/(IS+OOS)), where these are the
> in-sample and out-of-sample results, is difficult to compute and often even
> difficult to define. I have not found it to be of value in estimating the
> future performance of the system.
>
> My understanding of Bob Pardo's book is that he feels that the length of the
> out-of-sample period can be determined by a calculation based on the length
> of the in-sample period. If that were true, then the ratio would be a
> little easier to compute. But, in my experience, there is no relationship
> between the length of the out-of-sample period and the length of the
> in-sample period. And gathering the data and performing the calculation of
> the ratio for some objective functions would be difficult.
>
> Isn't it the out-of-sample results we are trying to estimate? Do we care
> what the in-sample results look like? If the out-of-sample results are
> terrible, why bother computing the ratio. If the out-of-sample results are
> good, why bother computing the ratio -- how will that information be used to
> improve the system? And if it is used to modify the system, then the
> previously out-of-sample data has become in-sample.
>
> Do any of the forum member have examples they can contribute where computing
> the ratio is helpful?
>
> Thanks,
> Howard
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|