|
PureBytes Links
Trading Reference Links
|
<snip> That is similar to Pardo's WFE (Walk forward efficiency), or a measure of how much curve fitting inflated test results. Pardo suggests taking the concatenated out of sample returns and divide by the result treating the entire combined data set as in sample. Anything below 0.65 will probably not trade well live. The higher, the better.<snip>
There is where the subject becomes very interesting ... it is hard to find an absolute answer i.e. find a firm footing that doesn't slip out from under us e.g.
- is 0.65 an empirical number ... where did Pardo get it from ... experience or by calculation (based on observations)?
- this is one place where Howard and I disagree .. Howard is emphatic that the metrics from the IS are worthless whereas I think that if the OOS encourages us to continue then the combined IS and OOS provides us with useful stats ... I expect Howard reiterates that point, without compromise, as a teaching device.
- examining the principles ... using W/L as the example ... say our IS test returns W/L 65/35 and our PayOffRatio is 1/1 ... this translates into equity curves etc eventually i.e. once we introduce MM. We use the OOS metric to decide if the IS development process was successful but how exactly do we do that? Presumably if the IS metric was good and the OOS is the same +- X% we are happy i.e. if the OOS W/L == 65/35 and the PR == 1/1 i.e. matches the IS, then we can realistically expect to achieve the same eq curves, providing we trade over a stationary market (stationary == the same mean and dispersion as the market used for our IS dataset?).
How much X% should we tolerate and does it vary from metric to metric?
Comparing it to our benchmark, the null hypothesis (a fair coin toss) ... from my labtests, variance == sample error.
So the ratio of the IS metric/OOS metric depends on the size of the sample:
IS W/L +- sample error/OOS W/L +- sample error == ratio;
The concatenated IS and OOS creates a larger sample and converges on the mean, as N approaches infinity (so this is a valid method afterall?).
The IS is always smaller than the combined IS and OOS and sometimes very small (according to user preference).
Since W/L always == 1, for a fair coin toss, the ratio of IS/(IS + OOS) metric will always equal the sample error difference.
+- sample error *1 == 1 StDev frequency
+- sample error *2 == 2 StDev frequency
etc
So, 100N IS/ infinite N(IS + OOS) ==
1 +- 10%/1 +- 0% == 0.9/1;// 66% of the time
0.8;// with 2 StDev significance
0.7;// for 3 StDev significance.
etc
For biased systems (65/35 W/L) it works in a similar way (give or take a bit)..... it looks like the 0.65 number is an approximation of significant variance.
The really interesting part is that I can't (mathematically) tell the difference between market variance and non-stationarity.
--- In amibroker@xxxxxxxxxxxxxxx, "dloyer123" <dloyer123@xxx> wrote:
>
> --- In amibroker@xxxxxxxxxxxxxxx, Rajiv Arya <rajivarya87@> wrote:
> >
> >
> > I like to compute a ratio of the out-sample metric and divide it by the in-sample metric.
> >
> > And I like to look for multiple runs of out-sample/in-sample ratio to be above 0.5 and with little fluctuation.
> >
>
> That is similar to Pardo's WFE (Walk forward efficiency), or a measure of how much curve fitting inflated test results. Pardo suggests taking the concatenated out of sample returns and divide by the result treating the entire combined data set as in sample. Anything below 0.65 will probably not trade well live. The higher, the better.
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|