|
PureBytes Links
Trading Reference Links
|
Very interesting questions and they are not easy ones to find a definitive answer for.
I had a quick read of Howard's QTS book to refresh my memory on the basics.
Using Howard's recommendations as the benchmark (nothing personal but since Howard is an AB user and many of us have the book etc):
- we select our objective function before we start the testing process and this is our measure of goodness ... staying with the W/L ratio, from my previous post, as the example ... we need to define a value, or range of values, for the metric's goodness or badness.
How to do that later.
- at the end of the IS testing we choose the system with the highest quotient of goodness (the best system) ... we have to since we chose the function and managed the test process (we will assume we did that part correctly).
How do we know if the IS W/L ratio is good?
Presumably it is a personal decision..... for some metrics the goodness is predefined e.g. for the K-ratio there is a minimum level recommended by it's creator. Where goodness is not defined then we need to mark out the boundaries for ourselves.
At this step we could apply some objective means to establihs goodness, or lack of it (for this example):
- we could test the market, or the portion of it we want to run our testing on, and establish a W/L ratio benchmark (the one to beat!) ... Howard's book, and the forum, contain examples and discussion on benchmarking for system development.
- we could apply statistical tests to establish significant levels.
- others?
Once the best system is chosen, from the IS candidates, we can proceed to OOS testing with it (once again we will assume we did this correctly).
Once the OOS data is in we need to validate the OOS result.
Howard tells us there are two ways to do this:
- prediction
- statistical validation
What does it actually mean if we say that the system has good predictive ability .... I assume it means that if the IS W/L ratio was good and the OOS W/L ratio is good that the system has good predictive value OR simply that if the OOS W/L ratio is good then the system has good predictive value (that is what you are suggesting isn't it)?
Once again statistical validation comes down to significance tests, and rejection of the null hypothesis, but there are difficulties there ... trade series produced by systems don't necessarily have a normal distribution and significance is only stated in probabilities ... no stats test can produce a definite measure of goodness e.g. there is 5/100 chances that a high W/L ratio could have been produced by a break even system (a fair coin toss) and 0.05*0.05 chance that a break even system could produce two trials in a row with significant W/L ratios.
Also - I don't think significance tests for non-normal dists are as accurate as those for normal dists?
Relating your questions to the above:
- If the OOS sample is validated then it validates the IS sample (we won't split hairs over how the samples were collected at this stage).
The anchored IS sample then has value ... it is a larger sample than the OOS test. I haven't stipulated what it is useful for ... it might have more than one use.... give me a while and I might think of something!
- Is the objective function metric (the W/L ratio in this case), as measured by the IS test, relevant once the OOS sample has been validated?
Howard states that the OOS is often lower than the IS (remember that he is an optimizer so his viewpoint is biased towards typical opt outcomes).
Not all systems are designed using IS optimization, or even IS testing for that matter.
IMO there is room for further thought on that subject.
- Is the ratio of the OOS metric to the combined IS + OOS metric relevant?
This was Mr Pardo's recommendation, not mine.
In my previous post I explained why the ratio was unlikely to be useful.
The numbers I used in the example aren't correct .. they are actually conservative but even then they prove that a good deal of caution is necessary when using significance for validation purposes.
- If the OOS sample is good what else do I need?
Once again you have to measure goodness and then validate it.
Assuming that the IS sample is worthless .... is it realistic to suggest that you are going to do that without reference to the IS sample? No need to answer that question, however, because it doesn't make much difference in the long run.
So far we only have two measures of goodness .. beat the market and statistical significance.
Significance testing on an OOS sample, with or without regard to the IS test, is problematic (as discussed) .... if the IS counts then you could be experiencing a 2.5/1000 probability event ... if it doesn't then you could be experiencing a 5/100 event and that doesn't change the probability of a worthless system producing a significantly good result when traded live.
Beat the market metrics, are also problematic ... if we select a statistically valid number of trades, randomly, from some OOS data we will beat the market, on average, 50% of the time (I haven't tried this ... it is a prediction on my part). That then leaves us in the position of having to decide if the amount we beat the market by, in our OOS sample, is statistically significant.
--- In amibroker@xxxxxxxxxxxxxxx, Keith McCombs <kmccombs@xxx> wrote:
>
> I've been told that no question is a "dumb question". So here goes:
> If I have a system with good OOS performance, why should I care what the
> IS performance is? And similarly, why should I care what the OOS/IS
> ratio is?
>
> Couldn't it be more important that I have a high OOS/BH (buy and hold)
> ratio, so that I don't "confuse brains with a bull market"? Or at least
> something that gives me confidence that I haven't just accidentally
> stumbled on a once in a lifetime event, that, of coarse, will disappear
> the minute I start trading real money? Does OOS/IS ratio somehow help?
> -- Keith
>
> brian_z111 wrote:
> >
> >
> > > I also create a t-test of the ave returns.
> >
> > How do you do that?
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <mailto:amibroker%40yahoogroups.com>,
> > Rajiv Arya <rajivarya87@> wrote:
> > >
> > >
> > > I also create a t-test of the ave returns.
> > >
> > > The in-sample is almost always significant
> > >
> > > And try to have the out of sample t-test greater than 1.64, which
> > happens for about 50% for the out-of sample results.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > To: amibroker@xxxxxxxxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> > > From: dloyer123@
> > > Date: Sat, 9 May 2009 03:03:16 +0000
> > > Subject: [amibroker] Re: Expectancy - and related--specifically K-rato
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx
> > <mailto:amibroker%40yahoogroups.com>, Rajiv Arya <rajivarya87@> wrote:
> > > >
> > > >
> > > > I like to compute a ratio of the out-sample metric and divide it
> > by the in-sample metric.
> > > >
> > > > And I like to look for multiple runs of out-sample/in-sample ratio
> > to be above 0.5 and with little fluctuation.
> > > >
> > >
> > > That is similar to Pardo's WFE (Walk forward efficiency), or a
> > measure of how much curve fitting inflated test results. Pardo
> > suggests taking the concatenated out of sample returns and divide by
> > the result treating the entire combined data set as in sample.
> > Anything below 0.65 will probably not trade well live. The higher, the
> > better.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > __________________________________________________________
> > > Hotmail® has a new way to see what's up with your friends.
> > >
> > http://windowslive.com/Tutorial/Hotmail/WhatsNew?ocid=TXT_TAGLM_WL_HM_Tutorial_WhatsNew1_052009
> > <http://windowslive.com/Tutorial/Hotmail/WhatsNew?ocid=TXT_TAGLM_WL_HM_Tutorial_WhatsNew1_052009>
> > >
> >
> >
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|