Howard,
Thanks for a very nice summary of the framework.
I would say that, since the training search is exhaustive (therefore
we must have identified all possible candidates for the strategy) the
best we can hope for, in the guidance phase, is to change our choice
of top model to one or another of the 'training top models', or
abandon the strategy altogether.
Also I wonder, if the training model/guidance model combination, that
passes a minimum requirement in both phases, and shows less variance
between the training and guidance results, is the most generic model
of them all i.e. suited to a wider range of conditions but not
necessarily returning the highest possible result in any particular
market?
brian_z
--- In amibroker@xxxxxxxxxps.com,
"Howard B" <howardbandy@...> wrote:
>
> Greetings all --
>
> I am coming to this discussion a little late. I just returned from
giving a
> talk at the NAAIM conference in Irvine.
Some of the discussions I
had with
> conference attendees was exactly the topic of this thread.
>
> If you are using some data and results to guide the selection of
logic and
> parameter values (as described in the earliest postings as OOS
data), that
> incorporates that data into the In-Sample data set. In this case,
there
> must be three data sets. They go by various names -- Training,
Guiding, and
> Validation will be adequate for now.
>
> Optimization, by itself, begins by generating a lot of alternatives.
> Optimization with selection of the "best" alternatives means
using
an
> objective function (or fitness function) to assign a score to each
> alternative.
>
> The method of searching for good trading systems used in AmiBroker's
> automated walk forward procedure uses a series of: search over an
in-sample
> period, select the best using the score, test over the out-of-sample
> period. Use the concatenated results from the out-of-sample
periods to
> decide whether to trade the system or not.
>
> Another method of searching for good systems (that might be what
some of the
> posters to this thread were suggesting) is to perform extensive
searches of
> the data and manipulations of the logic using the Training data,
then
> evaluate using the Guiding data. Repeat this process as desired or
required
> as long as the results using the Guiding data continue to improve.
When
> they show signs of having peaked, roll back to the system that
produced the
> best result up to that point. Then make one evaluation using the
Validation
> data. Now, step forward in time and repeat the process. It is now
the
> concatenated results of the Validation data sets that are used to
decide
> whether to trade the system or not.
>
> Thanks,
> Howard
>
> On Thu, May 8, 2008 at 9:24 AM, Edward Pottasch <empottasch@...>
> wrote:
>
> > thanks. Will have a look,
> >
> > Ed
> >
> >
> >
> > ----- Original Message -----
> > *From:* Fred <ftonetti@xx.>
> > *To:* amibroker@xxxxxxxxxps.com
> > *Sent:* Thursday, May 08, 2008 5:42 PM
> > *Subject:* [amibroker] Re: Fitness Criteria that incorporates
Walk Forward
> > Result
> >
> > There's a simple example of this in the UKB under Intelligent
> > Optimization ...
> >
> > --- In amibroker@xxxxxxxxxps.com,
"Edward Pottasch" <empottasch@>
> > wrote:
> > >
> > > hi,
> > >
> > > "While optimization can be employed to search for a good
system
via
> > > methods utilizing automated rule creation, selection and
> > combination
> > > or generic pattern recognition"
> > >
> > > anyone care to explain how this works? Some kind of inversion
> > technique? Here is what I want now give me the rules to get
there :)
> > >
> > > thanks,
> > >
> > > Ed
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From: Fred
> > > To: amibroker@xxxxxxxxxps.com
<amibroker%40yahoogroups.com>
> > > Sent: Thursday, May 08, 2008 2:37 PM
> > > Subject: [amibroker] Re: Fitness Criteria that incorporates Walk
> > Forward Result
> > >
> > >
> > > While optimization can be employed to search for a good system
> > via
> > > methods utilizing automated rule creation, selection and
> > combination
> > > or generic pattern recognition most people typically use
> > optimization
> > > to search for a good set of parameter values. The success of the
> > > latter of course assumes one has a good rule set i.e. system to
> > begin
> > > with.
> > >
> > > As far as your prediction is concerned ... I suspect there are
> > lots
> > > of people, some of who post here, who could demonstrate
otherwise
> > if
> > > they chose to ...
> > >
> > > --- In amibroker@xxxxxxxxxps.com
<amibroker%40yahoogroups.com>,
> > "brian_z111" <brian_z111@>
> > wrote:
> > > >
> > > > "IS metrics are always good because we keep optimizing
until
> > they
> > > > are" (or words to that effect by HB) which is true.
> > > >
> > > > It is not until we submit the system to an unknown sample,
> > either
> > > an
> > > > OOS test, paper or live trading that we validate the
system.
> > > >
> > > > Discussing your points:
> > > >
> > > > IMO we are talking about two different trading approaches,
or
> > > styles
> > > > (there is no reason we can't understand both very well).
> > > >
> > > > One is the search for a good system, via optimization, with
the
> > > > attendant subsequent tuning of the system to match a
changing
> > > market.
> > > >
> > > > If I understand Howard correctly he is an exponent of this
> > style.
> > > >
> > > > It is my prediction that where we are optimising, using
> > lookback
> > > > periods, that the max possible PA% return will be around
30,
> > maybe
> > > > 40, for EOD trading.
> > > >
> > > > Do we ever optimise anything other than indicators with
> > lookback
> > > > periods?
> > > > If so that might be a different story.
> > > >
> > > > Bastardising Marshall McCluhans famous line I could say
"the
> > > > optimization is the method".
> > > >
> > > > It is also possible to conceptually optimize the system,
before
> > > > testing, to the point that little, or no, optimization is
> > required
> > > > (experienced traders with a certain disposition do this
quite
> > > > comfortably but it doesn't suit the inexperienced and/or
those
> > who
> > > > don't have the temperament for it).
> > > >
> > > > So, if a system has a sound reason to exist, and it is not
> > > optimized
> > > > at all, and it has a statistically valid IS test then it
his
> > highly
> > > > likely to be a robust system, especially if it is robust
across
> > a
> > > > range of stocks/instruments.
> > > > The chances that this is due to pure luck are probably
longer
> > than
> > > > the chance that an optimized IS test, with a confirming OOS
> > test,
> > > is
> > > > also a chance event.
> > > >
> > > > However, if I had plenty of data e.g. I was an intraday
trader,
> > > then
> > > > I would go ahead and do an OOS test anyway (since the cost
is
> > > > negligible)
> > > >
> > > > Re testing on several stocks.
> > > >
> > > > If the system is 'good' on one symbol, (the sample size is
> > valid)
> > > and
> > > > it is also good on a second symbol (also with a valid
sample
> > size)
> > > is
> > > > that any different from performing an IS and an OOS test?
> > > >
> > > > For stock trading, I call the relative performance, on a
set
of
> > > > symbols, 'vertical' testing as compared to 'horizontal'
testing
> > > > (where horizontal testing is an equity curve).
> > > >
> > > > Yes, if an IS test, with no optimization, beat the buy
& hold
> > on
> > > > every occasion (or a significant number of times) in a
vertical
> > > test
> > > > and the sum of that test was statistically valid and the
> > horizontal
> > > > test (the combined equity curve) was 'good' it would give
you
> > > > something to think about for sure.
> > > > If some of the symbols, in the vertical stack, had contrary
> > > returns,
> > > > compared to the bias of my system, I probably would start
to
> > get a
> > > > little excited.
> > > >
> > > > (I think perhaps you were alluding to something along those
> > lines).
> > > >
> > > > BTW did you know that the Singapore Slingers play in the
> > Australian
> > > > basketball league?
> > > >
> > > > Cheers,
> > > >
> > > > brian_z
> > > >
> > >
> >
> >
> >
>