|
PureBytes Links
Trading Reference Links
|
Greetings all --
I am coming to this discussion a little late. I just returned from giving a talk at the NAAIM conference in Irvine. Some of the discussions I had with conference attendees was exactly the topic of this thread.
If you are using some data and results to guide the selection of logic and parameter values (as described in the earliest postings as OOS data), that incorporates that data into the In-Sample data set. In this case, there must be three data sets. They go by various names -- Training, Guiding, and Validation will be adequate for now.
Optimization, by itself, begins by generating a lot of alternatives. Optimization with selection of the "best" alternatives means using an objective function (or fitness function) to assign a score to each alternative.
The method of searching for good trading systems used in AmiBroker's automated walk forward procedure uses a series of: search over an in-sample period, select the best using the score, test over the out-of-sample period. Use the concatenated results from the out-of-sample periods to decide whether to trade the system or not.
Another method of searching for good systems (that might be what some of the posters to this thread were suggesting) is to perform extensive searches of the data and manipulations of the logic using the Training data, then evaluate using the Guiding data. Repeat this process as desired or required as long as the results using the Guiding data continue to improve. When they show signs of having peaked, roll back to the system that produced the best result up to that point. Then make one evaluation using the Validation data. Now, step forward in time and repeat the process. It is now the concatenated results of the Validation data sets that are used to decide whether to trade the system or not.
Thanks,
Howard
On Thu, May 8, 2008 at 9:24 AM, Edward Pottasch < empottasch@xxxxxxxxx> wrote:
thanks. Will have a look,
Ed
----- Original Message -----
Sent: Thursday, May 08, 2008 5:42
PM
Subject: [amibroker] Re: Fitness Criteria
that incorporates Walk Forward Result
There's a simple example of this in the UKB under Intelligent
Optimization ...
--- In amibroker@xxxxxxxxxxxxxxx,
"Edward Pottasch" <empottasch@xxx>
wrote:
>
>
hi,
>
> "While optimization can be employed to search for a good
system via
> methods utilizing automated rule creation, selection and
combination
> or generic pattern recognition"
>
>
anyone care to explain how this works? Some kind of inversion
technique?
Here is what I want now give me the rules to get there :)
>
>
thanks,
>
> Ed
>
>
>
> ----- Original
Message -----
> From: Fred
> To: amibroker@xxxxxxxxxxxxxxx
> Sent: Thursday, May 08, 2008 2:37 PM
> Subject: [amibroker] Re:
Fitness Criteria that incorporates Walk
Forward Result
>
>
> While optimization can be employed to search for a good system
via
> methods utilizing automated rule creation, selection and
combination
> or generic pattern recognition most people typically
use
optimization
> to search for a good set of parameter values.
The success of the
> latter of course assumes one has a good rule set
i.e. system to
begin
> with.
>
> As far as your
prediction is concerned ... I suspect there are
lots
> of people,
some of who post here, who could demonstrate otherwise
if
> they
chose to ...
>
> --- In amibroker@xxxxxxxxxxxxxxx,
"brian_z111" <brian_z111@>
wrote:
> >
> > "IS
metrics are always good because we keep optimizing until
they
>
> are" (or words to that effect by HB) which is true.
> >
>
> It is not until we submit the system to an unknown sample,
either
> an
> > OOS test, paper or live trading that we validate the
system.
> >
> > Discussing your points:
> >
> > IMO we are talking about two different trading approaches, or
> styles
> > (there is no reason we can't understand both
very well).
> >
> > One is the search for a good system,
via optimization, with the
> > attendant subsequent tuning of the
system to match a changing
> market.
> >
> > If I
understand Howard correctly he is an exponent of this
style.
> >
> > It is my prediction that where we are optimising, using
lookback
> > periods, that the max possible PA% return will be
around 30,
maybe
> > 40, for EOD trading.
> >
>
> Do we ever optimise anything other than indicators with
lookback
> > periods?
> > If so that might be a different
story.
> >
> > Bastardising Marshall McCluhans famous line
I could say "the
> > optimization is the method".
> >
> > It is also possible to conceptually optimize the system, before
> > testing, to the point that little, or no, optimization is
required
> > (experienced traders with a certain disposition do
this quite
> > comfortably but it doesn't suit the inexperienced
and/or those
who
> > don't have the temperament for it).
>
>
> > So, if a system has a sound reason to exist, and it is not
> optimized
> > at all, and it has a statistically valid IS
test then it his
highly
> > likely to be a robust system,
especially if it is robust across
a
> > range of
stocks/instruments.
> > The chances that this is due to pure luck are
probably longer
than
> > the chance that an optimized IS test,
with a confirming OOS
test,
> is
> > also a chance
event.
> >
> > However, if I had plenty of data e.g. I was
an intraday trader,
> then
> > I would go ahead and do an OOS
test anyway (since the cost is
> > negligible)
> >
>
> Re testing on several stocks.
> >
> > If the system is
'good' on one symbol, (the sample size is
valid)
> and
>
> it is also good on a second symbol (also with a valid sample
size)
> is
> > that any different from performing an IS and an OOS
test?
> >
> > For stock trading, I call the relative
performance, on a set of
> > symbols, 'vertical' testing as compared
to 'horizontal' testing
> > (where horizontal testing is an equity
curve).
> >
> > Yes, if an IS test, with no optimization,
beat the buy & hold
on
> > every occasion (or a significant
number of times) in a vertical
> test
> > and the sum of that
test was statistically valid and the
horizontal
> > test (the
combined equity curve) was 'good' it would give you
> > something to
think about for sure.
> > If some of the symbols, in the vertical
stack, had contrary
> returns,
> > compared to the bias of my
system, I probably would start to
get a
> > little
excited.
> >
> > (I think perhaps you were alluding to
something along those
lines).
> >
> > BTW did you know
that the Singapore Slingers play in the
Australian
> >
basketball league?
> >
> > Cheers,
> >
>
> brian_z
> >
>
__._,_.___
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
![]()
__,_._,___
|