Re: [amibroker] Re: Detecting data mining bias with modified Monte Carlo procedure, AmiBroker Email List Archive

On 3/17/07, brian.z123 <brian.z123@xxxxxxxxxxxx> wrote:

Howard,

Thanks for your post.
I value the opportunity to talk to quality people on a quality topic.

In laymans terms; does the servitude to the objective function
require compromising the multiple sub-objectives?
If so who makes that decision; when, where and how is it made?
According to the Calresco site, these decisions are made by the
*programmer* on a subjective basis.

If there are 3 or more sub-objectives with, with 2 or 3 metrics for
each and, say a range of 20 for each of the metrics parameters ,
doesn't the range of possible compromises become somewhat wide?

If we could extract the metrics for each of those multiple paths on a
point by point basis, I wonder if, with hindsight, *we* might like to
change our prior subjective choices.

Re walk OOS/walk-forward.

A money game derived from the toss of a fair coin with +1 and -1
win/loss values assigned to each side of the coin is played by 1000
people.
They are not aware of the nature of the game, only their individual
outcomes i.e. their equity curves or trade value time series as they
unfold.

An observer, who is a mathematician, knows the game is break-even and
also all of the metrics for the game in advance, but the players
don't.

At the end of 1000 plays, in all probability, not one single player
will have exactly broken even.
Approx 60% of players will have a small win or loss record, while on
the other end of the scale < 5% of players will have either an
extreme win or extreme loss outcome.
In all probability not one single equity curve will be exactly the
same as any other.

If one of the extreme winners happened to be a mathematician and
decided to calculate his/her future in the game, would muliple valued
objective analysis, arrive at any conclusion other than that it is a
rosy one?
By any measure, would the extreme winners not be justified in the
belief that their future in the game is rosy?

To the observer the 1000 equity curves, when expressed as a
probability, represent a good approximation of all possible future
outcomes of playing the game, for that exact number of plays.
This is true to the basic tenents of maths, which state that the
future can only be described in terms of probability.

If all of the equity outcomes are written on a marble, placed in a
basket, and drawn at random, this test would represent a true model
of a blind or walk forward test.

For a trader, who has backtested, the basket also represents
historical results.

Go ahead and draw your 2 marbles; one for your in sample (IS) equity
curve and one for your out of sample (OOS) equity curve.

What is the chance that your first outcome will truely represent your
future in the game?

What is the chance that your second equity curve will truely
represent your future in the game?

What is the chance that your first equity curve will be within coeee
of (Aussie slang for close to) the first?
Should we define close as 50%, as Mr Pardo does?
Why not 49 or 51?

Does the first marble tell you (more/less/the same) about your future
in the game, compared to the second?

What are the chances that your first and second marbles will both be
*good* ones?
Is that a better or worse *sign* than drawing one *good* one followed
by one *bad* one?

BrianB2 *:-)

--- In amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@xxx> wrote:
>
> Greetings --
>
> I'd like to add a comment on multivalued objective functions,
particularly
> as they relate to Monte Carlo analysis and walk-forward testing.
>
> As your trading system development moves to the stage of having a
> walk-forward process performed automatically, it needs to be guided
by an
> objective function that incorporates all of the features that are
important
> to you and can be expressed as a single scalar value.
>
> As you know, the walk-forward process divides the entire time
series being
> used into a sequence of in-sample periods, each followed by an out-
of-sample
> period. A search procedure picks the single Best set of values
for the
> trading system's optimizable variables for a given in-sample
period, then
> records the results of using those values to simulate trading over
the
> out-of-sample period. Then it slides the starting dates for both
the
> in-sample period and out-of-sample period forward, usually by the
length of
> the out-of-sample period, and does the search over again. This
process
> continues until all of the data has been processed. The results
from the
> out-of-sample periods are concatenated together and are used to
decide
> whether the trading system is a good one or not.
>
> The key point here is this: the search procedure must make its
decision on
> which set of variables is Best based on a single value -- the value
of the
> objective function. By the time the development reaches this
stage, we, as
> system developers or programmers, will not have an opportunity to
look down
> the list of alternative trading systems to see if we would have
picked one
> other than the one at the top of the list. So, as we are working
with
> multivalued objective functions, we must incorporate them into a
> single-valued objective function that fits our trading requirements
or
> personality and that we trust to sort the alternatives into the
order we
> prefer.
>
> AmiBroker has the capability to creating custom objective
functions. There
> is an extensive discussion in my book about objective functions.
The
> discussion includes an example of using AmiBroker's custom
backtester to
> create a single-valued objective function by starting with a central
> objective function, then applying penalties (which can be positive
or
> negative) to it to take secondary goals into account.
>
> In fact, I think that objective functions are so important that
selection of
> the objective function should be the First step in trading system
design.
> If the objective function fits the trader, most of the problems
related to
> the difficulty of following a trading system and of the psychology
of
> trading disappear.
>
> Thanks for listening,
> Howard
> www.quantitativetradingsystems.com
>
>
>
>
>
>
> On 17 Mar 2007 03:06:02 -0700, thomasdrewyallop <drewyallop@xxx>

wrote:
> >
> > Hello all,
> >
> > I have been working on the MCP technique described in Aronson's
book
> > for some time now. I have just completed conversion of the C++
code on
> > the web site to C# plus some associated utlities to massage AB
data
> > into the required format yet. No test yet; I will update under
this
> > thread.
> >
> > A few words on the theoretical underpinnings. There has been new
> > information since the book was published and the code written. I
> > believe an update is in the works. Also you need to be cautious
when
> > running MCP on IS data. This is only valid under certain
conditions.
> > Otherwise you must run OOS. I had an email from Aronson
explaining all
> > this but can't find it. You might want to contact David directly -
a
> > good guy and willing to talk with readers.
> >
> > Finally, I would not reject walk forward. A very useful technique
> > despite Aronson's reservations. Well integrated with AB too via
Fred
> > Tonetti's IO add-in.
> >
> > Best regards,
> >
> > Drew Yallop
> >
> > p.s. just remebered that there is discussion on MCP as a possible
> > future addition to AB. Look in the AB suggestions section of the
web site.
> >
> >
> >
>