|
PureBytes Links
Trading Reference Links
|
Howard,
Thank you for your patience and detailed reply.
I won't comment further until I have read your book.
No it doesn't seem hard.
Regards,
BrianB2.
--- In amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@xxx> wrote:
>
> Hi Brian --
>
> Thanks for your comments. First my response regarding objective
functions.
>
> The choice of the objective function is strictly a management
decision, and
> must be made before any serious testing begins. All sub-objectives
must be
> combined into The single-valued objective function. The programmer
has
> nothing to do with the choice of the objective function and the
objective
> function is not available for optimization. If the manager happens
to also
> do the programming, then it is with the manager's hat on that the
objective
> function is chosen.
>
> The Quantitative Trading Systems book has a chapter on objective
functions,
> and a lot of material throughout the book discussing the
application of
> objective functions to testing and optimization. There is an
appendix that
> describes in complete detail how to combine sub-objectives to
create a
> single-valued objective function, and then use that objective
function in
> AmiBroker testing and optimization. Whether there is only one
metric that
> is important to the trader / manager or 20, they must each be
evaluated in
> terms of importance, have a range of acceptable values determined,
assigned
> a relative weight, and combined.
>
> Think ahead to the automated walk-forward procedure we all must use
to
> validate a trading system. At the end of the search / optimization
for each
> in-sample period, the Best set of values must be chosen, and these
values
> used to evaluate the out-of-sample period. At the end of the last
available
> in-sample period, the concatenated results from all the out-of-
sample
> periods are evaluated. If those results are acceptable, then the
Best set
> of values from the final in-sample optimization will be used to
trade the
> system with real money. There can be only one Best. If the
manager has an
> opportunity to look at the trades associated with each of the
alternatives
> to the best, and if that manager picks any set of trades -- that
is, trades
> associated with a set of values -- other than the one that is ranked
> highest, then that manager is saying that there are additional
metrics that
> are not in the current objective function, but that must be added
to it.
> The set of values that are ranked highest Must represent the
trading that
> the manager believes is Best for himself, herself, or the
organization.
>
> Switching to questions about in-sample testing and out-of-sample
testing.
>
> The Only results that matter are those observed in Strictly out-of-
sample
> testing. There is No number of in-sample trades that are adequate
to
> validate a trading system. Not 30, not 130! I have documented two
separate
> trading systems that each have over one million (sic) closed trades
> in-sample and appear to be profitable, yet neither of those systems
> validates out-of-sample.
>
> In-sample results are Meaningless!!!! They are always good. We do
not
> quite searching / optimizing until they are good. Basing a
decision to
> trade based on in-sample results is dangerous to the wealth.
>
> Remember -- truly out-of-sample testing begins tomorrow with real
money.
> Only rigorous validation procedures can prepare us for that. And
even if a
> trading system seems to validate, there is no guarantee of future
> profitability -- just an increased probability of future
profitability.
>
> I realize these statements sound hard -- both in tone and in
difficulty of
> implementation. It is Absolutely Necessary that every systems
developer who
> hopes to be profitable understand and implement objective functions
and
> in-sample / out-of-sample testing. Unfortunately, there is no easy
> solution. Until Tomasz provided the fantastic tools within
AmiBroker, there
> was not even a difficult solution.
>
> Thanks for listening,
> Howard
> www.quantitativetradingsystems.com
>
>
>
>
> On 3/17/07, brian.z123 <brian.z123@xxx> wrote:
> >
> > Howard,
> >
> > Thanks for your post.
> > I value the opportunity to talk to quality people on a quality
topic.
> >
> > In laymans terms; does the servitude to the objective function
> > require compromising the multiple sub-objectives?
> > If so who makes that decision; when, where and how is it made?
> > According to the Calresco site, these decisions are made by the
> > *programmer* on a subjective basis.
> >
> > If there are 3 or more sub-objectives with, with 2 or 3 metrics
for
> > each and, say a range of 20 for each of the metrics parameters ,
> > doesn't the range of possible compromises become somewhat wide?
> >
> > If we could extract the metrics for each of those multiple paths
on a
> > point by point basis, I wonder if, with hindsight, *we* might
like to
> > change our prior subjective choices.
> >
> > Re walk OOS/walk-forward.
> >
> > A money game derived from the toss of a fair coin with +1 and -1
> > win/loss values assigned to each side of the coin is played by
1000
> > people.
> > They are not aware of the nature of the game, only their
individual
> > outcomes i.e. their equity curves or trade value time series as
they
> > unfold.
> >
> > An observer, who is a mathematician, knows the game is break-even
and
> > also all of the metrics for the game in advance, but the players
> > don't.
> >
> > At the end of 1000 plays, in all probability, not one single
player
> > will have exactly broken even.
> > Approx 60% of players will have a small win or loss record, while
on
> > the other end of the scale < 5% of players will have either an
> > extreme win or extreme loss outcome.
> > In all probability not one single equity curve will be exactly the
> > same as any other.
> >
> > If one of the extreme winners happened to be a mathematician and
> > decided to calculate his/her future in the game, would muliple
valued
> > objective analysis, arrive at any conclusion other than that it
is a
> > rosy one?
> > By any measure, would the extreme winners not be justified in the
> > belief that their future in the game is rosy?
> >
> > To the observer the 1000 equity curves, when expressed as a
> > probability, represent a good approximation of all possible future
> > outcomes of playing the game, for that exact number of plays.
> > This is true to the basic tenents of maths, which state that the
> > future can only be described in terms of probability.
> >
> > If all of the equity outcomes are written on a marble, placed in a
> > basket, and drawn at random, this test would represent a true
model
> > of a blind or walk forward test.
> >
> > For a trader, who has backtested, the basket also represents
> > historical results.
> >
> > Go ahead and draw your 2 marbles; one for your in sample (IS)
equity
> > curve and one for your out of sample (OOS) equity curve.
> >
> > What is the chance that your first outcome will truely represent
your
> > future in the game?
> >
> > What is the chance that your second equity curve will truely
> > represent your future in the game?
> >
> > What is the chance that your first equity curve will be within
coeee
> > of (Aussie slang for close to) the first?
> > Should we define close as 50%, as Mr Pardo does?
> > Why not 49 or 51?
> >
> > Does the first marble tell you (more/less/the same) about your
future
> > in the game, compared to the second?
> >
> > What are the chances that your first and second marbles will both
be
> > *good* ones?
> > Is that a better or worse *sign* than drawing one *good* one
followed
> > by one *bad* one?
> >
> > BrianB2 *:-)
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com>, "Howard B"
> > <howardbandy@> wrote:
> > >
> > > Greetings --
> > >
> > > I'd like to add a comment on multivalued objective functions,
> > particularly
> > > as they relate to Monte Carlo analysis and walk-forward testing.
> > >
> > > As your trading system development moves to the stage of having
a
> > > walk-forward process performed automatically, it needs to be
guided
> > by an
> > > objective function that incorporates all of the features that
are
> > important
> > > to you and can be expressed as a single scalar value.
> > >
> > > As you know, the walk-forward process divides the entire time
> > series being
> > > used into a sequence of in-sample periods, each followed by an
out-
> > of-sample
> > > period. A search procedure picks the single Best set of values
> > for the
> > > trading system's optimizable variables for a given in-sample
> > period, then
> > > records the results of using those values to simulate trading
over
> > the
> > > out-of-sample period. Then it slides the starting dates for both
> > the
> > > in-sample period and out-of-sample period forward, usually by
the
> > length of
> > > the out-of-sample period, and does the search over again. This
> > process
> > > continues until all of the data has been processed. The results
> > from the
> > > out-of-sample periods are concatenated together and are used to
> > decide
> > > whether the trading system is a good one or not.
> > >
> > > The key point here is this: the search procedure must make its
> > decision on
> > > which set of variables is Best based on a single value -- the
value
> > of the
> > > objective function. By the time the development reaches this
> > stage, we, as
> > > system developers or programmers, will not have an opportunity
to
> > look down
> > > the list of alternative trading systems to see if we would have
> > picked one
> > > other than the one at the top of the list. So, as we are working
> > with
> > > multivalued objective functions, we must incorporate them into a
> > > single-valued objective function that fits our trading
requirements
> > or
> > > personality and that we trust to sort the alternatives into the
> > order we
> > > prefer.
> > >
> > > AmiBroker has the capability to creating custom objective
> > functions. There
> > > is an extensive discussion in my book about objective functions.
> > The
> > > discussion includes an example of using AmiBroker's custom
> > backtester to
> > > create a single-valued objective function by starting with a
central
> > > objective function, then applying penalties (which can be
positive
> > or
> > > negative) to it to take secondary goals into account.
> > >
> > > In fact, I think that objective functions are so important that
> > selection of
> > > the objective function should be the First step in trading
system
> > design.
> > > If the objective function fits the trader, most of the problems
> > related to
> > > the difficulty of following a trading system and of the
psychology
> > of
> > > trading disappear.
> > >
> > > Thanks for listening,
> > > Howard
> > > www.quantitativetradingsystems.com
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 17 Mar 2007 03:06:02 -0700, thomasdrewyallop <drewyallop@>
> > wrote:
> > > >
> > > > Hello all,
> > > >
> > > > I have been working on the MCP technique described in
Aronson's
> > book
> > > > for some time now. I have just completed conversion of the C++
> > code on
> > > > the web site to C# plus some associated utlities to massage AB
> > data
> > > > into the required format yet. No test yet; I will update under
> > this
> > > > thread.
> > > >
> > > > A few words on the theoretical underpinnings. There has been
new
> > > > information since the book was published and the code
written. I
> > > > believe an update is in the works. Also you need to be
cautious
> > when
> > > > running MCP on IS data. This is only valid under certain
> > conditions.
> > > > Otherwise you must run OOS. I had an email from Aronson
> > explaining all
> > > > this but can't find it. You might want to contact David
directly -
> > a
> > > > good guy and willing to talk with readers.
> > > >
> > > > Finally, I would not reject walk forward. A very useful
technique
> > > > despite Aronson's reservations. Well integrated with AB too
via
> > Fred
> > > > Tonetti's IO add-in.
> > > >
> > > > Best regards,
> > > >
> > > > Drew Yallop
> > > >
> > > > p.s. just remebered that there is discussion on MCP as a
possible
> > > > future addition to AB. Look in the AB suggestions section of
the
> > web site.
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
------------------------ Yahoo! Groups Sponsor --------------------~-->
Transfer from your equities account.
Receive up to $1,000 from GFT. Click here to learn more.
http://us.click.yahoo.com/aZttyC/X_xQAA/cosFAA/GHeqlB/TM
--------------------------------------------------------------------~->
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|