|
PureBytes Links
Trading Reference Links
|
> 2. After the system has been developed and tested, Monte Carlo
>techniques
> can be used to reorder (or re-sample from the estimated
>distribution) the
> Out-of-Sample results to estimate the statistical distribution of
>the equity
> curve and trading results.
IMO that is a must do but others disagree on that point.
Re MCP
I have a high regard for Aronson so I haven't thrown the idea of MCP
away but I'm not excited enough to pay the price of learning the math.
Different story if I was an 'optimizer.
I might study it one day, as an academic exercise, since I make an
effort to learn evaluation well (don't know how anyone trades without
it but some do).
thanks
brian_z
--- In amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@xxx> wrote:
>
> Hi Brian --
>
> Monte Carlo is a name given to a broad category of techniques used
in
> modeling and simulation to evaluate the robustness of a model. The
name
> means different things to different people. The techniques
typically
> involve adding some random component -- for example, adding random
noise or
> reordering results in a random manner.
>
> The one use of Monte Carlo analysis that I am cautioning about is
this:
> 1. Make an in-sample run.
> 2. Reorder the trades observed for the in-sample period.
> 3. Attempt to estimate the robustness of the system.
>
> My point is that simply reordering the in-sample trades will not
provide any
> additional information about the likelihood that the system will be
> profitable when traded with real money.
>
> There are many ways of applying Monte Carlo that Will provide
additional
> information. Here are two examples:
> 1. During any test runs, use Monte Carlo techniques to perturb
either the
> data or the values of the parameters.
> A. The data can be perturbed by adding some amount of random noise.
> B. The parameter values can be perturbed by testing not only the
specific
> value requested, but also a cloud of values nearby.
> If the results of applying the system to perturbed data result in
values of
> the objective function that are similar to those of the specific
value
> requested (before any random component is added), that Does increase
> confidence that the system is robust, and it May increase
confidence that it
> will perform well out-of-sample. An explicit out-of-sample test is
still
> required.
> 2. After the system has been developed and tested, Monte Carlo
techniques
> can be used to reorder (or re-sample from the estimated
distribution) the
> Out-of-Sample results to estimate the statistical distribution of
the equity
> curve and trading results. For example, given a set of OOS
results, if the
> trades happened in a different order, what is the expected, best,
and worst
> case drawdown?
>
> Thanks for listening,
> Howard
>
>
> On Sun, Mar 16, 2008 at 4:43 PM, brian_z111 <brian_z111@xxx> wrote:
>
> > Howard,
> >
> >
> > >(Running Monte Carlo permutations on in-sample results does
Nothing
> > >to improve the likelihood of out-of-sample profitability.)
> >
> > I haven't gone down the MCP path because the math required would
be a
> > steep challenge for me and also my gut feeling for it is NO.
> >
> > (Since I am not a mathematician I often substitute conceptual
> > analysis for ratiocination - conceptually it doesn't add up to a
> > compelling argument IMO).
> >
> > Are you referring to MCP?
> > Are you in the position to elaborate a little further?
> >
> > BTW - thanks for everything you are doing.
> > I'm bouncing off you (and others) - its keeping me fired up.
> >
> > brian_z
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
40yahoogroups.com>, "Howard B"
> > <howardbandy@> wrote:
> > >
> > > Greetings all --
> > >
> > > I received the following email privately. I think the
discussion is
> > > important enough to post it to the group.
> > >
> > > //--------------------------------------
> > >
> > > Dear Howard,
> > >
> > > Thanks for your reply in the Amibroker forum. The topic of how
to
> > > sort optimisation results is an important one.
> > >
> > > You mention several statistics:
> > >
> > > KRatio, RRR, UPI, CAR/MDD, RAR/MDD, Recovery Ratio.
> > >
> > > Do you have any experience in using trading systems optimised
to one
> > > of these?
> > >
> > > I'm finding that by optimising for highest account value, I go
> > > through long periods of drawdown, then a small % of trades make
a
> > > killing (eg: 46% wins, 2:1 Win to Loss ratio overall). This is
not
> > > what I'm looking for.
> > >
> > > I'd like to try and smooth out the equity curve to give more of
a
> > > balance between making a large capital gain and also regular
> > > cashflow. I have the following stats:
> > >
> > > - % Wins
> > > - Win:Loss Ratio (All profits/All losses)
> > > - Risk:Reward
> > >
> > > I have found that by sorting results by Risk:Reward does not
give
> > > good overall account balances.
> > >
> > > //-----------------------------------------------
> > >
> > > The experience that writer has is exactly the point I am making
> > when I say
> > > that net profit is usually a poor objective function to use when
> > developing
> > > trading systems.
> > >
> > > Each of the metrics I mentioned have three positive
characteristics:
> > > 1. they reward equity growth
> > > 2. they penalize drawdowns
> > > 3. when used as The objective function for walk forward testing,
> > they tend
> > > to select systems that perform well out-of-sample
> > >
> > > My feeling is that the choice of objective comes very early on
in
> > the
> > > design, test, and validation process. The objective function
> > incorporates
> > > the features that the trader wants in his or her trading, so
> > systems that
> > > rank well by using the score of that objective function are
systems
> > that the
> > > person knows they will be comfortable with.
> > >
> > > If one of those metrics already built in to AmiBroker is not
> > satisfactory,
> > > it is easy to create your own metric and have AmiBroker report
it
> > for every
> > > run and use it in the walk forward testing. The custom metric
can
> > include
> > > setting limits on percent winners, win to loss ratio, and so
forth.
> > >
> > > There are several posts over the past few days that discuss this
> > and show
> > > examples of how to do it.
> > >
> > > Try out whatever you think might work for you. Keep in mind that
> > optimizing
> > > using an objective function that does not include some penalty
for
> > drawdowns
> > > and / or reward for equity smoothness is likely to result in
> > systems that
> > > have very strange results and do not perform well out-of-sample.
> > >
> > > As always, be sure to evaluate the results by examining out-of-
> > sample
> > > results -- in-sample results have no value in estimating the
likely
> > > out-of-sample performance. NO value. Trading based on in-sample
> > results
> > > alone is a certain way to lose your trading account. (Running
> > Monte Carlo
> > > permutations on in-sample results does Nothing to improve the
> > likelihood of
> > > out-of-sample profitability.)
> > >
> > > Thanks for listening,
> > > Howard
> > > www.quantitativetradingsystems.com
> > >
> > >
> > >
> > > On Fri, Mar 14, 2008 at 8:00 AM, dralexchambers
<dralexchambers@>
> > > wrote:
> > >
> > > > Does anyone use other Optimisation statistics to rank
> > optimisations
> > > > by?
> > > >
> > > > For example, instead of using raw profit gained, does anyone
use
> > > > Win:Loss Ratio or Risk:Reward ratio - and what have been your
> > > > experiences.
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
------------------------------------
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|