|
PureBytes Links
Trading Reference Links
|
An extension too my comments (for those who archive and/or correlate
posts).
MCS is currently the only method I have for obtaining statistics that
I use for estimating Portfolio risk i.e. variance of the equity
outcomes (I want to know the worst case scenario per trade system and
also the same at a portfolio level).
Essentially I am using MCS to report on variance (for various
statistics).
I am not arguing the rights or wrongs of this approach, or saying it
is the only, or best, way to do it - it is the only method I have, at
this time.
I am (sort of) researching (off and on) to find a way to make
a 'quick estimate' of the stats that MCS reports, so I can cut out
the processing time (do it on the fly 'in me head') and also to
improve my understanding of the subject (by pulling it apart and
seeing if I can put it back together).
My tentative hypothesis is this:
a) Trade series analysis is a special (limited) class of maths (a
subset of statistics) because it is binomial (every event is either
win or loss) and because 'we' control the dataset (it is not an
entirely unknown sample set since we place our own stops OR elect not
to).
b) The binomoial variance is known.
c) The distribution of the trade values is not normal but it quickly
approaches normality after 30 trades (central limit theorem)
therefore we can quickly and easily estimate variance of the values.
d) binomial probability is well known (to mathematicians).
e) the distribution of the possible equity outcomes is lognormal
f) the 'product' of variance is a well known maths method
Based on this my view is that there is a possibility a one off
equation can summarize the simulation process (for this limited
class).
I have got part of the way with this 'research' (although my naive
methods hardly qualify as math research) and intend to do a bit more,
here and there.
After that (for something to do on the weekends) I might have a go at
the challenging subject of "Single Sample Testing" (keep in mind that
I am not a data miner).
MCP is the most relevant maths method (to SST) that I have heard
about to date.
The starting line for my thinking there is:
a) if we data mine we have corrupted (or potentially corrupted) the
sample test therefore we always need OOS after datamining.
b) if we don't data mine (or think we don't) it is very difficult to
be absolutely sure we haven't corrupted the data test.
c) the only value in two sample tests is it returns an estimate of
variance
d) if we can be aboslutely certain that we haven't corrupted the
sample test then a binomial simulation equation could (potentially)
tell us everything we need to know about variance (for any metric).
e) It seems that the key to understanding SST is to understand how,
where and why we are corrupting the test (if we are in fact doing
that).
f)the exception could be that one sample reports on the efficiency of
our system in one type of market and the second sample test reports
on another type of market.
The challenge there is to design systems that are market indepenedent
and where the test is not corrupted by our methods.
Not so impossible afterall?
(re message extensions: apologies to Phssst for getting cranky with
you for 5 secs - no good ever comes of it - my answer is that I can't
work any other way - that is me - I think the overall benefits
outweigh the annoying habits but I will try to keep it down).
brian_z
--- In amibroker@xxxxxxxxxxxxxxx, "brian_z111" <brian_z111@xxx> wrote:
>
> > 2. After the system has been developed and tested, Monte Carlo
> >techniques
> > can be used to reorder (or re-sample from the estimated
> >distribution) the
> > Out-of-Sample results to estimate the statistical distribution of
> >the equity
> > curve and trading results.
>
> IMO that is a must do but others disagree on that point.
>
> Re MCP
>
> I have a high regard for Aronson so I haven't thrown the idea of
MCP
> away but I'm not excited enough to pay the price of learning the
math.
> Different story if I was an 'optimizer.
>
> I might study it one day, as an academic exercise, since I make an
> effort to learn evaluation well (don't know how anyone trades
without
> it but some do).
>
> thanks
>
> brian_z
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@> wrote:
> >
> > Hi Brian --
> >
> > Monte Carlo is a name given to a broad category of techniques
used
> in
> > modeling and simulation to evaluate the robustness of a model.
The
> name
> > means different things to different people. The techniques
> typically
> > involve adding some random component -- for example, adding
random
> noise or
> > reordering results in a random manner.
> >
> > The one use of Monte Carlo analysis that I am cautioning about is
> this:
> > 1. Make an in-sample run.
> > 2. Reorder the trades observed for the in-sample period.
> > 3. Attempt to estimate the robustness of the system.
> >
> > My point is that simply reordering the in-sample trades will not
> provide any
> > additional information about the likelihood that the system will
be
> > profitable when traded with real money.
> >
> > There are many ways of applying Monte Carlo that Will provide
> additional
> > information. Here are two examples:
> > 1. During any test runs, use Monte Carlo techniques to perturb
> either the
> > data or the values of the parameters.
> > A. The data can be perturbed by adding some amount of random
noise.
> > B. The parameter values can be perturbed by testing not only the
> specific
> > value requested, but also a cloud of values nearby.
> > If the results of applying the system to perturbed data result in
> values of
> > the objective function that are similar to those of the specific
> value
> > requested (before any random component is added), that Does
increase
> > confidence that the system is robust, and it May increase
> confidence that it
> > will perform well out-of-sample. An explicit out-of-sample test
is
> still
> > required.
> > 2. After the system has been developed and tested, Monte Carlo
> techniques
> > can be used to reorder (or re-sample from the estimated
> distribution) the
> > Out-of-Sample results to estimate the statistical distribution of
> the equity
> > curve and trading results. For example, given a set of OOS
> results, if the
> > trades happened in a different order, what is the expected, best,
> and worst
> > case drawdown?
> >
> > Thanks for listening,
> > Howard
> >
> >
> > On Sun, Mar 16, 2008 at 4:43 PM, brian_z111 <brian_z111@> wrote:
> >
> > > Howard,
> > >
> > >
> > > >(Running Monte Carlo permutations on in-sample results does
> Nothing
> > > >to improve the likelihood of out-of-sample profitability.)
> > >
> > > I haven't gone down the MCP path because the math required
would
> be a
> > > steep challenge for me and also my gut feeling for it is NO.
> > >
> > > (Since I am not a mathematician I often substitute conceptual
> > > analysis for ratiocination - conceptually it doesn't add up to a
> > > compelling argument IMO).
> > >
> > > Are you referring to MCP?
> > > Are you in the position to elaborate a little further?
> > >
> > > BTW - thanks for everything you are doing.
> > > I'm bouncing off you (and others) - its keeping me fired up.
> > >
> > > brian_z
> > >
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%
> 40yahoogroups.com>, "Howard B"
> > > <howardbandy@> wrote:
> > > >
> > > > Greetings all --
> > > >
> > > > I received the following email privately. I think the
> discussion is
> > > > important enough to post it to the group.
> > > >
> > > > //--------------------------------------
> > > >
> > > > Dear Howard,
> > > >
> > > > Thanks for your reply in the Amibroker forum. The topic of
how
> to
> > > > sort optimisation results is an important one.
> > > >
> > > > You mention several statistics:
> > > >
> > > > KRatio, RRR, UPI, CAR/MDD, RAR/MDD, Recovery Ratio.
> > > >
> > > > Do you have any experience in using trading systems optimised
> to one
> > > > of these?
> > > >
> > > > I'm finding that by optimising for highest account value, I go
> > > > through long periods of drawdown, then a small % of trades
make
> a
> > > > killing (eg: 46% wins, 2:1 Win to Loss ratio overall). This
is
> not
> > > > what I'm looking for.
> > > >
> > > > I'd like to try and smooth out the equity curve to give more
of
> a
> > > > balance between making a large capital gain and also regular
> > > > cashflow. I have the following stats:
> > > >
> > > > - % Wins
> > > > - Win:Loss Ratio (All profits/All losses)
> > > > - Risk:Reward
> > > >
> > > > I have found that by sorting results by Risk:Reward does not
> give
> > > > good overall account balances.
> > > >
> > > > //-----------------------------------------------
> > > >
> > > > The experience that writer has is exactly the point I am
making
> > > when I say
> > > > that net profit is usually a poor objective function to use
when
> > > developing
> > > > trading systems.
> > > >
> > > > Each of the metrics I mentioned have three positive
> characteristics:
> > > > 1. they reward equity growth
> > > > 2. they penalize drawdowns
> > > > 3. when used as The objective function for walk forward
testing,
> > > they tend
> > > > to select systems that perform well out-of-sample
> > > >
> > > > My feeling is that the choice of objective comes very early
on
> in
> > > the
> > > > design, test, and validation process. The objective function
> > > incorporates
> > > > the features that the trader wants in his or her trading, so
> > > systems that
> > > > rank well by using the score of that objective function are
> systems
> > > that the
> > > > person knows they will be comfortable with.
> > > >
> > > > If one of those metrics already built in to AmiBroker is not
> > > satisfactory,
> > > > it is easy to create your own metric and have AmiBroker
report
> it
> > > for every
> > > > run and use it in the walk forward testing. The custom metric
> can
> > > include
> > > > setting limits on percent winners, win to loss ratio, and so
> forth.
> > > >
> > > > There are several posts over the past few days that discuss
this
> > > and show
> > > > examples of how to do it.
> > > >
> > > > Try out whatever you think might work for you. Keep in mind
that
> > > optimizing
> > > > using an objective function that does not include some
penalty
> for
> > > drawdowns
> > > > and / or reward for equity smoothness is likely to result in
> > > systems that
> > > > have very strange results and do not perform well out-of-
sample.
> > > >
> > > > As always, be sure to evaluate the results by examining out-
of-
> > > sample
> > > > results -- in-sample results have no value in estimating the
> likely
> > > > out-of-sample performance. NO value. Trading based on in-
sample
> > > results
> > > > alone is a certain way to lose your trading account. (Running
> > > Monte Carlo
> > > > permutations on in-sample results does Nothing to improve the
> > > likelihood of
> > > > out-of-sample profitability.)
> > > >
> > > > Thanks for listening,
> > > > Howard
> > > > www.quantitativetradingsystems.com
> > > >
> > > >
> > > >
> > > > On Fri, Mar 14, 2008 at 8:00 AM, dralexchambers
> <dralexchambers@>
> > > > wrote:
> > > >
> > > > > Does anyone use other Optimisation statistics to rank
> > > optimisations
> > > > > by?
> > > > >
> > > > > For example, instead of using raw profit gained, does
anyone
> use
> > > > > Win:Loss Ratio or Risk:Reward ratio - and what have been
your
> > > > > experiences.
> > > > >
> > > > > Thanks,
> > > > > Alex
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> >
>
------------------------------------
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|