[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] Re: Your opinion - Ranking Optimisations



PureBytes Links

Trading Reference Links

Hi Brian --

Monte Carlo is a name given to a broad category of techniques used in modeling and simulation to evaluate the robustness of a model.  The name means different things to different people.  The techniques typically involve adding some random component -- for example, adding random noise or reordering results in a random manner. 

The one use of Monte Carlo analysis that I am cautioning about is this:
1.  Make an in-sample run.
2.  Reorder the trades observed for the in-sample period.
3.  Attempt to estimate the robustness of the system.

My point is that simply reordering the in-sample trades will not provide any additional information about the likelihood that the system will be profitable when traded with real money. 

There are many ways of applying Monte Carlo that Will provide additional information.  Here are two examples:
1.  During any test runs, use Monte Carlo techniques to perturb either the data or the values of the parameters. 
A.  The data can be perturbed by adding some amount of random noise. 
B.  The parameter values can be perturbed by testing not only the specific value requested, but also a cloud of values nearby. 
If the results of applying the system to perturbed data result in values of the objective function that are similar to those of the specific value requested (before any random component is added), that Does increase confidence that the system is robust, and it May increase confidence that it will perform well out-of-sample.  An explicit out-of-sample test is still required.
2.  After the system has been developed and tested, Monte Carlo techniques can be used to reorder (or re-sample from the estimated distribution) the Out-of-Sample results to estimate the statistical distribution of the equity curve and trading results.  For example, given a set of OOS results, if the trades happened in a different order, what is the expected, best, and worst case drawdown?

Thanks for listening,
Howard
 

On Sun, Mar 16, 2008 at 4:43 PM, brian_z111 <brian_z111@xxxxxxxxx> wrote:

Howard,



>(Running Monte Carlo permutations on in-sample results does Nothing
>to improve the likelihood of out-of-sample profitability.)

I haven't gone down the MCP path because the math required would be a
steep challenge for me and also my gut feeling for it is NO.

(Since I am not a mathematician I often substitute conceptual
analysis for ratiocination - conceptually it doesn't add up to a
compelling argument IMO).

Are you referring to MCP?
Are you in the position to elaborate a little further?

BTW - thanks for everything you are doing.
I'm bouncing off you (and others) - its keeping me fired up.

brian_z


--- In amibroker@xxxxxxxxxxxxxxx, "Howard B" <howardbandy@xxx> wrote:
>
> Greetings all --
>
> I received the following email privately. I think the discussion is
> important enough to post it to the group.
>
> //--------------------------------------
>
> Dear Howard,
>
> Thanks for your reply in the Amibroker forum. The topic of how to
> sort optimisation results is an important one.
>
> You mention several statistics:
>
> KRatio, RRR, UPI, CAR/MDD, RAR/MDD, Recovery Ratio.
>
> Do you have any experience in using trading systems optimised to one
> of these?
>
> I'm finding that by optimising for highest account value, I go
> through long periods of drawdown, then a small % of trades make a
> killing (eg: 46% wins, 2:1 Win to Loss ratio overall). This is not
> what I'm looking for.
>
> I'd like to try and smooth out the equity curve to give more of a
> balance between making a large capital gain and also regular
> cashflow. I have the following stats:
>
> - % Wins
> - Win:Loss Ratio (All profits/All losses)
> - Risk:Reward
>
> I have found that by sorting results by Risk:Reward does not give
> good overall account balances.
>
> //-----------------------------------------------
>
> The experience that writer has is exactly the point I am making
when I say
> that net profit is usually a poor objective function to use when
developing
> trading systems.
>
> Each of the metrics I mentioned have three positive characteristics:
> 1. they reward equity growth
> 2. they penalize drawdowns
> 3. when used as The objective function for walk forward testing,
they tend
> to select systems that perform well out-of-sample
>
> My feeling is that the choice of objective comes very early on in
the
> design, test, and validation process. The objective function
incorporates
> the features that the trader wants in his or her trading, so
systems that
> rank well by using the score of that objective function are systems
that the
> person knows they will be comfortable with.
>
> If one of those metrics already built in to AmiBroker is not
satisfactory,
> it is easy to create your own metric and have AmiBroker report it
for every
> run and use it in the walk forward testing. The custom metric can
include
> setting limits on percent winners, win to loss ratio, and so forth.
>
> There are several posts over the past few days that discuss this
and show
> examples of how to do it.
>
> Try out whatever you think might work for you. Keep in mind that
optimizing
> using an objective function that does not include some penalty for
drawdowns
> and / or reward for equity smoothness is likely to result in
systems that
> have very strange results and do not perform well out-of-sample.
>
> As always, be sure to evaluate the results by examining out-of-
sample
> results -- in-sample results have no value in estimating the likely
> out-of-sample performance. NO value. Trading based on in-sample
results
> alone is a certain way to lose your trading account. (Running
Monte Carlo
> permutations on in-sample results does Nothing to improve the
likelihood of
> out-of-sample profitability.)
>
> Thanks for listening,
> Howard
> www.quantitativetradingsystems.com
>
>
>
> On Fri, Mar 14, 2008 at 8:00 AM, dralexchambers <dralexchambers@xxx>

> wrote:
>
> > Does anyone use other Optimisation statistics to rank
optimisations
> > by?
> >
> > For example, instead of using raw profit gained, does anyone use
> > Win:Loss Ratio or Risk:Reward ratio - and what have been your
> > experiences.
> >
> > Thanks,
> > Alex
> >
> >
> >
>


__._,_.___

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html




Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___