[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[amibroker] Re: Detecting data mining bias with modified Monte Carlo procedure



PureBytes Links

Trading Reference Links

Song, 

Thankyou.
A little piece of the puzzle I was looking for.
I don't think it is from this archive so I probably wouldn't have 
found it by myself.

For my own use I would convert to % profit or loss, which would also 
mean changing the range
I prefer to use standardised units.

BrianB2.

--- In amibroker@xxxxxxxxxxxxxxx, "drsquar" <drsong123@xxx> wrote:
>
> Hi,
> I found following post regarding MCS for amibroker during my 
googling:
> 
> In "Casino Trading," Phil Abel presents methods of evaluating 
trading 
> system robustness using Monte Carlo simulation. To perform a Monte 
> Carlo simulation, we would need a "decision tree" -- a set of data 
> showing probabilities of profits in a given range. 
> 
> As the author suggests, we could derive such a decision tree from 
an 
> individual trade list produced by the backtester, but we will show 
a 
> slightly more automatic method that involves using AmiBroker's 
> exploration mode. 
> 
> Listing 1 shows the exploration formula to be applied in the 
> Automatic Analysis window. In our example, we have used a very 
simple 
> EMA crossover system, but the user can replace it with any other 
> system. The code produces a table that shows the number of 
> winning/losing trades, total number of trades, and percentage 
> probabilities of profits in user-defined ranges. In the sample code 
> below, we have used 25 ranges for winning trades (0..100, 
> 100..200, .., 2400...infinity), and a similar 25 ranges for losing 
> trades. The user can modify both the number of ranges evaluated and 
> the size of the range by modifying appropriate variables in the 
code. 
> Once the exploration is complete, the results can be exported to a 
> CSV file for further analysis in Excel. 
> 
> LISTING 1
> ///// Trading system to be evaluated
> Buy = Cross( Close, EMA(Close, 15) );
> Sell = Cross( EMA(Close, 15), Close );
> ///////////////////////////////////
> e = Equity(1);
> tradeend = Sell;
> profit = e - ValueWhen( Buy, e );
> endprofit = IIf( tradeend , profit, 0 );
> LosingTrades = LastValue( Cum( endprofit < 0 ) );
> WiningTrades = LastValue( Cum( endprofit > 0 ) );
> TotalTrades = LastValue( Cum( tradeend ) );
> AddColumn( LosingTrades, "Losing trades", 1 );
> AddColumn( WiningTrades, "Wining trades", 1 );
> AddColumn( TotalTrades, "Total trades", 1 );
> Filter = Status("lastbarinrange");
> // now we will count the number of
> // count trades winning in given $ ranges
> NumRanges = 25; // number of ranges to generate
> RangeSize = 100; // controls $ size of the range
> for( i = 0; i < NumRanges; i++ )
> {
>   rangestart = i * RangeSize;
>   rangeend = (i+1)*RangeSize;
>   if( i == NumRanges - 1 ) rangeend = 999999;
>   AddColumn(
>   LastValue( 100* Cum( endprofit > rangestart AND endprofit < 
> rangeend ) /TotalTrades ),
>   "% with profit " + rangestart + ".." + rangeend );
> }
> for( i = 0; i < NumRanges; i++ )
> {
>   rangestart = (-i - 1 ) * RangeSize;
>   rangeend = -i*RangeSize;
>   if( i == NumRanges - 1 ) rangestart = -999999;
>   AddColumn(
>   LastValue( 100* Cum( endprofit > rangestart AND endprofit < 
> rangeend ) /TotalTrades ),
>   " % with loss " + rangeend + ".." + rangestart );
> }
> 
>  A sample Monte Carlo spreadsheet for AmiBroker can be found in the 
> files area of AmiBroker's mailing list at 
> http://groups.yahoo.com/group/amibroker/files/. 
> 
> --Tomasz Janeczko, AmiBroker.com 
> www.amibroker.com
> 
> Song
> 
> --- In amibroker@xxxxxxxxxxxxxxx, "brian.z123" <brian.z123@> wrote:
> >
> > Hello Whitne,
> > 
> > Thanks for your post.
> > Two of your leads provided new material for me.
> > 
> > I am not using MCP but I have it under consideration.
> > Conceptually, if not practically, I have rejected the walk 
forward 
> > method, which means I am bound to consider all methods that 
> validate 
> > single sample tests.
> > 
> > So far I am considering three approaches and measurements for 
each: 
> > error, significance and symptoms of overfitting.
> > Obviously something from amongst those groups will have to do 
> better 
> > than the walk forward hypothesis for me to accept it.
> > 
> > It might end up that none of them will provide a conclusive 
answer 
> > but *in the land of the blind, the one eyed person is King*.
> > 
> > At the least study, in this area will improve my depth of 
knowledge 
> > of evaluation techniques.
> > 
> > BrianB2.
> > 
> > --- In amibroker@xxxxxxxxxxxxxxx, "whitneybroach" 
<WhitneyBroach@> 
> > wrote:
> > >
> > > While reading David Aronson's book _Evidence-based Technical
> > > Analysis_, I stumbled across a modified Monte Carlo permutation
> > > (MCP) procedure that compensates for data mining bias, assuming 
> that
> > > the "best" permutation of rules was not selected with a 
directed 
> > search.
> > > 
> > > From Aronson's perspective, this is good news.  He views data 
> mining
> > > as a useful procedure in the discovery phase of research.  
Plus, 
> MCP
> > > does not require out-of-sample data.  Thus it is possible to 
use 
> > more
> > > data for mining and still minimize data mining bias in test 
> > results. 
> > > The likely result:  fewer false positives for systems that are
> > > worthless, and fewer false negatives for systems that are 
> valuable.
> > > 
> > > The paper with discussion and C# code is here:
> > > <http://www.evidencebasedta.com/MonteDoc12.15.06.pdf>.
> > > 
> > > Aronson's book site, including a link to Amazon, is:
> > > <http://www.evidencebasedta.com>.  Separately, I'm looking 
> forward 
> > to
> > > the imminent books from Howard
> > > <http://www.quantitativetradingsystems.com/> and Ralph Vince
> > > <http://tinyurl.com/2os2p7>.
> > > 
> > > Not being a user of IO (or other AB add-ons), I have no idea if 
> this
> > > MCP approach is already being used in the AB community.  It 
looks
> > > interesting to me.  MCP appears to require market data and 
trade 
> > data
> > > from every run, not simply the trade data.  That suggests to me 
> that
> > > an AB add-on, rather than a completely external program, would 
be 
> a
> > > more straightforward implementation.
> > > 
> > > Aronson also refers to a patented boostrap procedure that 
> > accomplishes
> > > much the same thing, White's Reality Check, named for Halbert 
> White,
> > > the patent holder.  Apparently WRC is not available 
commercially.
> > > 
> > > Best,
> > >
> >
>




------------------------ Yahoo! Groups Sponsor --------------------~--> 
See what's inside the new Yahoo! Groups email.
http://us.click.yahoo.com/0It09A/bOaOAA/yQLSAA/GHeqlB/TM
--------------------------------------------------------------------~-> 

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to 
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html
 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:amibroker-digest@xxxxxxxxxxxxxxx 
    mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/