[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[amibroker] Re: Walk-Forward Out of Sample (OOS) Testing



PureBytes Links

Trading Reference Links

I was looking for help in implementing what I described, not a
critique  (there IS a benefit in "anchoring" the front data point if
you consider the results accordingly, BTW) and not a general statement
that it can be done "with a properly written AFL."  Thanks anyway.

--- In amibroker@xxxxxxxxxxxxxxx, "Fred" <fctonetti@xxxx> wrote:
> I can't say as I see any benefit in anchoring the front data point as 
> this causes current data to be less and less important as time goes 
> along.  In addition I don't see why with a properly written AFL there 
> would be any need for human intervention.  Basically you supply the 
> system and it does the rest much like the way PortfolioTrader that I 
> posted uses a user supplied scoring routine to determine which 
> securities to invest this would need to decide what parameter values 
> to use as time rolls along.
> 
> --- In amibroker@xxxxxxxxxxxxxxx, "MarkF2" <feierstein@xxxx> wrote:
> > But how about doing it the way I described?  I would prefer to
> > minimize human interaction and maximize automation.  Imagine 
> testing a
> > system in this manner on all SP500 stocks, for example.
> > 
> > --- In amibroker@xxxxxxxxxxxxxxx, "Fred" <fctonetti@xxxx> wrote:
> > > This would be simple enough to do if you think about it terms of 
> how 
> > > many bars you are going to use for a lookback period and then 
> armed 
> > > with the results of optimization of that period how far ahead 
> from 
> > > that point in time you are going to trade before you reoptimize.  
> As 
> > > Dingo says this could be done with automation or it could all be 
> done 
> > > interally in AB/AFL/ABTool.  This is a different form of Score 
> and 
> > > Rank if you will where instead of scoring and ranking individual 
> > > issues in a portfolio on a rolling basis you are scoring and 
> ranking 
> > > parameters for your timing system on a rolling basis.  
> > > 
> > > --- In amibroker@xxxxxxxxxxxxxxx, "dingo" <dingo@xxxx> wrote:
> > > > perhaps I can help - contact me privately at dingo at udsnet 
> dot 
> > > com.
> > > >  
> > > > d
> > > > 
> > > > -----Original Message-----
> > > > From: MarkF2 [mailto:feierstein@x...] 
> > > > Sent: Tuesday, June 17, 2003 4:00 PM
> > > > To: amibroker@xxxxxxxxxxxxxxx
> > > > Subject: [amibroker] Walk-Forward Out of Sample (OOS) Testing
> > > > 
> > > > 
> > > > Jayson, I believe Owen wants to use AB for walk-forward, out of 
> > > sample
> > > > testing.  I'd love to be able to do this also but lack the 
> > > programming
> > > > skills.  Can anyone help? I think this would be incredibly 
> useful!!!
> > > > 
> > > > What I'd like to do can, I believe, be done with loops and 
> would go
> > > > something like this:
> > > > 
> > > > Let's say you're testing a simple system on MSFT where price 
> crosses
> > > > over and under a moving average so you're optimizing only one
> > > > parameter, length, to keep things simple. The loop code would 
> allow
> > > > two inputs: 1) # of segments to break data into and 2) # of 
> segments
> > > > to use for initial optimization.  Let's say you choose 10 and 
> 3.  AB
> > > > would divide MSFT data into 10 equal segments, then do the 
> initial
> > > > optimization on the first 3.  It would then take the best 
> performing
> > > > length and test it on segment #4 (which is the first OOS 
> segment). 
> > > > Then it would record two things for this and every other OOS 
> > > segment:
> > > > 1) the length used and 2)the result (I'd like profit factor).  
> Then 
> > > AB
> > > > would reoptimize the system over segments 1-4 and use the best
> > > > parameter on segment 5 (the second OOS segment), recording 
> length 
> > > used
> > > > and result, then reoptimize over segments 1-5, use the best 
> > > parameter
> > > > on segment 6, etc.  
> > > > 
> > > > What this would do is automatically tell you how robust your 
> system
> > > > concept is over the OOS segments (4-10 in this case).
> > > > 
> > > > Mark 
> > > > 
> > > > 
> > > > --- In amibroker@xxxxxxxxxxxxxxx, "Jayson" <jcasavant@xxxx> 
> wrote:
> > > > > Owen,
> > > > > Am I over simplifying by suggesting you simply use the
> > > > > From-To range settings in AA?
> > > > > 
> > > > > Example choose 12/1/2002-12/25/2002 to optimize then simply 
> move 
> > > the
> > > > window
> > > > > forward 25 days to test the results.......
> > > > > 
> > > > > Regards,
> > > > > Jayson
> > > > > -----Original Message-----
> > > > > From: Owen Davies [mailto:owen5819@x...]
> > > > > Sent: Tuesday, June 17, 2003 2:04 PM
> > > > > To: amibroker@xxxxxxxxxxxxxxx
> > > > > Subject: [amibroker] Annoying programming problem: ideas 
> please?
> > > > > 
> > > > > 
> > > > > Active Trader has taken to doing something I've always 
> considered
> > > > BS.  Some
> > > > > articles--at least one a month of late--optimize an intraday 
> > > trading
> > > > system,
> > > > > usually some variation on a volatility breakout, over a month 
> or 
> > > so
> > > > of data
> > > > > and then do an out-of-sample test for the next month or so to
> > > > "prove" how
> > > > > well it works.  Over time, this has come to irritate me 
> enough 
> > > that
> > > > I'd like
> > > > > to do a long-term study, either to prove that it's nonsense 
> or to
> > > > learn
> > > > > something new and highly unlikely.  So:
> > > > > 
> > > > > Can anyone think of a way to optimize a technique on, say, 25 
> bars
> > > > of data,
> > > > > test it on the next 25 bars, and then step the window forward 
> and
> > > > do it
> > > > > again?  I'd settle for re-optimizing daily on the previous 25 
> (or
> > > > whatever)
> > > > > bars.  Whatever is easiest.
> > > > > 
> > > > > For the sake of simplicity, something that runs on EOD data 
> will 
> > > do.
> > > > I mean
> > > > > to test breakouts in the direction of an existing trend and 
> close 
> > > at
> > > > the end
> > > > > of the day, so there should be no problem with days that 
> break out
> > > > in one
> > > > > direction, reverse, and break out in the other; a day that 
> > > reverses
> > > > and ends
> > > > > badly will just count as a loss.  Also, I can do the 
> conversion to
> > > > intraday
> > > > > data myself, rather than ask others to hand me the complete 
> > > package.
> > > > All I
> > > > > need is some way to optimize on a window.
> > > > > 
> > > > > Offhand, I can't see any way to do it within AFL, and I don't 
> have
> > > > the skill
> > > > > to handle it with external programming.
> > > > > 
> > > > > Many thanks.
> > > > > 
> > > > > Owen Davies
> > > > > 
> > > > > 
> > > > >       Yahoo! Groups Sponsor
> > > > >             ADVERTISEMENT
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > Send BUG REPORTS to bugs@xxxx
> > > > > Send SUGGESTIONS to suggest@xxxx
> > > > > -----------------------------------------
> > > > > Post AmiQuote-related messages ONLY to: 
> amiquote@xxxxxxxxxxxxxxx
> > > > > (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> > > > > --------------------------------------------
> > > > > Check group FAQ at:
> > > > > http://groups.yahoo.com/group/amibroker/files/groupfaq.html
> > > > > 
> > > > > Your use of Yahoo! Groups is subject to the Yahoo! Terms of 
> > > Service.
> > > > 
> > > > 
> > > > 
> > > > Yahoo! Groups Sponsor	
> > > >  
> > > > 
> > > 
> <http://rd.yahoo.com/M=251812.3170658.4537139.1261774/D=egroupweb/S=17
> > > 05
> > > > 
> > > 
> 632198:HM/A=1564416/R=0/SIG=11ti81skc/*http://www.netflix.com/Default?
> > > mq
> > > > so=60164797&partid=3170658> 	
> > > >  
> > > > <http://us.adserver.yahoo.com/l?
> > > M=251812.3170658.4537139.1261774/D=egrou
> > > > pmail/S=:HM/A=1564416/rand=707349073> 	
> > > > 
> > > > Send BUG REPORTS to bugs@xxxx
> > > > Send SUGGESTIONS to suggest@xxxx
> > > > -----------------------------------------
> > > > Post AmiQuote-related messages ONLY to: 
> amiquote@xxxxxxxxxxxxxxx 
> > > > (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> > > > --------------------------------------------
> > > > Check group FAQ at:
> > > > http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
> > > > 
> > > > Your use of Yahoo! Groups is subject to the Yahoo! Terms of 
> Service
> > > > <http://docs.yahoo.com/info/terms/> .


------------------------ Yahoo! Groups Sponsor ---------------------~-->
Get A Free Psychic Reading! Your Online Answer To Life's Important Questions.
http://us.click.yahoo.com/Lj3uPC/Me7FAA/ySSFAA/GHeqlB/TM
---------------------------------------------------------------------~->

Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx 
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/