[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[amibroker] Re: Walk-Forward Out of Sample (OOS) Testing



PureBytes Links

Trading Reference Links

Mark,

It was not my intention to be a smartass although I can see that's 
how your taking my response.  If you and possibly others are 
interested in this sort of AFL, I'd be happy to look into writing one 
as there may be benefits that I get out of this as well.  I would 
certainly think that it would be possible to implement whether the 
period of optimization had a sliding window or an anchored beginning 
based on a parameter.  If you'd like to proceed please state 
precisely what your requirements are.  After that I will probably 
have questions. 


--- In amibroker@xxxxxxxxxxxxxxx, "MarkF2" <feierstein@xxxx> wrote:
> I was looking for help in implementing what I described, not a
> critique  (there IS a benefit in "anchoring" the front data point if
> you consider the results accordingly, BTW) and not a general 
statement
> that it can be done "with a properly written AFL."  Thanks anyway.
> 
> --- In amibroker@xxxxxxxxxxxxxxx, "Fred" <fctonetti@xxxx> wrote:
> > I can't say as I see any benefit in anchoring the front data 
point as 
> > this causes current data to be less and less important as time 
goes 
> > along.  In addition I don't see why with a properly written AFL 
there 
> > would be any need for human intervention.  Basically you supply 
the 
> > system and it does the rest much like the way PortfolioTrader 
that I 
> > posted uses a user supplied scoring routine to determine which 
> > securities to invest this would need to decide what parameter 
values 
> > to use as time rolls along.
> > 
> > --- In amibroker@xxxxxxxxxxxxxxx, "MarkF2" <feierstein@xxxx> 
wrote:
> > > But how about doing it the way I described?  I would prefer to
> > > minimize human interaction and maximize automation.  Imagine 
> > testing a
> > > system in this manner on all SP500 stocks, for example.
> > > 
> > > --- In amibroker@xxxxxxxxxxxxxxx, "Fred" <fctonetti@xxxx> wrote:
> > > > This would be simple enough to do if you think about it terms 
of 
> > how 
> > > > many bars you are going to use for a lookback period and then 
> > armed 
> > > > with the results of optimization of that period how far ahead 
> > from 
> > > > that point in time you are going to trade before you 
reoptimize.  
> > As 
> > > > Dingo says this could be done with automation or it could all 
be 
> > done 
> > > > interally in AB/AFL/ABTool.  This is a different form of 
Score 
> > and 
> > > > Rank if you will where instead of scoring and ranking 
individual 
> > > > issues in a portfolio on a rolling basis you are scoring and 
> > ranking 
> > > > parameters for your timing system on a rolling basis.  
> > > > 
> > > > --- In amibroker@xxxxxxxxxxxxxxx, "dingo" <dingo@xxxx> wrote:
> > > > > perhaps I can help - contact me privately at dingo at 
udsnet 
> > dot 
> > > > com.
> > > > >  
> > > > > d
> > > > > 
> > > > > -----Original Message-----
> > > > > From: MarkF2 [mailto:feierstein@x...] 
> > > > > Sent: Tuesday, June 17, 2003 4:00 PM
> > > > > To: amibroker@xxxxxxxxxxxxxxx
> > > > > Subject: [amibroker] Walk-Forward Out of Sample (OOS) 
Testing
> > > > > 
> > > > > 
> > > > > Jayson, I believe Owen wants to use AB for walk-forward, 
out of 
> > > > sample
> > > > > testing.  I'd love to be able to do this also but lack the 
> > > > programming
> > > > > skills.  Can anyone help? I think this would be incredibly 
> > useful!!!
> > > > > 
> > > > > What I'd like to do can, I believe, be done with loops and 
> > would go
> > > > > something like this:
> > > > > 
> > > > > Let's say you're testing a simple system on MSFT where 
price 
> > crosses
> > > > > over and under a moving average so you're optimizing only 
one
> > > > > parameter, length, to keep things simple. The loop code 
would 
> > allow
> > > > > two inputs: 1) # of segments to break data into and 2) # of 
> > segments
> > > > > to use for initial optimization.  Let's say you choose 10 
and 
> > 3.  AB
> > > > > would divide MSFT data into 10 equal segments, then do the 
> > initial
> > > > > optimization on the first 3.  It would then take the best 
> > performing
> > > > > length and test it on segment #4 (which is the first OOS 
> > segment). 
> > > > > Then it would record two things for this and every other 
OOS 
> > > > segment:
> > > > > 1) the length used and 2)the result (I'd like profit 
factor).  
> > Then 
> > > > AB
> > > > > would reoptimize the system over segments 1-4 and use the 
best
> > > > > parameter on segment 5 (the second OOS segment), recording 
> > length 
> > > > used
> > > > > and result, then reoptimize over segments 1-5, use the best 
> > > > parameter
> > > > > on segment 6, etc.  
> > > > > 
> > > > > What this would do is automatically tell you how robust 
your 
> > system
> > > > > concept is over the OOS segments (4-10 in this case).
> > > > > 
> > > > > Mark 
> > > > > 
> > > > > 
> > > > > --- In amibroker@xxxxxxxxxxxxxxx, "Jayson" <jcasavant@xxxx> 
> > wrote:
> > > > > > Owen,
> > > > > > Am I over simplifying by suggesting you simply use the
> > > > > > From-To range settings in AA?
> > > > > > 
> > > > > > Example choose 12/1/2002-12/25/2002 to optimize then 
simply 
> > move 
> > > > the
> > > > > window
> > > > > > forward 25 days to test the results.......
> > > > > > 
> > > > > > Regards,
> > > > > > Jayson
> > > > > > -----Original Message-----
> > > > > > From: Owen Davies [mailto:owen5819@x...]
> > > > > > Sent: Tuesday, June 17, 2003 2:04 PM
> > > > > > To: amibroker@xxxxxxxxxxxxxxx
> > > > > > Subject: [amibroker] Annoying programming problem: ideas 
> > please?
> > > > > > 
> > > > > > 
> > > > > > Active Trader has taken to doing something I've always 
> > considered
> > > > > BS.  Some
> > > > > > articles--at least one a month of late--optimize an 
intraday 
> > > > trading
> > > > > system,
> > > > > > usually some variation on a volatility breakout, over a 
month 
> > or 
> > > > so
> > > > > of data
> > > > > > and then do an out-of-sample test for the next month or 
so to
> > > > > "prove" how
> > > > > > well it works.  Over time, this has come to irritate me 
> > enough 
> > > > that
> > > > > I'd like
> > > > > > to do a long-term study, either to prove that it's 
nonsense 
> > or to
> > > > > learn
> > > > > > something new and highly unlikely.  So:
> > > > > > 
> > > > > > Can anyone think of a way to optimize a technique on, 
say, 25 
> > bars
> > > > > of data,
> > > > > > test it on the next 25 bars, and then step the window 
forward 
> > and
> > > > > do it
> > > > > > again?  I'd settle for re-optimizing daily on the 
previous 25 
> > (or
> > > > > whatever)
> > > > > > bars.  Whatever is easiest.
> > > > > > 
> > > > > > For the sake of simplicity, something that runs on EOD 
data 
> > will 
> > > > do.
> > > > > I mean
> > > > > > to test breakouts in the direction of an existing trend 
and 
> > close 
> > > > at
> > > > > the end
> > > > > > of the day, so there should be no problem with days that 
> > break out
> > > > > in one
> > > > > > direction, reverse, and break out in the other; a day 
that 
> > > > reverses
> > > > > and ends
> > > > > > badly will just count as a loss.  Also, I can do the 
> > conversion to
> > > > > intraday
> > > > > > data myself, rather than ask others to hand me the 
complete 
> > > > package.
> > > > > All I
> > > > > > need is some way to optimize on a window.
> > > > > > 
> > > > > > Offhand, I can't see any way to do it within AFL, and I 
don't 
> > have
> > > > > the skill
> > > > > > to handle it with external programming.
> > > > > > 
> > > > > > Many thanks.
> > > > > > 
> > > > > > Owen Davies
> > > > > > 
> > > > > > 
> > > > > >       Yahoo! Groups Sponsor
> > > > > >             ADVERTISEMENT
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Send BUG REPORTS to bugs@xxxx
> > > > > > Send SUGGESTIONS to suggest@xxxx
> > > > > > -----------------------------------------
> > > > > > Post AmiQuote-related messages ONLY to: 
> > amiquote@xxxxxxxxxxxxxxx
> > > > > > (Web page: 
http://groups.yahoo.com/group/amiquote/messages/)
> > > > > > --------------------------------------------
> > > > > > Check group FAQ at:
> > > > > > 
http://groups.yahoo.com/group/amibroker/files/groupfaq.html
> > > > > > 
> > > > > > Your use of Yahoo! Groups is subject to the Yahoo! Terms 
of 
> > > > Service.
> > > > > 
> > > > > 
> > > > > 
> > > > > Yahoo! Groups Sponsor	
> > > > >  
> > > > > 
> > > > 
> > 
<http://rd.yahoo.com/M=251812.3170658.4537139.1261774/D=egroupweb/S=17
> > > > 05
> > > > > 
> > > > 
> > 
632198:HM/A=1564416/R=0/SIG=11ti81skc/*http://www.netflix.com/Default?
> > > > mq
> > > > > so=60164797&partid=3170658> 	
> > > > >  
> > > > > <http://us.adserver.yahoo.com/l?
> > > > M=251812.3170658.4537139.1261774/D=egrou
> > > > > pmail/S=:HM/A=1564416/rand=707349073> 	
> > > > > 
> > > > > Send BUG REPORTS to bugs@xxxx
> > > > > Send SUGGESTIONS to suggest@xxxx
> > > > > -----------------------------------------
> > > > > Post AmiQuote-related messages ONLY to: 
> > amiquote@xxxxxxxxxxxxxxx 
> > > > > (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> > > > > --------------------------------------------
> > > > > Check group FAQ at:
> > > > > http://groups.yahoo.com/group/amibroker/files/groupfaq.html 
> > > > > 
> > > > > Your use of Yahoo! Groups is subject to the Yahoo! Terms of 
> > Service
> > > > > <http://docs.yahoo.com/info/terms/> .


------------------------ Yahoo! Groups Sponsor ---------------------~-->
Get A Free Psychic Reading! Your Online Answer To Life's Important Questions.
http://us.click.yahoo.com/Lj3uPC/Me7FAA/ySSFAA/GHeqlB/TM
---------------------------------------------------------------------~->

Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx 
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/