[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[amibroker] Re: What is a valid number of Back test results to Optimize?



PureBytes Links

Trading Reference Links

Correction - sorry that should be

Plot sample error (on the Y axis) against the number (N) of data
POINTS IN THE SAMPLE (on the X axis) and it is an inverse exponential 
curve.

I don't want to confuse it with Standard Error which measures 
variance between several samples (for our purposes a sample is a 
backtest and the number of datapoints is the number of closed trades 
for the backtested period).

brian_z111



--- In amibroker@xxxxxxxxxxxxxxx, "brian_z111" <brian_z111@xxx> wrote:
>
> Chris,
> 
> No, we don't have to be Quants to trade.
> I actually only use basic stats in a simplistic way - it is how and 
> where you use it that counts (I did have to dig into a lot 
> of 'theory' to find that out and find what I wanted though).
> 
> All of the authors I mentioned, except for Kennedy & Bernstein, are 
> trading orientated so pick one or two that you like and dig in 
(that 
> should take you up the easy 80% of the learning curve).
> 
> Some further discussion: 
> 
> (I'll try to keep it short and give you some practical pointers - 
> which everyone knows is hard for me to do).
> 
> 1) SAMPLE ERROR
> 
> Sample error is very real.
> 
> I have plotted thousands of pseudo stock charts using the random 
> generator in Excel and watched sample error play out time and time 
> again.
> 
> Plot sample error (on the Y axis) against the number (N) of data 
> samples (on the X axis) and it is an inverse exponential curve.
> 
> Sample error% = 1/sqrt(N) * 100 
> 
> The reduction in error that we gain as we increase sample size 
> decreases exponentially.
> There is an optimal point somewhere around N = 1000 where we don't 
> get much improvement in accuracy with increased sample size.
> 
> Having seen sample error in thousands of labtests I find it very 
> difficult to accept any analysis that uses less than 300-400 
samples 
> (as a minimum). If I can get more I use them (in my newest system 
> that I am working on I am getting thousands of trade samples per 
> test).
> I go so far as to use data from other markets and timeframes to 
> produce my large datasets (plus other tricks as well).
> 
> 2) SAMPLE ERROR IN OPTIMIZATION
> 
> I am very sceptical about optimization that ignores sample size.
> When you have a nice round optimization 3D graph how meaningful 
could 
> it possibly be if it is produced from only 30 trade data points. 
The 
> sample size behind the plots on that graph are not evenly 
distributed 
> and possibly not statistically valid.
> 
> I haven't found any trading commentator who discusses this point or 
> gives examples of how to account for sample error when optimizing.
> I assume it could be included as part of the test criteria 
(objective 
> function)?
> 
> Perhaps people are doing things like that in private and just 
haven't 
> told anyone about it. 
> 
> 3) SYNCHRONIZING TO THE CURRENT MARKET
> 
> I am very sceptical about this (once again it places the trader in 
> the position of relying on very small datasets).
> 
> Returning to the analogy of vehicle performance tests.
> 
> Unlike F1 racing we don't know anything about the track or the 
> weather (they know everything about in advance and prepare their 
car 
> accordingly).
> 
> Since we can't predict what track we will be racing on or what 
> conditions we will be racing under why would we prepare as if we do?
> 
> I would rather use a system that performs reasonably well under all 
> conditions than try to tune-up to unpredictable and ever changing 
> conditions.
> 
> brian_z111
> 
> 
> --- In amibroker@xxxxxxxxxxxxxxx, ChrisB <kris45mar@> wrote:
> >
> > Thanks Brian, just the sort of feedback and references I was 
> looking for:
> > 
> > I guess you have put this into English better than I did:
> > 
> > 3) ongoing re-optimization of a system to 'synchronise' with 
current
> > (short term) 'market conditions'.
> > 
> > This is the crux of the question, as I have tried to verbalize in 
> my 
> > response to Howard above, but not as succinctly as you have.
> > 
> > This then is the philosophical question: is it better 
> to "synchronize" 
> > frequently and often and stay in tune with market movements, or 
> better 
> > to perform endless walkforwards? I for one wouldn't care less if 
a 
> > system stopped working as long as I could detect that early 
enough 
> and 
> > then just stop trading it. After all, as Howard says in his book 
> (and in 
> > earlier posts to this board), all systems stop working after a 
> while. I 
> > want to be trading this before it stops working and not later.  I 
> look 
> > forward to the rest of Howard's book (and then a number of re-
> reads) and 
> > may post some more questions here.
> > 
> > One tedious approach would be to place sample trades from a 
number 
> of 
> > systems on a demo account, collate the data, plot the expectancy 
> and 
> > other stats into Excel and then just trade whichever one is 
> currently 
> > performing best. My goal in understanding how the AB backtester 
> works 
> > and how to apply it practically to my chosen market, is to speed 
> this up 
> > and remove the tedium from this.
> > 
> > Thanks for the names of the other authors: will need to look 
these 
> up 
> > but hope that they are not too "quantsy" for my intellectual 
level. 
> As 
> > you say we should be able to trade without needing PhDs. (If I am 
> wrong 
> > here please correct me asap: at my age I may prefer to engage in 
> other 
> > activities in my declining years rather than pursuing a futile 
> exercise.)
> > 
> > Thanks for the UKB link and associated references. That amount of 
> > reading should keep me off the streets for awhile!
> > 
> > Regards
> > 
> > ChrisB
> > 
> > 
> > 
> > 
> > 
> > brian_z111 wrote:
> > >
> > > Hello Chris,
> > >
> > > Yes, I have some thoughts on the subject and a reference or two.
> > >
> > > In fact I am probably part of the way towards a decent book on 
the
> > > subject (presented from a traders point of view) although I 
doubt
> > > that I will ever write it.
> > >
> > > You actually have three topics there:
> > >
> > > 1) statistics (statistical significance) ,
> > > 2) statistical significance as it is applied to optimization
> > > (selecting the 'best performer' from a number of possible 
> parameters),
> > > 3) ongoing re-optimization of a system to 'synchronise' with 
> current
> > > (short term) 'market conditions'.
> > >
> > > For effective consideration it would be better to separate the 
> three
> > > subjects, as far as possible, although from a traders point of 
> view
> > > they all come under the one heading of "system design, testing 
and
> > > evaluation" )SDT&E.
> > >
> > > The best people for you to ask about that subject, in the 
context 
> of
> > > your question, would be those who are making a living using 
> systems
> > > that they designed, using optimization as a key part of the 
> process,
> > > or specialists in the field.
> > >
> > > I'm not in either of those categories.
> > >
> > > I do, however, know a few who are. For a start we have our very 
> own
> > > Fred and Howard, who fit into one, or both.
> > >
> > > Then, there is Pardo, who is the father of SDT&E as we know it
> > > (outside of academic circles)as well as Prof David Aronson, Lars
> > > Kestner, Stridsman, Taleb.
> > >
> > > None of them are bad reads; some are better than others.
> > >
> > > In fact Aronson is my No1 favorite trading book and Howards 
book 
> is
> > > in my top five. Aronson has a strong intellect and couples that 
> with
> > > the ability to make a difficult subject sound easy as well as 
> being
> > > readable and entertaining.
> > >
> > > IMO one of the handicaps we are under is that the subject 
evolved
> > > from, and has been dominated by, academia while what we 
actually 
> need
> > > is a pragmatic amnual of "applied stats for traders". Aronson's 
> book
> > > goes a fair way, albeit not all the way, towards achieving that.
> > >
> > > (As far as I am concerned, anyone who thoroughly understands 
> Aronsons
> > > work, and can apply it, knows 95%, or more, of everything they 
> need
> > > to know on the subject).
> > >
> > > Howards book is amongst my top 5 because it is training 
orientated
> > > and full of examples written in my one and only programming 
> langauge
> > > (AFL). It complements Aronson nicely.
> > >
> > > Pardo is a little old and IMO doesn't go far enough, although 
to 
> be
> > > fair I feel he did intend it to be an introductory text.
> > >
> > > Stridsman isn't too bad at all.
> > >
> > > Klestner disappointed a little although his claim to fame is 
the K
> > > Ratio.
> > >
> > > Taleb I found neither entertaining nor informative although 
others
> > > seem to recommend it.
> > >
> > > There's a rudimentary post on statistics at the UKB - I will 
file
> > > details of the authors I have mentioned above to that post in 
the
> > > next day or two. It's easier for me to put them there which 
saves 
> me
> > > having to repeat this post in the future. There are a few bits 
and
> > > pieces there - nothing much really - I hope to add to it one 
day.
> > >
> > > If you search this forum under WhitneyBroach OR Whitney 
(author) 
> AND
> > > Aronson (contents) you will find some discussion (Whitney first
> > > posted on Aronson).
> > >
> > > I might post some more discussion later but in the meantime 
here 
> is
> > > something to think about:
> > >
> > > Your doctor informs you that you have an incurable, painless, 
> disease
> > > and that you only have two weeks to live (there's absolutely no
> > > chance of a cure and you will die peacefully in your sleep).
> > >
> > > As a last resort he offers you an experimental pill that has 
been
> > > designed to extend our longevity.
> > >
> > > The pill has been trialled on 100 people.
> > > 53% were totally cured and lived healthily and happily until 
they
> > > were 120 (they died peacefully in their sleep too).
> > >
> > > The other 47% also lived until they were 120- but they suffered
> > > horrendous side effects and lived out their lifes in 
unimagineable
> > > pain (the pill over-rides any attempts at euthenasia).
> > >
> > > Will you take the pill or not?
> > >
> > > For you statistics is no longer a theoretical subject.
> > > Your decision will have very real, life changing consequences.
> > >
> > > Now it is a fact that the majority of people will choose to 
take 
> the
> > > pill; it's in our makeup to do so (the pain is in the future and
> > > not 'real' at the time of making the decision, we emphasis the
> > > positive and eliminate the negative, 53/47 sounds like good 
odds 
> to
> > > most of us etc).
> > >
> > > But, if we step back a pace and evaluate the problem 
objectively,
> > > rationally, logically (would one of those have been enough?) 
what
> > > decision would we make then?
> > >
> > > How about if we are a trained statistician, or maybe a 
> professional
> > > trader?
> > >
> > > How about if the Doctor said' "You don't have to decide now. 
There
> > > will be a new trial, using 10,000 people, starting tomorrow and
> > > finishing in three days; the results will be known immediately. 
> You
> > > can wait until then to decide if you like, but there is only a
> > > limited number of pills available and there is no guarantee 
that 
> you
> > > will still receive one in four days time"?
> > >
> > > What is your answer now?
> > >
> > > IMO it's a binomial problem.
> > >
> > > http://www.amibroke r.org/userkb/ ?s=Statistics 
> > > <http://www.amibroker.org/userkb/?s=Statistics>
> > >
> > > brian_z
> > >
> > > --- In amibroker@xxxxxxxxx ps.com 
> > > <mailto:amibroker%40yahoogroups.com>, ChrisB <kris45mar@ ..> 
> wrote:
> > > >
> > > > What is a valid or reasonable number of backtest results to 
> subject
> > > to
> > > > Optimization?
> > > >
> > > > For general statistics a minimum of 30 or so is needed to 
start
> > > getting
> > > > valid StdDevs etc.
> > > >
> > > > If I run a backtest on hourly currency data over three months 
I 
> get
> > > > around 16 -20 tradeable signals per currency.
> > > > This give a nice smooth plateau on 3D optimization.
> > > >
> > > > If I test over two months of data I get around 10 - 12 trades
> > > >
> > > > If I test over only 1 month I get only 5 or 6 trades.
> > > >
> > > > These shorter time periods still give visually acceptable 3D
> > > plateaus
> > > > but I am wondering if there is enough data to be statistically
> > > significant.
> > > >
> > > > I am trying to get a handle on how close I can get to current
> > > > fluctuations in the market without hitting noise. The idea 
> being to
> > > redo
> > > > the Optimization every x time frame and shift the entry and 
exit
> > > > parameters to stay in the middle of the plateau.
> > > >
> > > > Of course I can backtest over longer time frames, say 6 
months 
> of
> > > data,
> > > > shifting the starting date forward by one month at a time, 
but 
> this
> > > > would seem to introduce more "lag" into my selection of best
> > > parameters
> > > > to trade.
> > > >
> > > > Does anyone have any thoughts/references on this?
> > > >
> > > > --
> > > > Regards
> > > >
> > > > ChrisB
> > > >
> > >
> > >  
> > 
> > 
> > -- 
> > Regards
> > 
> > ChrisB
> >
>




Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to 
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html
 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:amibroker-digest@xxxxxxxxxxxxxxx 
    mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/