[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[amibroker] Re: What is a valid number of Back test results to Optimize?



PureBytes Links

Trading Reference Links

Chris,

No, we don't have to be Quants to trade.
I actually only use basic stats in a simplistic way - it is how and 
where you use it that counts (I did have to dig into a lot 
of 'theory' to find that out and find what I wanted though).

All of the authors I mentioned, except for Kennedy & Bernstein, are 
trading orientated so pick one or two that you like and dig in (that 
should take you up the easy 80% of the learning curve).

Some further discussion: 

(I'll try to keep it short and give you some practical pointers - 
which everyone knows is hard for me to do).

1) SAMPLE ERROR

Sample error is very real.

I have plotted thousands of pseudo stock charts using the random 
generator in Excel and watched sample error play out time and time 
again.

Plot sample error (on the Y axis) against the number (N) of data 
samples (on the X axis) and it is an inverse exponential curve.

Sample error% = 1/sqrt(N) * 100 

The reduction in error that we gain as we increase sample size 
decreases exponentially.
There is an optimal point somewhere around N = 1000 where we don't 
get much improvement in accuracy with increased sample size.

Having seen sample error in thousands of labtests I find it very 
difficult to accept any analysis that uses less than 300-400 samples 
(as a minimum). If I can get more I use them (in my newest system 
that I am working on I am getting thousands of trade samples per 
test).
I go so far as to use data from other markets and timeframes to 
produce my large datasets (plus other tricks as well).

2) SAMPLE ERROR IN OPTIMIZATION

I am very sceptical about optimization that ignores sample size.
When you have a nice round optimization 3D graph how meaningful could 
it possibly be if it is produced from only 30 trade data points. The 
sample size behind the plots on that graph are not evenly distributed 
and possibly not statistically valid.

I haven't found any trading commentator who discusses this point or 
gives examples of how to account for sample error when optimizing.
I assume it could be included as part of the test criteria (objective 
function)?

Perhaps people are doing things like that in private and just haven't 
told anyone about it. 

3) SYNCHRONIZING TO THE CURRENT MARKET

I am very sceptical about this (once again it places the trader in 
the position of relying on very small datasets).

Returning to the analogy of vehicle performance tests.

Unlike F1 racing we don't know anything about the track or the 
weather (they know everything about in advance and prepare their car 
accordingly).

Since we can't predict what track we will be racing on or what 
conditions we will be racing under why would we prepare as if we do?

I would rather use a system that performs reasonably well under all 
conditions than try to tune-up to unpredictable and ever changing 
conditions.

brian_z111


--- In amibroker@xxxxxxxxxxxxxxx, ChrisB <kris45mar@xxx> wrote:
>
> Thanks Brian, just the sort of feedback and references I was 
looking for:
> 
> I guess you have put this into English better than I did:
> 
> 3) ongoing re-optimization of a system to 'synchronise' with current
> (short term) 'market conditions'.
> 
> This is the crux of the question, as I have tried to verbalize in 
my 
> response to Howard above, but not as succinctly as you have.
> 
> This then is the philosophical question: is it better 
to "synchronize" 
> frequently and often and stay in tune with market movements, or 
better 
> to perform endless walkforwards? I for one wouldn't care less if a 
> system stopped working as long as I could detect that early enough 
and 
> then just stop trading it. After all, as Howard says in his book 
(and in 
> earlier posts to this board), all systems stop working after a 
while. I 
> want to be trading this before it stops working and not later.  I 
look 
> forward to the rest of Howard's book (and then a number of re-
reads) and 
> may post some more questions here.
> 
> One tedious approach would be to place sample trades from a number 
of 
> systems on a demo account, collate the data, plot the expectancy 
and 
> other stats into Excel and then just trade whichever one is 
currently 
> performing best. My goal in understanding how the AB backtester 
works 
> and how to apply it practically to my chosen market, is to speed 
this up 
> and remove the tedium from this.
> 
> Thanks for the names of the other authors: will need to look these 
up 
> but hope that they are not too "quantsy" for my intellectual level. 
As 
> you say we should be able to trade without needing PhDs. (If I am 
wrong 
> here please correct me asap: at my age I may prefer to engage in 
other 
> activities in my declining years rather than pursuing a futile 
exercise.)
> 
> Thanks for the UKB link and associated references. That amount of 
> reading should keep me off the streets for awhile!
> 
> Regards
> 
> ChrisB
> 
> 
> 
> 
> 
> brian_z111 wrote:
> >
> > Hello Chris,
> >
> > Yes, I have some thoughts on the subject and a reference or two.
> >
> > In fact I am probably part of the way towards a decent book on the
> > subject (presented from a traders point of view) although I doubt
> > that I will ever write it.
> >
> > You actually have three topics there:
> >
> > 1) statistics (statistical significance) ,
> > 2) statistical significance as it is applied to optimization
> > (selecting the 'best performer' from a number of possible 
parameters),
> > 3) ongoing re-optimization of a system to 'synchronise' with 
current
> > (short term) 'market conditions'.
> >
> > For effective consideration it would be better to separate the 
three
> > subjects, as far as possible, although from a traders point of 
view
> > they all come under the one heading of "system design, testing and
> > evaluation" )SDT&E.
> >
> > The best people for you to ask about that subject, in the context 
of
> > your question, would be those who are making a living using 
systems
> > that they designed, using optimization as a key part of the 
process,
> > or specialists in the field.
> >
> > I'm not in either of those categories.
> >
> > I do, however, know a few who are. For a start we have our very 
own
> > Fred and Howard, who fit into one, or both.
> >
> > Then, there is Pardo, who is the father of SDT&E as we know it
> > (outside of academic circles)as well as Prof David Aronson, Lars
> > Kestner, Stridsman, Taleb.
> >
> > None of them are bad reads; some are better than others.
> >
> > In fact Aronson is my No1 favorite trading book and Howards book 
is
> > in my top five. Aronson has a strong intellect and couples that 
with
> > the ability to make a difficult subject sound easy as well as 
being
> > readable and entertaining.
> >
> > IMO one of the handicaps we are under is that the subject evolved
> > from, and has been dominated by, academia while what we actually 
need
> > is a pragmatic amnual of "applied stats for traders". Aronson's 
book
> > goes a fair way, albeit not all the way, towards achieving that.
> >
> > (As far as I am concerned, anyone who thoroughly understands 
Aronsons
> > work, and can apply it, knows 95%, or more, of everything they 
need
> > to know on the subject).
> >
> > Howards book is amongst my top 5 because it is training orientated
> > and full of examples written in my one and only programming 
langauge
> > (AFL). It complements Aronson nicely.
> >
> > Pardo is a little old and IMO doesn't go far enough, although to 
be
> > fair I feel he did intend it to be an introductory text.
> >
> > Stridsman isn't too bad at all.
> >
> > Klestner disappointed a little although his claim to fame is the K
> > Ratio.
> >
> > Taleb I found neither entertaining nor informative although others
> > seem to recommend it.
> >
> > There's a rudimentary post on statistics at the UKB - I will file
> > details of the authors I have mentioned above to that post in the
> > next day or two. It's easier for me to put them there which saves 
me
> > having to repeat this post in the future. There are a few bits and
> > pieces there - nothing much really - I hope to add to it one day.
> >
> > If you search this forum under WhitneyBroach OR Whitney (author) 
AND
> > Aronson (contents) you will find some discussion (Whitney first
> > posted on Aronson).
> >
> > I might post some more discussion later but in the meantime here 
is
> > something to think about:
> >
> > Your doctor informs you that you have an incurable, painless, 
disease
> > and that you only have two weeks to live (there's absolutely no
> > chance of a cure and you will die peacefully in your sleep).
> >
> > As a last resort he offers you an experimental pill that has been
> > designed to extend our longevity.
> >
> > The pill has been trialled on 100 people.
> > 53% were totally cured and lived healthily and happily until they
> > were 120 (they died peacefully in their sleep too).
> >
> > The other 47% also lived until they were 120- but they suffered
> > horrendous side effects and lived out their lifes in unimagineable
> > pain (the pill over-rides any attempts at euthenasia).
> >
> > Will you take the pill or not?
> >
> > For you statistics is no longer a theoretical subject.
> > Your decision will have very real, life changing consequences.
> >
> > Now it is a fact that the majority of people will choose to take 
the
> > pill; it's in our makeup to do so (the pain is in the future and
> > not 'real' at the time of making the decision, we emphasis the
> > positive and eliminate the negative, 53/47 sounds like good odds 
to
> > most of us etc).
> >
> > But, if we step back a pace and evaluate the problem objectively,
> > rationally, logically (would one of those have been enough?) what
> > decision would we make then?
> >
> > How about if we are a trained statistician, or maybe a 
professional
> > trader?
> >
> > How about if the Doctor said' "You don't have to decide now. There
> > will be a new trial, using 10,000 people, starting tomorrow and
> > finishing in three days; the results will be known immediately. 
You
> > can wait until then to decide if you like, but there is only a
> > limited number of pills available and there is no guarantee that 
you
> > will still receive one in four days time"?
> >
> > What is your answer now?
> >
> > IMO it's a binomial problem.
> >
> > http://www.amibroke r.org/userkb/ ?s=Statistics 
> > <http://www.amibroker.org/userkb/?s=Statistics>
> >
> > brian_z
> >
> > --- In amibroker@xxxxxxxxx ps.com 
> > <mailto:amibroker%40yahoogroups.com>, ChrisB <kris45mar@ ..> 
wrote:
> > >
> > > What is a valid or reasonable number of backtest results to 
subject
> > to
> > > Optimization?
> > >
> > > For general statistics a minimum of 30 or so is needed to start
> > getting
> > > valid StdDevs etc.
> > >
> > > If I run a backtest on hourly currency data over three months I 
get
> > > around 16 -20 tradeable signals per currency.
> > > This give a nice smooth plateau on 3D optimization.
> > >
> > > If I test over two months of data I get around 10 - 12 trades
> > >
> > > If I test over only 1 month I get only 5 or 6 trades.
> > >
> > > These shorter time periods still give visually acceptable 3D
> > plateaus
> > > but I am wondering if there is enough data to be statistically
> > significant.
> > >
> > > I am trying to get a handle on how close I can get to current
> > > fluctuations in the market without hitting noise. The idea 
being to
> > redo
> > > the Optimization every x time frame and shift the entry and exit
> > > parameters to stay in the middle of the plateau.
> > >
> > > Of course I can backtest over longer time frames, say 6 months 
of
> > data,
> > > shifting the starting date forward by one month at a time, but 
this
> > > would seem to introduce more "lag" into my selection of best
> > parameters
> > > to trade.
> > >
> > > Does anyone have any thoughts/references on this?
> > >
> > > --
> > > Regards
> > >
> > > ChrisB
> > >
> >
> >  
> 
> 
> -- 
> Regards
> 
> ChrisB
>




Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to 
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html
 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:amibroker-digest@xxxxxxxxxxxxxxx 
    mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/