[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[amibroker] Re: What is a valid number of Back test results to Optimize?



PureBytes Links

Trading Reference Links

Hello Chris,

Yes, I have some thoughts on the subject and a reference or two.

In fact I am probably part of the way towards a decent book on the 
subject (presented from a traders point of view) although I doubt 
that I will ever write it.

You actually have three topics there:

1) statistics (statistical significance),
2) statistical significance as it is applied to optimization 
(selecting the 'best performer' from a number of possible parameters),
3) ongoing re-optimization of a system to 'synchronise' with current 
(short term) 'market conditions'. 

For effective consideration it would be better to separate the three 
subjects, as far as possible, although from a traders point of view 
they all come under the one heading of "system design, testing and 
evaluation" )SDT&E.

The best people for you to ask about that subject, in the context of 
your question, would be those who are making a living using systems 
that they designed, using optimization as a key part of the process, 
or specialists in the field. 

I'm not in either of those categories.

I do, however, know a few who are. For a start we have our very own 
Fred and Howard, who fit into one, or both.

Then, there is Pardo, who is the father of SDT&E as we know it 
(outside of academic circles)as well as Prof David Aronson, Lars 
Kestner, Stridsman, Taleb.

None of them are bad reads; some are better than others.

In fact Aronson is my No1 favorite trading book and Howards book is 
in my top five. Aronson has a strong intellect and couples that with 
the ability to make a difficult subject sound easy as well as being 
readable and entertaining. 

IMO one of the handicaps we are under is that the subject evolved 
from, and has been dominated by, academia while what we actually need 
is a pragmatic amnual of "applied stats for traders". Aronson's book 
goes a fair way, albeit not all the way, towards achieving that.

(As far as I am concerned, anyone who thoroughly understands Aronsons 
work, and can apply it, knows 95%, or more, of everything they need 
to know on the subject).

Howards book is amongst my top 5 because it is training orientated 
and full of examples written in my one and only programming langauge 
(AFL). It complements Aronson nicely. 

Pardo is a little old and IMO doesn't go far enough, although to be 
fair I feel he did intend it to be an introductory text.

Stridsman isn't too bad at all.

Klestner disappointed a little although his claim to fame is the K 
Ratio.

Taleb I found neither entertaining nor informative although others 
seem to recommend it.

There's a rudimentary post on statistics at the UKB - I will file 
details of the authors I have mentioned above to that post in the 
next day or two. It's easier for me to put them there which saves me 
having to repeat this post in the future. There are a few bits and 
pieces there - nothing much really - I hope to add to it one day.

If you search this forum under WhitneyBroach OR Whitney (author) AND 
Aronson (contents) you will find some discussion (Whitney first 
posted on Aronson).

I might post some more discussion later but in the meantime here is 
something to think about:

Your doctor informs you that you have an incurable, painless, disease 
and that you only have two weeks to live (there's absolutely no 
chance of a cure and you will die peacefully in your sleep).

As a last resort he offers you an experimental pill that has been 
designed to extend our longevity.

The pill has been trialled on 100 people.
53% were totally cured and lived healthily and happily until they 
were 120 (they died peacefully in their sleep too).

The other 47% also lived until they were 120- but they suffered 
horrendous side effects and lived out their lifes in unimagineable 
pain (the pill over-rides any attempts at euthenasia).

Will you take the pill or not?

For you statistics is no longer a theoretical subject.
Your decision will have very real, life changing consequences.

Now it is a fact that the majority of people will choose to take the 
pill; it's in our makeup to do so (the pain is in the future and 
not 'real' at the time of making the decision, we emphasis the 
positive and eliminate the negative, 53/47 sounds like good odds to 
most of us etc).

But, if we step back a pace and evaluate the problem objectively, 
rationally, logically (would one of those have been enough?) what 
decision would we make then?

How about if we are a trained statistician, or maybe a professional 
trader?

How about if the Doctor said' "You don't have to decide now. There 
will be a new trial, using 10,000 people, starting tomorrow and 
finishing in three days; the results will be known immediately. You 
can wait until then to decide if you like, but there is only a 
limited number of pills available and there is no guarantee that you 
will still receive one in four days time"?

What is your answer now?

IMO it's a binomial problem.

http://www.amibroker.org/userkb/?s=Statistics

brian_z 




--- In amibroker@xxxxxxxxxxxxxxx, ChrisB <kris45mar@xxx> wrote:
>
> What is a valid or reasonable number of backtest results to subject 
to 
> Optimization?
> 
> For general statistics a minimum of 30 or so is needed to start 
getting 
> valid StdDevs etc.
> 
> If I run a backtest on hourly currency data over three months I get 
> around 16 -20 tradeable signals per currency.
> This give a nice smooth plateau on 3D optimization.
> 
> If I test over two months of data I get around 10 - 12 trades
> 
> If I test over only 1 month I get only 5 or 6 trades.
> 
> These shorter time periods still give visually acceptable 3D 
plateaus 
> but I am wondering if there is enough data to be statistically 
significant.
> 
> I am trying to get a handle on how close I can get to current 
> fluctuations in the market without hitting noise. The idea being to 
redo 
> the Optimization every x time frame and shift the entry and exit 
> parameters to stay in the middle of the plateau.
> 
> Of course I can backtest over longer time frames, say 6 months of 
data, 
> shifting the starting date forward by one month at a time, but this 
> would seem to introduce more "lag" into my selection of best 
parameters 
> to trade.
> 
> Does anyone have any thoughts/references on this?
> 
> -- 
> Regards
> 
> ChrisB
>




Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to 
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html
 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:amibroker-digest@xxxxxxxxxxxxxxx 
    mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/