|
PureBytes Links
Trading Reference Links
|
Correction - sorry that should be
Plot sample error (on the Y axis) against the number (N) of data
POINTS IN THE SAMPLE (on the X axis) and it is an inverse exponential
curve.
I don't want to confuse it with Standard Error which measures
variance between several samples (for our purposes a sample is a
backtest and the number of datapoints is the number of closed trades
for the backtested period).
brian_z111
--- In amibroker@xxxxxxxxxxxxxxx, "brian_z111" <brian_z111@xxx> wrote:
>
> Chris,
>
> No, we don't have to be Quants to trade.
> I actually only use basic stats in a simplistic way - it is how and
> where you use it that counts (I did have to dig into a lot
> of 'theory' to find that out and find what I wanted though).
>
> All of the authors I mentioned, except for Kennedy & Bernstein, are
> trading orientated so pick one or two that you like and dig in
(that
> should take you up the easy 80% of the learning curve).
>
> Some further discussion:
>
> (I'll try to keep it short and give you some practical pointers -
> which everyone knows is hard for me to do).
>
> 1) SAMPLE ERROR
>
> Sample error is very real.
>
> I have plotted thousands of pseudo stock charts using the random
> generator in Excel and watched sample error play out time and time
> again.
>
> Plot sample error (on the Y axis) against the number (N) of data
> samples (on the X axis) and it is an inverse exponential curve.
>
> Sample error% = 1/sqrt(N) * 100
>
> The reduction in error that we gain as we increase sample size
> decreases exponentially.
> There is an optimal point somewhere around N = 1000 where we don't
> get much improvement in accuracy with increased sample size.
>
> Having seen sample error in thousands of labtests I find it very
> difficult to accept any analysis that uses less than 300-400
samples
> (as a minimum). If I can get more I use them (in my newest system
> that I am working on I am getting thousands of trade samples per
> test).
> I go so far as to use data from other markets and timeframes to
> produce my large datasets (plus other tricks as well).
>
> 2) SAMPLE ERROR IN OPTIMIZATION
>
> I am very sceptical about optimization that ignores sample size.
> When you have a nice round optimization 3D graph how meaningful
could
> it possibly be if it is produced from only 30 trade data points.
The
> sample size behind the plots on that graph are not evenly
distributed
> and possibly not statistically valid.
>
> I haven't found any trading commentator who discusses this point or
> gives examples of how to account for sample error when optimizing.
> I assume it could be included as part of the test criteria
(objective
> function)?
>
> Perhaps people are doing things like that in private and just
haven't
> told anyone about it.
>
> 3) SYNCHRONIZING TO THE CURRENT MARKET
>
> I am very sceptical about this (once again it places the trader in
> the position of relying on very small datasets).
>
> Returning to the analogy of vehicle performance tests.
>
> Unlike F1 racing we don't know anything about the track or the
> weather (they know everything about in advance and prepare their
car
> accordingly).
>
> Since we can't predict what track we will be racing on or what
> conditions we will be racing under why would we prepare as if we do?
>
> I would rather use a system that performs reasonably well under all
> conditions than try to tune-up to unpredictable and ever changing
> conditions.
>
> brian_z111
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, ChrisB <kris45mar@> wrote:
> >
> > Thanks Brian, just the sort of feedback and references I was
> looking for:
> >
> > I guess you have put this into English better than I did:
> >
> > 3) ongoing re-optimization of a system to 'synchronise' with
current
> > (short term) 'market conditions'.
> >
> > This is the crux of the question, as I have tried to verbalize in
> my
> > response to Howard above, but not as succinctly as you have.
> >
> > This then is the philosophical question: is it better
> to "synchronize"
> > frequently and often and stay in tune with market movements, or
> better
> > to perform endless walkforwards? I for one wouldn't care less if
a
> > system stopped working as long as I could detect that early
enough
> and
> > then just stop trading it. After all, as Howard says in his book
> (and in
> > earlier posts to this board), all systems stop working after a
> while. I
> > want to be trading this before it stops working and not later. I
> look
> > forward to the rest of Howard's book (and then a number of re-
> reads) and
> > may post some more questions here.
> >
> > One tedious approach would be to place sample trades from a
number
> of
> > systems on a demo account, collate the data, plot the expectancy
> and
> > other stats into Excel and then just trade whichever one is
> currently
> > performing best. My goal in understanding how the AB backtester
> works
> > and how to apply it practically to my chosen market, is to speed
> this up
> > and remove the tedium from this.
> >
> > Thanks for the names of the other authors: will need to look
these
> up
> > but hope that they are not too "quantsy" for my intellectual
level.
> As
> > you say we should be able to trade without needing PhDs. (If I am
> wrong
> > here please correct me asap: at my age I may prefer to engage in
> other
> > activities in my declining years rather than pursuing a futile
> exercise.)
> >
> > Thanks for the UKB link and associated references. That amount of
> > reading should keep me off the streets for awhile!
> >
> > Regards
> >
> > ChrisB
> >
> >
> >
> >
> >
> > brian_z111 wrote:
> > >
> > > Hello Chris,
> > >
> > > Yes, I have some thoughts on the subject and a reference or two.
> > >
> > > In fact I am probably part of the way towards a decent book on
the
> > > subject (presented from a traders point of view) although I
doubt
> > > that I will ever write it.
> > >
> > > You actually have three topics there:
> > >
> > > 1) statistics (statistical significance) ,
> > > 2) statistical significance as it is applied to optimization
> > > (selecting the 'best performer' from a number of possible
> parameters),
> > > 3) ongoing re-optimization of a system to 'synchronise' with
> current
> > > (short term) 'market conditions'.
> > >
> > > For effective consideration it would be better to separate the
> three
> > > subjects, as far as possible, although from a traders point of
> view
> > > they all come under the one heading of "system design, testing
and
> > > evaluation" )SDT&E.
> > >
> > > The best people for you to ask about that subject, in the
context
> of
> > > your question, would be those who are making a living using
> systems
> > > that they designed, using optimization as a key part of the
> process,
> > > or specialists in the field.
> > >
> > > I'm not in either of those categories.
> > >
> > > I do, however, know a few who are. For a start we have our very
> own
> > > Fred and Howard, who fit into one, or both.
> > >
> > > Then, there is Pardo, who is the father of SDT&E as we know it
> > > (outside of academic circles)as well as Prof David Aronson, Lars
> > > Kestner, Stridsman, Taleb.
> > >
> > > None of them are bad reads; some are better than others.
> > >
> > > In fact Aronson is my No1 favorite trading book and Howards
book
> is
> > > in my top five. Aronson has a strong intellect and couples that
> with
> > > the ability to make a difficult subject sound easy as well as
> being
> > > readable and entertaining.
> > >
> > > IMO one of the handicaps we are under is that the subject
evolved
> > > from, and has been dominated by, academia while what we
actually
> need
> > > is a pragmatic amnual of "applied stats for traders". Aronson's
> book
> > > goes a fair way, albeit not all the way, towards achieving that.
> > >
> > > (As far as I am concerned, anyone who thoroughly understands
> Aronsons
> > > work, and can apply it, knows 95%, or more, of everything they
> need
> > > to know on the subject).
> > >
> > > Howards book is amongst my top 5 because it is training
orientated
> > > and full of examples written in my one and only programming
> langauge
> > > (AFL). It complements Aronson nicely.
> > >
> > > Pardo is a little old and IMO doesn't go far enough, although
to
> be
> > > fair I feel he did intend it to be an introductory text.
> > >
> > > Stridsman isn't too bad at all.
> > >
> > > Klestner disappointed a little although his claim to fame is
the K
> > > Ratio.
> > >
> > > Taleb I found neither entertaining nor informative although
others
> > > seem to recommend it.
> > >
> > > There's a rudimentary post on statistics at the UKB - I will
file
> > > details of the authors I have mentioned above to that post in
the
> > > next day or two. It's easier for me to put them there which
saves
> me
> > > having to repeat this post in the future. There are a few bits
and
> > > pieces there - nothing much really - I hope to add to it one
day.
> > >
> > > If you search this forum under WhitneyBroach OR Whitney
(author)
> AND
> > > Aronson (contents) you will find some discussion (Whitney first
> > > posted on Aronson).
> > >
> > > I might post some more discussion later but in the meantime
here
> is
> > > something to think about:
> > >
> > > Your doctor informs you that you have an incurable, painless,
> disease
> > > and that you only have two weeks to live (there's absolutely no
> > > chance of a cure and you will die peacefully in your sleep).
> > >
> > > As a last resort he offers you an experimental pill that has
been
> > > designed to extend our longevity.
> > >
> > > The pill has been trialled on 100 people.
> > > 53% were totally cured and lived healthily and happily until
they
> > > were 120 (they died peacefully in their sleep too).
> > >
> > > The other 47% also lived until they were 120- but they suffered
> > > horrendous side effects and lived out their lifes in
unimagineable
> > > pain (the pill over-rides any attempts at euthenasia).
> > >
> > > Will you take the pill or not?
> > >
> > > For you statistics is no longer a theoretical subject.
> > > Your decision will have very real, life changing consequences.
> > >
> > > Now it is a fact that the majority of people will choose to
take
> the
> > > pill; it's in our makeup to do so (the pain is in the future and
> > > not 'real' at the time of making the decision, we emphasis the
> > > positive and eliminate the negative, 53/47 sounds like good
odds
> to
> > > most of us etc).
> > >
> > > But, if we step back a pace and evaluate the problem
objectively,
> > > rationally, logically (would one of those have been enough?)
what
> > > decision would we make then?
> > >
> > > How about if we are a trained statistician, or maybe a
> professional
> > > trader?
> > >
> > > How about if the Doctor said' "You don't have to decide now.
There
> > > will be a new trial, using 10,000 people, starting tomorrow and
> > > finishing in three days; the results will be known immediately.
> You
> > > can wait until then to decide if you like, but there is only a
> > > limited number of pills available and there is no guarantee
that
> you
> > > will still receive one in four days time"?
> > >
> > > What is your answer now?
> > >
> > > IMO it's a binomial problem.
> > >
> > > http://www.amibroke r.org/userkb/ ?s=Statistics
> > > <http://www.amibroker.org/userkb/?s=Statistics>
> > >
> > > brian_z
> > >
> > > --- In amibroker@xxxxxxxxx ps.com
> > > <mailto:amibroker%40yahoogroups.com>, ChrisB <kris45mar@ ..>
> wrote:
> > > >
> > > > What is a valid or reasonable number of backtest results to
> subject
> > > to
> > > > Optimization?
> > > >
> > > > For general statistics a minimum of 30 or so is needed to
start
> > > getting
> > > > valid StdDevs etc.
> > > >
> > > > If I run a backtest on hourly currency data over three months
I
> get
> > > > around 16 -20 tradeable signals per currency.
> > > > This give a nice smooth plateau on 3D optimization.
> > > >
> > > > If I test over two months of data I get around 10 - 12 trades
> > > >
> > > > If I test over only 1 month I get only 5 or 6 trades.
> > > >
> > > > These shorter time periods still give visually acceptable 3D
> > > plateaus
> > > > but I am wondering if there is enough data to be statistically
> > > significant.
> > > >
> > > > I am trying to get a handle on how close I can get to current
> > > > fluctuations in the market without hitting noise. The idea
> being to
> > > redo
> > > > the Optimization every x time frame and shift the entry and
exit
> > > > parameters to stay in the middle of the plateau.
> > > >
> > > > Of course I can backtest over longer time frames, say 6
months
> of
> > > data,
> > > > shifting the starting date forward by one month at a time,
but
> this
> > > > would seem to introduce more "lag" into my selection of best
> > > parameters
> > > > to trade.
> > > >
> > > > Does anyone have any thoughts/references on this?
> > > >
> > > > --
> > > > Regards
> > > >
> > > > ChrisB
> > > >
> > >
> > >
> >
> >
> > --
> > Regards
> >
> > ChrisB
> >
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|