|
PureBytes Links
Trading Reference Links
|
Hello Fred,
Precisely.
I'm not going crazy after-all!
What method or methods will tell us how the system is likely to
perform out of sample; since in the end system trading is nothing
but a perpetual walk forward test?
BrianB2.
--- In amibroker@xxxxxxxxxxxxxxx, "Fred" <ftonetti@xxx> wrote:
>
> While MCS is a good tool for validating some things it is not a
> substitute for out of sample and/or walk forward testing ... If
for
> example I:
>
> - Write a system
> - Test it to make sure the rules are working as intended
> - Optimize the variables to have the system produce the best
results
> it can based on some metric or metrics within the confines of the
> rules
> - Use an MCS on the trades that are generated
>
> This tells me nothing about how the system is likely to perform
out
> of sample. It only tells me about the statistics related to the
> optimized rules of the system which are the result of scrambling
the
> order of the trades that resulted from using the same in sample
data
> that system was optimized on.
>
> --- In amibroker@xxxxxxxxxxxxxxx, "sebastiandanconia"
> <sebastiandanconia@> wrote:
> >
> > I only offer this as a consideration when using such testing,
not
> as a
> > criticism of Monte Carlo Simulations. A subtle but significant
> point
> > (IMO) when using MCS: They may or may not be applicable to
> > trading/investing, because the markets don't always behave
> randomly.
> >
> > An example of when a MCS would clearly be appropriate: Let's
say
> you're
> > a defense contractor manufacturing a part for the International
> Space
> > Station. The part is critical, but because of limitations in
> > engineering technology it has a high failure rate, and there's
no
> way of
> > forecasting in advance if a part will fail. However, although
the
> > failure rate is high, it's also very consistent. Until
technology
> > advances sufficiently the only practical solution is to keep
> plenty of
> > spares on-hand.
> >
> > A MCS could tell you what the optimal number of spares to keep
on-
> hand
> > would be. The part failures are random, but MC could tell you
the
> > likelihood of two, three, five, ten, etc., consecutive
failures.
> You
> > might determine that there would only be a 1/100,000 chance that
6
> > spares in a row would fail, so you might advise NASA to stock at
> least 6
> > spares at all times.
> >
> > Some trading systems, though, will be successful because they
take
> > advantage of repeating sequences of events, not random events.
> Business
> > cycles go through a specific sequence, company growth follows a
> certain
> > pattern from infancy to maturity, price trends/reversals follow a
> > sequence, etc. If trades based on reliable, repeating patterns
are
> > taken out of order by a MCS such that a massive drawdown or a
> > bankrupting series of losers occurs, that can distort the value
of
> the
> > trading method by putting the trades in an order that wouldn't
> occur.
> >
> > Soapbox alert!:) Another reason that "Why does it work?" is
such
> an
> > important question with trading systems, since a good answer to
> that
> > question can lead to a good answer for another important
> question, "When
> > WON'T it work?"
> >
> >
> > S.
> >
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "brian.z123" <brian.z123@>
wrote:
> > >
> > > OT:margin of error example.
> > >
> > > As the trader is more interested in the general population of
> future
> > > trades than the test sample, what can be learnt from the
sample?
> > >
> > > One answer is to trade the system for a decade or two and find
> out.
> > > Another option is to simulate decades or even centuries of
> trading
> > > by applying Monte Carlo analysis.
> > > In laymans terms MCS is a computer generated, random walk
through
> > > *all*, of the possible trading outcomes based on the trading
> sample
> > > provided.
> > > The result is a report or system profile that provides
> statistics on
> > > which to base our levels of trading confidence for the future.
> > > There are other ways of sneaking a peak into a trading systems
> > > future but MCS is the most commonly used.
> > > I have developed my own system, that I don¡¦t want to headline
> here
> > > for various reasons, not the lest of which is that I can¡¦t
> provide
> > a
> > > mathematical proof if called on to do so.
> > >
> > > Assuming that an MCS has been conducted on a sample of 50
trades
> > > produced by a back-tested system and the report indicates that
> the
> > > meanW/meanL for the system over a large number of trading
> > > simulations is 53/47. The StDev is 40% for both Wins and
Losses.
> > > How confident can we be in that result?
> > >
> > > From David Lanes statistical website:
> > > http://davidmlane.com/hyperstat/A103397.html
> > > The standard error of a statistic is the standard deviation of
> the
> > > sampling distribution of that statistic.
> > > The formula for the standard error of the mean is:
> > >
> > > StErrorOfMeanPopulation = StDevPopulation/SqRt(N)Sample
> > >
> > > For any statistic:
> > >
> > > StErrorOfMeanPopulation(statistic) = StDevPopulation
> (statistic)/SqRt
> > > (N)Sample
> > >
> > > Applying the StdErrorMean equation to the example:
> > >
> > > Back-test sample size N = = 50,
> > > MCS meanWin/meanLoss = = 53/47,
> > > MCS Win StDev% = = 40%,
> > > MCS Win StDev$ = = 40% x 53 = = 21.2,
> > > MCS Loss StDev% = = 40%,
> > > MCS Loss StDev$ = = 40% x 47 = = 18.8,
> > >
> > > StdError%Wins = = 40/SqRt(50) = = multiply mean by +/- 5.6 %,
> > > Trading Win range = = 50 ¡V 56,
> > > (min = = 53 x 0.943 = = 50, max = = 53 X 1.056 = = 56).
> > >
> > > The same result can be obtained using StDev as a number ($)
> rather
> > > than as a percentage.
> > >
> > > StdError$Wins = = 21.2/SqRt(50) = = +/- 3 = = Win range = =
47
> +/-3
> > > = = 50 -56 .
> > >
> > > Repeating the calculations for Losses shows the the mean Losses
> > > range between 44 ¡V 50.
> > >
> > > I chose this extreme example to demonstrate the outcome for a
> small
> > > back-test sample with high volatility trades and a small
win/loss
> > > margin.
> > >
> > > If the same trading pattern were generated from a back-test
> sample
> > > of 2500 trades and the simulated meanWins and mean Losses each
> had a
> > > StDev of 10% the range for the margin of error would be:
> > >
> > > Wins 52.9 ¡V 53.1,
> > > Losses 46.9 ¡V 47.1.
> > >
> > > This means that the we can be 95% confident the real mean
values
> are
> > > somewhere within those ranges.
> > > For a higher level of confidence the range will be greater.
> > >
> > > Resorting to the age-old teaching trick of asking the students
> for
> > > the answer while pretending to already know it yourself; can
> anyone
> > > in the forum tell me if this is the correct way to use
StdError
> when
> > > applied to trading?
> > >
> > >
> > >
> > > BrianB2 ?º
> > > --- In amibroker@xxxxxxxxxxxxxxx, "brian.z123" brian.z123@
> > > wrote:
> > > >
> > > > Part1 of Project Based Training No1.
> > > >
> > > > The objective of the project is to introduce new traders to
the
> > > main
> > > > concepts of system design/testing and demonstrate their
> > > application
> > > > in AmiBroker.
> > > > At the same time it is hoped that the ideas presented will
> provoke
> > > > discussion and provide trading stimulation.
> > > >
> > > > All of the stages in the design process will not be
> demonstrated
> > > as
> > > > most have already been covered elsewhere in the AmiBroker
> support
> > > > material.
> > > >
> > > > A basic understanding of the application of some statistical
> > > methods
> > > > to the trading environment is a pre-requisite.
> > > > The opening topics address this need.
> > > >
> > > > To those who find the subject matter new *the project* will
be
> a
> > > > workbook .
> > > > To those who have experience in the subject it will be an
> > > > opportunity to workshop.
> > > >
> > > > I would like to acknowledge my indebtedness to the academic
> > > > community .
> > > > I often refer to the material so generously interpreted for
the
> > > > layperson and made available at websites by academic
> specialists,
> > > > particularly those associated with Universities.
> > > >
> > > >
> *******************************************************************
> > > > Margin of Error.
> > > >
> > > > Back-testing of historical data provides traders with a
> sample,
> > > > typical of the trade they are testing. From that sample they
> make
> > > > inferences about the larger group, or population, of all past
> > > trades
> > > > and future trades, of the same type, that were not included
in
> > > their
> > > > test window.
> > > > Despite the fact that the people who teach them to back-test
> also
> > > > teach them that the past can not predict the future, some
> continue
> > > > to act as if it can.
> > > >
> > > > If the past can't predict the future. How can anyone trade
with
> > > > confidence?
> > > >
> > > > The answer is that while the future can't be predicted, the
> > > > likelihood of some mathematically defined outcomes can be
> > > predicted
> > > > with a degree of confidence.
> > > > Statistics is the mathematical discipline that manages that
> very
> > > > well.
> > > >
> > > > The caveat is that to apply statistical methods to trading
> > > samples,
> > > > the assumption is made that they are the result of a random
> > > process.
> > > > Where the trading system chosen is biased to non-random
> behaviour
> > > it
> > > > will be prone to failure if the market acts contrary to that
> bias.
> > > >
> > > > For that reason system traders are faced with a choice
between
> > > > attempting to define market behaviour e.g. a trend, and pick
a
> > > > system to suit that, or search for a universal signal that is
> > > > consistent irrespective of any assumed market bias.
> > > >
> > > > If statistics can predict the likelihood of future trading
> > > outcomes,
> > > > how accurate will it be?
> > > >
> > > > *Standard error* or *margin of error* offers traders a
> solution
> > > but
> > > > they are not subjects that are often discussed.
> > > >
> > > > In his book ,*Design, Testing, and Optimisation of Trading
> > > Systems*
> > > > (John Wiley & Sons, 1992), Robert Pardo raises the issue of
the
> > > > accuracy of trading *predictions* based on the size of the
> sample
> > > > used:
> > > >
> > > > * The sample size must be large enough to allow the trading
> system
> > > > to generate a statistically significant sample of trades.
> > > > A sample of one trade is certainly insignificant, whereas a
> sample
> > > > of 50 trades or more is generally adequate.*
> > > >
> > > > He uses Standard Error as a measure of significance:
> > > >
> > > > StdError = = 1/SquareRoot(sample size),
> > > >
> > > > 1/SqRt(50) = = 14.1%.
> > > >
> > > > There is little by way of further explanation provided.
> > > >
> > > > Applying the formula to a greater number of samples:
> > > >
> > > > Where N = = the number of trades in the sample
> > > >
> > > > StdError factor = = 1/SqRt(N)
> > > > StdError% = 1/SqRt(N) * 100
> > > >
> > > > If N = = 2500 the StdError% = = 1/SqRt(2500) * 100 = = +/- 2%
> > > > If N = = 10000 the StdError% = = 1/SqRt(10000) * 100 = = +/-
> 1%
> > > >
> > > > A trade sample of 10000 to provide statistical accuracy of
1%
> is
> > > not
> > > > easily achievable for traders, although a lot easier than
> > > accurately
> > > > surveying the eye colour of Polar Bears.
> > > >
> > > > Pardos equation is in fact, a rounding of the StdError
equation
> > > for
> > > > a 95% level of confidence:
> > > >
> > > > Margin of error at 99% confidence = = 1.29/SqRt(N)
> > > > Margin of error at 95% confidence = = 0.98/SqRt(N)
> > > > Margin of error at 90% confidence = = 0.82/SqRt(N)
> > > >
> > > > Later in the project I will use a basic random number
> generator,
> > > > within Xcel, to provide a visual aid that traders can use to
> > > > understand the *sample* concept and decide for themselves
what
> > > > constitutes an adequate sample.
> > > >
> > > > Wikipedia provides some additional clarity on the subject:
> > > >
> > > > http://en.wikipedia.org/wiki/Margin_of_error
> > > >
> > > > *The margin of error expresses the amount of the random
> variation
> > > > underlying a survey's results. This can be thought of as a
> measure
> > > > of the variation one would see in reported percentages if
the
> same
> > > > poll were taken multiple times. The larger the margin of
error,
> > > the
> > > > less confidence one has that the poll's reported percentages
> are
> > > > close to the "true" percentages, that is the percentages in
the
> > > > whole population.*
> > > >
> > > > *An interesting mathematical fact is that the margin of error
> > > > depends only on the sample size and not on the population
size,
> > > > provided that the population is significantly larger than the
> > > sample
> > > > size, and provided a simple random sample is used. Thus for
> > > > instance¡K¡K.the running example with 1,013 random
> > samples¡K¡Kwould
> > > > yield essentially the same margin of error (4% with a 99%
> level of
> > > > confidence) regardless of whether the
> > population¡K¡K¡K.consisted of
> > > > 100,000 or 100,000,000.*
> > > >
> > > > In short the tail of the trading system sample is swinging
the
> > > > trading system cat.
> > > >
> > > > BrianB2
> > > >
> > > > The material contained in this topic is for educational and
> > > > discussion use only.
> > > > It is not intended as financial advice and should not be
> construed
> > > > as such.
> > > > The author is not an accredited academic or financial
advisor.
> > > >
> > >
> >
>
Content-Description: "AVG certification"
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.14.11/542 - Release Date: 11/20/2006
|