|
PureBytes Links
Trading Reference Links
|
I know. I have both of them. J
From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Howard B
Sent: Thursday, October 15, 2009
10:03 AM
To: amibroker@xxxxxxxxxxxxxxx
Subject: Re: [amibroker] Re: Is
the Walk forward study useful?
Hi Richard --
Two books are already available. www.blueowlpress.com
The third, Advanced AmiBroker, should be out by about April 2010 and will cover
this topic. If I get it right, it should be as valuable to systems
developers as the Vince and Tharp books, with a lot more practical information
and implementation code.
Thanks,
Howard
On Thu, Oct 15, 2009 at 6:08 AM, Richard N Park <rnpark@xxxxxxxxxxnet> wrote:
Bravo, Howard!
You should write a book. J
Hi Bing, and all --
I think we need a reality check.
First -- computing the t-test, or any other metric using the results from
in-sample runs has no value. Almost any trading system can have the
parameters, logic, time frame, and reporting period adjusted so that the
in-sample results are very profitable. In my speeches, I show a series of
slides, each slide displayed in two steps. Step 1 is the in-sample equity
after optimizing over an in-sample period. The chart ends at the end of
the in-sample period. It always looks good. When the results of
testing a system do not look good, we do not spend any more time on that
system, but go on to others where it does look good. Step 2 adds an
out-of-sample data period, applies the same system, and shows the out-of-sample
equity. For some systems the equity continues to rise, for some it goes
flat, for others it drops sharply. There is no way to tell what will happen
in the out-of-sample period without testing the out-of-sample period.
That is -- there is no information in the in-sample results that predict the
out-of-sample results. There is no way to know whether the model has fit
itself to the signal component of the data that contains the pattern leading to
profitable trades or to some noise component of the data that does not exist in
the out-of-sample data.
Second -- if someone has a trading system that has produced a truly
out-of-sample set of closed trades where the t-test, or any other fitness
metric, would be embarrassingly high without limiting the value used as the
number of data points, he or she should call me. I can help them find a
semitrailer large enough to carry all the money they will make trading that
system. Limiting the number used as N is a non-issue, in addition to
being bad procedure.
Tharp admits that he is not a statistician, and that he finds some of the
mathematics involved in position sizing and fitness function analysis to be at
his comfort limit. It is apparent to me that the systems he shows as
standards for SQNs of 2, 3, 4, 5, and so forth are not the result of trading
system runs -- either in-sample or out-of-sample -- to which he has applied his
metric; they are artificial examples constructed so the results turn out as he
wants them to. That is not a bad thing in itself, but when he suggests
that we should go on searching for systems with SQNs of 10 or more (page 279 of
"Definitive Guide to Position Sizing"), that is completely
unrealistic and will send naive systems developers off on windmill-tilting
quests that will never be successful.
One of the difficulties using Tharp's data sets is that the standard deviation
of losing trades is zero for some of them. That is not only unrealistic,
but it makes computation of metrics that include standard deviation of losing
trades, such as Sortino ratio, difficult.
Do the following experiment. Put together a data set that represents
potentially realistic trading results that you hope to achieve -- be
optimistic, but realistic. If you would expect one trade a week, a set of
52 data points represents a year. 252 data points if you would have a
trading result every day. Each data point is the number of dollars gained
or lost from that closed trade, based on trading a single unit -- one futures
contract, one hundred shares of stock, or whatever your one unit is.
Compute the mean, standard deviation, and t-test score -- where the t-test is
"what is the level of confidence that the mean is greater than
0". Put the data into a text file and use it as input to Equity Monaco,
ProSizer, or Market System Analyzer. Run some Monte
Carlo simulations to see what the equity could look like from
trading one year. Systems with t-test scores, based on expectancy, of 3
will make you incredible amounts of money. Dummy up a data set with a
t-test / SQN of 10 and you will see how unrealistic that is.
The critical elements needed to apply aggressive position sizing are:
trading something that can be scaled up as desired, positive expectancy,
limited semi-deviation (standard deviation of losing trades), and frequent
trading.
Being able to limit the amount lost on a losing trade is essential. Risk
of bankruptcy (and all trading systems have a non-zero probability of going
bankrupt) increases dramatically as the standard deviation of losing trades
increases. Dummy up some data and runs some more tests. The effect
the amount lost on losing trades has on system performance is scary.
Now go back to your trading system development platform (AmiBroker, of course)
and test your trading systems, this time focusing on limiting the standard
deviation of losing trades. If your system holds a long time (more than a
week or two), pay attention to Maximum Adverse Excursion. If you attempt
to apply aggressive position sizing to a system that has reasonable results for
closed trades, but has large MAE intra-trade, you will get stopped out (or
scared out) intra-trade.
Remember -- if you are applying position sizing to your trading system, your
largest loss will come when you have your largest position. Read Ralph
Vince.
As usual -- be careful to base your analysis and decision whether to trade any
system on truly out-of-sample results. Decisions based on in-sample or
contaminated out-of-sample results seriously underestimate the probability of
bankruptcy.
Repeat after me -- keep a positive expectancy, limit losing trades, trade
frequently.
Thanks for listening,
Howard
On Thu, Oct 15, 2009 at 1:11 AM, bingk66 <bing.kwok@xxxxxxxxxcom.au>
wrote:
Hi Howard,
If there are no means to limit the number of transactions in the calcs, then
one seriously runs the risk of challenging the mystical t-test score of 7 that
you spoke about previously.
As an example, if the OOS test was run over a 5 year period with 5000
transactions (a mere 1000 transaction/year, which is not excessive,
especially for very short term trades), sqrt(5000) alone would yield in excess
of 70 for the multiplier. This would leave expectancy/StdDev of R with just a
target of 0.1, to reach the 7 t-tests score.
Now, if you had 1,000,000 tranasctions in your OOS test....
The concept of limiting the trade count does make sense to me. Maybe 100 is too
low, and should be set higher. There does come a point whereby the sqrt(N) part
of the equation will render the rest of the equation irrelevant once N gets too
large.
$0.02
Bing
> Hi Zozu --
>
> I must disagree with Van Tharp on this.
>
> If the runs are truly out-of-sample, then each and every one contributes
to
> the computation. It makes no sense to limit the count to 100. It is poor
> procedure to limit the count. It is bad science to limit the count. Do not
> limit the count.
>
> If the runs are in-sample, then the test has no meaning anyway. Computing
> the t-test statistic using any N will be misleading. Do not even do the
> computation. If a decision to trade a system is made after computing the
> t-test statistic on trades that came solely from in-sample results, there
is
> an extremely high probability that a Type I error will be committed. That
> is, the trader will believe that his system is better than random, when it
> is in fact not better than random. Type I errors result in loss of money.
>
> Thanks,
> Howard
>
>
> On Tue, Oct 13, 2009 at 10:54 AM, zozuzoza <zozuka@xxx>
wrote:
>
> >
> >
> > Hi Howard,
> >
> > Limiting the number of N doesn't mean that you are not using all
trades for
> > the calculation of SQN. Only the sqrt(N) part of the formula is
limited in
> > order not to distort the results if there are many trades. It makes
sense.
> > The other part of the formula does count on all the trades.
> >
> > Zozu
> >
> >
> >
>
__._,_.___
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
![]()
__,_._,___
|
|