|
PureBytes Links
Trading Reference Links
|
Hi Howard,
I follow this discussion with interest and I thank you for your invaluable comments. I am still confused about the statement .."look for systems that have truly out-of-sample t-test scores of 2.0".
In AB terms, what is the objective function we need to compare it to "t-test"? Are we looking for CAR/ADD to be greater than 2.0?
Kind Regards
Richard
--- In amibroker@xxxxxxxxxxxxxxx, Howard B <howardbandy@xxx> wrote:
>
> Hi Mike, and all --
>
> There is a lot to like in Tharp's book, Definitive Guide to Position
> Sizing.
>
> But it appears to me that he developed the concept of system quality number
> without using it as an objective function used in testing and validating
> trading systems. He describes several idealized trading systems that have a
> range of SQNs -- from negative to about 7. One on page 38 is described as:
> Mean expectancy: 3.42
> Standard deviation: 4.89
> Win percentage: 90
> Win/Loss Ratio: 2.35
> Number Trades: 100
> SQN: 6.99
>
> Plugging those numbers into the formula for the t-test, which is
> (mean/stdev) * sqrt(N)
> we get (3.42/4.89)*10 which equals 6.99 -- the same as the SQN.
>
> While he does not call SQN t-test, for each of the examples I worked out,
> the two are the same.
>
> So, one of my points is that SQN is equivalent to t-test where the
> hypothesis that is being tested is "is expectancy greater than 0?"
>
> I like the idea that the statistical test is incorporated into the metric.
> I have been using that concept for some time, and I think it works well. I
> wish he had not called it SQN, and I wish he had not put a Service Mark
> next to it every time it appears in his book. But that concept is good.
>
> I seriously doubt that he, or anyone else, has a series of truly
> out-of-sample closed trades where the t-test of expectancy is 7. It would
> take only a few weeks to turn $10,000 into enough to buy Manhattan with a
> system like that. Which goes to my point that I doubt that any of the
> idealized trading system he describes with large SQNs reflect real trading
> systems. Idealized, yes. In-sample, perhaps, but I doubt it.
> Out-of-sample, almost impossible and I very seriously doubt it.
>
> As a basis for my comment, look at a table of t-test percentile values. I
> use a rule of thumb that the t-test should be above about 2.0, just for its
> ease of computation. With n=10 trades (9 degrees of freedom), 95%
> confidence comes with t-statistic of 1.83. With n=120, 99.5% confidence
> comes with a t-statistic of 2.62.
>
> For sample sizes greater than about 60, the t-distribution and the normal
> distribution are essentially identical. Something 3 standard deviations
> away from the mean happens much much less than 1% of the time. Something 7
> standard deviations away is so rare as to be essentially impossible on our
> time scale. It is nice to see the fantasy equity curve, but it will never
> happen.
>
> If expectancy works as an objective function for a particular trader and
> system, then by all means, use it. As the characteristics of the trade
> returns change, the advantages and disadvantages of using expectancy change.
>
> All of that said, I think Tharp's book is good. It brings out many
> excellent points about the importance of positions sizing. Read it in
> conjunction with Ralph Vince's "The Leverage Space Trading Model." My own
> research and experience is in agreement with both of those.
>
> There is an implication -- one that many people will not find comfortable
> and will argue strongly against. It is this: The only way to generate
> large increases in equity safely is to have a trading system that trades
> frequently, holds a short period of time, and carefully limits losing
> trades.
>
> Yes, there are examples of people who have made a lot of money trading
> infrequently and holding long periods. My contention is that
> 1. we are hearing from those who were successful. Not many people who
> tried to do this succeeded. Most who tried and failed have not remained in
> the trading community and we are not hearing their stories of failure.
> 2. The period from 1982 to 2000, or even 2007, in US equities, is probably
> a once in a millenium event. I doubt that it will be repeated -- I am
> certain it will not be repeated in my lifetime. Long-only, buy-and-hold
> systems did very well during that period. At the risk of being flip, it was
> easy to confuse brains with a bull market.
>
> The price earning ratio for the S&P 500 companies is now about 120 (per
> S&P's numbers on their website). The price earnings ratio at equity market
> bottoms is single digits, and in normal times is mid-teens. Excursions to
> an extreme tend to correct, and the correction seldom stops at the mean --
> it makes an excursion to the other extreme. Can we expect fundamentals to
> improve enough so that earnings increase by a factor of 10? Or will prices
> decrease by a factor of 10? Or some of each? I expect that one or more of
> those events will be taking place in the next twelve months. In any event,
> 10 percent per year annual increases in stock prices seem extremely
> unlikely. We need trading systems that do not depend on a rising market.
>
> I suggest that we read Tharp and Vince, use good modeling and simulation
> technique, test the significance of our out-of-sample results, and look for
> systems that have truly out-of-sample t-test scores of 2.0 or more -- they
> are hard enough to find -- forget 6.99. Then work on position sizing
> methods that use anti-martingale techniques (increase position size when
> winning, reduce it when losing) to determine what position size will produce
> the goal the trader wants while keeping the probability of bankruptcy at a
> low enough level so he or she is confortable trading.
>
> There are two very distinct components to this:
> 1. Develop good trading systems that have positive expectancy, limiting the
> losing trades to small amounts, and trade frequently.
> 2. Apply aggressive position sizing when winning.
>
>
> Thanks for listening,
> Howard
>
>
>
>
>
>
>
> On Thu, Oct 8, 2009 at 11:15 PM, Mike <sfclimbers@xxx> wrote:
>
> >
> >
> > Hi Howard,
> >
> > I'm a bit confused by your remarks.
> >
> > 1. You have said that Definitive Guide To Positoin Sizing is an excellent
> > book.
> > 2. You have said that SQN is a t-test (of expectancy?).
> > 3. You have said that a t-test of expectancy is functionally equivalent to
> > CAR.
> > 4. You have said that CAR is a poor fitness function.
> >
> > Given that the book repeatedly promotes usage of SQN as a comparator (i.e.
> > a fitness function) with which to measure two systems, how do you reconcile
> > point 1 with points 2-4?
> >
> > Just to pin it down without any indirection, are you saying that you
> > consider SQN as poor a fitness function as CAR? Or, have I misunderstood
> > something you've said?
> >
> > Thanks,
> >
> >
> > Mike
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%40yahoogroups.com>, Howard B
> > <howardbandy@> wrote:
> > >
> > > Hi Mike --
> > >
> > > CAR is compound annual rate of return. Expectancy is the percentage (or
> > > dollar amount, but not for this discussion) gain on the average trade.
> > The
> > > final value of the trading account, Terminal Relative Wealth in some
> > > descriptions, is: (1 + expectancy) raised to the power of the number of
> > > trades. When there are a few large trades included, this relationship is
> > > slightly different, but not enough to worry this discussion. CAR is: (1 +
> > > annual gain) raised to the power of the number of years. These are both
> > > based on geometric means. Aren't they the same, or so close that they act
> > > the same when used as an objective function?
> > >
> > > Van Tharp defines System Quality Number on page 28 of "Definitive Guide
> > to
> > > Position Sizing." SQN = (expectancy / standard deviation) times
> > squareroot
> > > of number of trades. SQN decreases when a median trade is replaced by
> > > either a large win or a large loss. It is possible to redefine the metric
> > > so that large wins are not penalized, such as by using the semi-deviation
> > > instead of the standard deviation as the denominator. But without making
> > > that change, outliers, both good and bad, do affect SQN by reducing it.
> > >
> > > What happens with open (or potentially open) trades at the boundaries of
> > > walk forward periods is difficult to handle in both theory and practice.
> > > Tomasz' implementation is to close all open trades at the end of each WF
> > > period; and go into each WF period flat, not taking a new position until
> > > there is a new signal. This creates a potentially serious distortion of
> > > results when trades are typically held a long time (a large proportion of
> > a
> > > WF period) or when there are a few large trades that comprise the
> > majority
> > > of a system's profit or loss.
> > >
> > > I think this leads to a conclusion that we both agree on -- high quality
> > > trading systems that can benefit from position sizing are based on high
> > > frequency trading with very careful control over losses, and even control
> > > over gains.
> > >
> > > Thanks for listening,
> > > Howard
> > >
> > >
> > > On Wed, Oct 7, 2009 at 2:36 PM, Mike <sfclimbers@> wrote:
> > >
> > > >
> > > >
> > > > Howard,
> > > >
> > > > Assuming that SQN is the t-test for expectancy, then optimizing on the
> > > > t-test of expectancy (i.e. SQN) is not the same as optimizing on CAR.
> > > >
> > > > The primary reason that CAR is a poor target for optimization is that
> > > > outliers can significantly improve the calculation. The exact opposite
> > is
> > > > true for SQN.
> > > >
> > > > SQN rewards consistency and punishes outliers. Consistent winners with
> > a
> > > > few large wins will improve CAR but hurt SQN, resulting in different
> > > > parameter combinations being selected during an optimization.
> > > >
> > > > As for writing a custom method, AmiBroker's stats are calculated based
> > on
> > > > the assumption that all open trades are closed out at the backtest
> > boundary
> > > > date. Many open trades, or even just a few large open trades, can skew
> > these
> > > > values.
> > > >
> > > > For high frequency strategies or strategies using heavy position
> > sizing,
> > > > creating a custom function is the only way to get reliable
> > measurements.
> > > >
> > > > Mike
> > > >
> > > >
> > > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%40yahoogroups.com><amibroker%
> > 40yahoogroups.com>, Howard B
> >
> > > > <howardbandy@> wrote:
> > > > >
> > > > > Greetings all --
> > > > >
> > > > > There has been a lot of activity on this thread. I'll not respond to
> > each
> > > > > point individually, but will make a couple of general comments.
> > > > >
> > > > > I know David Aronson, speak with him regularly, and collaborate with
> > him
> > > > on
> > > > > projects. I have a copy of his book, "Evidence-Based Technical
> > Analysis."
> > > > > His book is excellent and I highly recommend it. I think David and I
> > are
> > > > in
> > > > > pretty close agreement on most of the modeling, simulation, testing,
> > and
> > > > > validation issues.
> > > > >
> > > > > I have spoken with Robert Pardo and have exchanged several emails and
> > > > forum
> > > > > postings with him. I think his earlier book was very good,
> > particularly
> > > > at
> > > > > the time it was published. And his more recent book is not quite up
> > to
> > > > > those standards. There are several important areas he did not cover
> > and
> > > > > several areas where I see things considerably differently than
> > Robert.
> > > > >
> > > > > I have spoken with and exchanged emails with Van Tharp, and I have
> > copies
> > > > of
> > > > > his books "Trade Your Way to Financial Freedom" and "Definitive Guide
> > to
> > > > > Position Sizing." Both are excellent, and I recommend them both
> > highly.
> > > > Be
> > > > > sure to get the second edition of Trade Your Way to Financial Freedom
> > --
> > > > it
> > > > > has some important corrections and clarifications.
> > > > >
> > > > > Permit me a short rant on my soapbox. I really dislike it when people
> > > > claim
> > > > > ownership of common terms. Tom DeMark, Robert Pardo, Van Tharp, and
> > > > others
> > > > > put Service Mark symbols on terms that they think are unique to them,
> > but
> > > > > are not. I appreciate Tharp's enthusiasm over what he calls System
> > > > Quality
> > > > > Number, but I wish he would not put the Service Mark symbol next to
> > every
> > > > > occurrence of it. And trying to Service Mark the term Position Sizing
> > is
> > > > > like a dietician service marking "calorie counting." Robert Pardo
> > claims
> > > > > "Walk Forward." I used exactly that term describing exactly that
> > process
> > > > in
> > > > > research papers I delivered at conferences in the late 1960s. The
> > mark
> > > > has
> > > > > been registered, not by Robert, but by a company I used to work for
> > and
> > > > with
> > > > > which Robert was not associated, over my strong objection. End of
> > rant.
> > > > >
> > > > > System quality number is equivalent to t-test. Systems with SQNs
> > above 2
> > > > > work well for exactly the same reasons that systems with t-test
> > scores
> > > > above
> > > > > 2 work well. In fact, it is possible to create a custom objective
> > > > function
> > > > > that Is the t-test and use it for optimization. Attendees at my
> > workshops
> > > > > in Melbourne later this month will see that demonstrated. Optimizing
> > for
> > > > > the t-test of expectancy is equivalent to optimizing for CAR, so
> > don't
> > > > > bother creating the custom function unless you have a better
> > candidate
> > > > for
> > > > > your objective function than CAR.
> > > > >
> > > > > Back to the topic at hand -----
> > > > >
> > > > > There is No rule of thumb to determine how long the in-sample period
> > > > should
> > > > > be. The Only way to determine that is by testing the model and the
> > data
> > > > > together. And be prepared for that length to change over time. Some
> > > > > writers suggest a relationship between the number of free parameters
> > and
> > > > the
> > > > > number of data points, or some proportional division of the available
> > > > data.
> > > > > Those techniques do work on industrial time-series data which is
> > usually
> > > > > stationary, but they do not work on financial time-series data which
> > is
> > > > > non-stationary and changes as trading systems become better at
> > extracting
> > > > > inefficiencies from it.
> > > > >
> > > > > No matter how good the in-sample results look, no matter how high the
> > > > t-test
> > > > > score is, no matter how many closed trades are represented --
> > in-sample
> > > > > results have no value in estimating the future performance of the
> > system.
> > > > > None. The only information you have that gives any indication of
> > future
> > > > > performance are the out-of-sample results from testing on data that
> > was
> > > > > never used at all -- not even once -- during system development.
> > > > >
> > > > > Tomorrow is out-of-sample. The only way to prepare for real-money
> > trading
> > > > > tomorrow is to be rigorous during the system testing and validation
> > > > > process. Anything less will overestimate the probability of success.
> > > > >
> > > > > Thanks for listening,
> > > > > Howard
> > > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|