|
PureBytes Links
Trading Reference Links
|
Howard,
Thanks for the thorough reply.
When I first read the book, I was left with the distinct impression that Tharp was using SQN as a differentiator between the quality of systems, most notably from Table 3-11 on page 31 titled:
"Using the System Quality Number to Rate Your System Based On 100 Trades"
Flipping through again, I now notice hedging language to the effect that SQN alone might not be enough for some. That the combination of SQN and number of trades might be the better measure for others. Also, that the rating was more for suitability to position sizing, as opposed to overall system superiority.
However, I think that the two (suitability to position sizing and overall quality) are tightly linked. I found the subject, as a whole, to be revealing when applied to my own strategy. Using SQN as a fitness function has shown itself to be useful, particularly when weighted for number of trades.
You might have missed it. But, he actually does come right out and say that SQN is a t-test. See "Statistical Assumptions of Using This Material" on page 33, where he says that it's a t-test comparing a sample against an assumed mean of zero.
Note, however, that almost every example, including the one you quoted in your reply, has an error in it. If you enter the trades in a spreadsheet, you will see that he has miscalculated standard deviation, resulting in understated SQN values.
Anyone interested in seeing the corrected examples can refer to the simulator found here:
http://www.afltools.com/positionsize.html
I wrote the simulator in order to apply the teachings of the book to my own trade results. I found the exercise to be very educational, albeit frustrating when the simulator did not line up with the published examples. Through an email exchange with the Van Tharp Institute, it was confirmed that, yes, there were errors in the examples, as enumerated at the link above.
I particularly liked seeing the different position sizing based on the different notions of "optimal". This principle is readily seen just by running the simulator on the samples, even without our own trade results.
Mike
--- In amibroker@xxxxxxxxxxxxxxx, Howard B <howardbandy@xxx> wrote:
>
> Hi Mike, and all --
>
> There is a lot to like in Tharp's book, Definitive Guide to Position
> Sizing.
>
> But it appears to me that he developed the concept of system quality number
> without using it as an objective function used in testing and validating
> trading systems. He describes several idealized trading systems that have a
> range of SQNs -- from negative to about 7. One on page 38 is described as:
> Mean expectancy: 3.42
> Standard deviation: 4.89
> Win percentage: 90
> Win/Loss Ratio: 2.35
> Number Trades: 100
> SQN: 6.99
>
> Plugging those numbers into the formula for the t-test, which is
> (mean/stdev) * sqrt(N)
> we get (3.42/4.89)*10 which equals 6.99 -- the same as the SQN.
>
> While he does not call SQN t-test, for each of the examples I worked out,
> the two are the same.
>
> So, one of my points is that SQN is equivalent to t-test where the
> hypothesis that is being tested is "is expectancy greater than 0?"
>
> I like the idea that the statistical test is incorporated into the metric.
> I have been using that concept for some time, and I think it works well. I
> wish he had not called it SQN, and I wish he had not put a Service Mark
> next to it every time it appears in his book. But that concept is good.
>
> I seriously doubt that he, or anyone else, has a series of truly
> out-of-sample closed trades where the t-test of expectancy is 7. It would
> take only a few weeks to turn $10,000 into enough to buy Manhattan with a
> system like that. Which goes to my point that I doubt that any of the
> idealized trading system he describes with large SQNs reflect real trading
> systems. Idealized, yes. In-sample, perhaps, but I doubt it.
> Out-of-sample, almost impossible and I very seriously doubt it.
>
> As a basis for my comment, look at a table of t-test percentile values. I
> use a rule of thumb that the t-test should be above about 2.0, just for its
> ease of computation. With n=10 trades (9 degrees of freedom), 95%
> confidence comes with t-statistic of 1.83. With n=120, 99.5% confidence
> comes with a t-statistic of 2.62.
>
> For sample sizes greater than about 60, the t-distribution and the normal
> distribution are essentially identical. Something 3 standard deviations
> away from the mean happens much much less than 1% of the time. Something 7
> standard deviations away is so rare as to be essentially impossible on our
> time scale. It is nice to see the fantasy equity curve, but it will never
> happen.
>
> If expectancy works as an objective function for a particular trader and
> system, then by all means, use it. As the characteristics of the trade
> returns change, the advantages and disadvantages of using expectancy change.
>
> All of that said, I think Tharp's book is good. It brings out many
> excellent points about the importance of positions sizing. Read it in
> conjunction with Ralph Vince's "The Leverage Space Trading Model." My own
> research and experience is in agreement with both of those.
>
> There is an implication -- one that many people will not find comfortable
> and will argue strongly against. It is this: The only way to generate
> large increases in equity safely is to have a trading system that trades
> frequently, holds a short period of time, and carefully limits losing
> trades.
>
> Yes, there are examples of people who have made a lot of money trading
> infrequently and holding long periods. My contention is that
> 1. we are hearing from those who were successful. Not many people who
> tried to do this succeeded. Most who tried and failed have not remained in
> the trading community and we are not hearing their stories of failure.
> 2. The period from 1982 to 2000, or even 2007, in US equities, is probably
> a once in a millenium event. I doubt that it will be repeated -- I am
> certain it will not be repeated in my lifetime. Long-only, buy-and-hold
> systems did very well during that period. At the risk of being flip, it was
> easy to confuse brains with a bull market.
>
> The price earning ratio for the S&P 500 companies is now about 120 (per
> S&P's numbers on their website). The price earnings ratio at equity market
> bottoms is single digits, and in normal times is mid-teens. Excursions to
> an extreme tend to correct, and the correction seldom stops at the mean --
> it makes an excursion to the other extreme. Can we expect fundamentals to
> improve enough so that earnings increase by a factor of 10? Or will prices
> decrease by a factor of 10? Or some of each? I expect that one or more of
> those events will be taking place in the next twelve months. In any event,
> 10 percent per year annual increases in stock prices seem extremely
> unlikely. We need trading systems that do not depend on a rising market.
>
> I suggest that we read Tharp and Vince, use good modeling and simulation
> technique, test the significance of our out-of-sample results, and look for
> systems that have truly out-of-sample t-test scores of 2.0 or more -- they
> are hard enough to find -- forget 6.99. Then work on position sizing
> methods that use anti-martingale techniques (increase position size when
> winning, reduce it when losing) to determine what position size will produce
> the goal the trader wants while keeping the probability of bankruptcy at a
> low enough level so he or she is confortable trading.
>
> There are two very distinct components to this:
> 1. Develop good trading systems that have positive expectancy, limiting the
> losing trades to small amounts, and trade frequently.
> 2. Apply aggressive position sizing when winning.
>
>
> Thanks for listening,
> Howard
>
>
>
>
>
>
>
> On Thu, Oct 8, 2009 at 11:15 PM, Mike <sfclimbers@xxx> wrote:
>
> >
> >
> > Hi Howard,
> >
> > I'm a bit confused by your remarks.
> >
> > 1. You have said that Definitive Guide To Positoin Sizing is an excellent
> > book.
> > 2. You have said that SQN is a t-test (of expectancy?).
> > 3. You have said that a t-test of expectancy is functionally equivalent to
> > CAR.
> > 4. You have said that CAR is a poor fitness function.
> >
> > Given that the book repeatedly promotes usage of SQN as a comparator (i.e.
> > a fitness function) with which to measure two systems, how do you reconcile
> > point 1 with points 2-4?
> >
> > Just to pin it down without any indirection, are you saying that you
> > consider SQN as poor a fitness function as CAR? Or, have I misunderstood
> > something you've said?
> >
> > Thanks,
> >
> >
> > Mike
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%40yahoogroups.com>, Howard B
> > <howardbandy@> wrote:
> > >
> > > Hi Mike --
> > >
> > > CAR is compound annual rate of return. Expectancy is the percentage (or
> > > dollar amount, but not for this discussion) gain on the average trade.
> > The
> > > final value of the trading account, Terminal Relative Wealth in some
> > > descriptions, is: (1 + expectancy) raised to the power of the number of
> > > trades. When there are a few large trades included, this relationship is
> > > slightly different, but not enough to worry this discussion. CAR is: (1 +
> > > annual gain) raised to the power of the number of years. These are both
> > > based on geometric means. Aren't they the same, or so close that they act
> > > the same when used as an objective function?
> > >
> > > Van Tharp defines System Quality Number on page 28 of "Definitive Guide
> > to
> > > Position Sizing." SQN = (expectancy / standard deviation) times
> > squareroot
> > > of number of trades. SQN decreases when a median trade is replaced by
> > > either a large win or a large loss. It is possible to redefine the metric
> > > so that large wins are not penalized, such as by using the semi-deviation
> > > instead of the standard deviation as the denominator. But without making
> > > that change, outliers, both good and bad, do affect SQN by reducing it.
> > >
> > > What happens with open (or potentially open) trades at the boundaries of
> > > walk forward periods is difficult to handle in both theory and practice.
> > > Tomasz' implementation is to close all open trades at the end of each WF
> > > period; and go into each WF period flat, not taking a new position until
> > > there is a new signal. This creates a potentially serious distortion of
> > > results when trades are typically held a long time (a large proportion of
> > a
> > > WF period) or when there are a few large trades that comprise the
> > majority
> > > of a system's profit or loss.
> > >
> > > I think this leads to a conclusion that we both agree on -- high quality
> > > trading systems that can benefit from position sizing are based on high
> > > frequency trading with very careful control over losses, and even control
> > > over gains.
> > >
> > > Thanks for listening,
> > > Howard
> > >
> > >
> > > On Wed, Oct 7, 2009 at 2:36 PM, Mike <sfclimbers@> wrote:
> > >
> > > >
> > > >
> > > > Howard,
> > > >
> > > > Assuming that SQN is the t-test for expectancy, then optimizing on the
> > > > t-test of expectancy (i.e. SQN) is not the same as optimizing on CAR.
> > > >
> > > > The primary reason that CAR is a poor target for optimization is that
> > > > outliers can significantly improve the calculation. The exact opposite
> > is
> > > > true for SQN.
> > > >
> > > > SQN rewards consistency and punishes outliers. Consistent winners with
> > a
> > > > few large wins will improve CAR but hurt SQN, resulting in different
> > > > parameter combinations being selected during an optimization.
> > > >
> > > > As for writing a custom method, AmiBroker's stats are calculated based
> > on
> > > > the assumption that all open trades are closed out at the backtest
> > boundary
> > > > date. Many open trades, or even just a few large open trades, can skew
> > these
> > > > values.
> > > >
> > > > For high frequency strategies or strategies using heavy position
> > sizing,
> > > > creating a custom function is the only way to get reliable
> > measurements.
> > > >
> > > > Mike
> > > >
> > > >
> > > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%40yahoogroups.com><amibroker%
> > 40yahoogroups.com>, Howard B
> >
> > > > <howardbandy@> wrote:
> > > > >
> > > > > Greetings all --
> > > > >
> > > > > There has been a lot of activity on this thread. I'll not respond to
> > each
> > > > > point individually, but will make a couple of general comments.
> > > > >
> > > > > I know David Aronson, speak with him regularly, and collaborate with
> > him
> > > > on
> > > > > projects. I have a copy of his book, "Evidence-Based Technical
> > Analysis."
> > > > > His book is excellent and I highly recommend it. I think David and I
> > are
> > > > in
> > > > > pretty close agreement on most of the modeling, simulation, testing,
> > and
> > > > > validation issues.
> > > > >
> > > > > I have spoken with Robert Pardo and have exchanged several emails and
> > > > forum
> > > > > postings with him. I think his earlier book was very good,
> > particularly
> > > > at
> > > > > the time it was published. And his more recent book is not quite up
> > to
> > > > > those standards. There are several important areas he did not cover
> > and
> > > > > several areas where I see things considerably differently than
> > Robert.
> > > > >
> > > > > I have spoken with and exchanged emails with Van Tharp, and I have
> > copies
> > > > of
> > > > > his books "Trade Your Way to Financial Freedom" and "Definitive Guide
> > to
> > > > > Position Sizing." Both are excellent, and I recommend them both
> > highly.
> > > > Be
> > > > > sure to get the second edition of Trade Your Way to Financial Freedom
> > --
> > > > it
> > > > > has some important corrections and clarifications.
> > > > >
> > > > > Permit me a short rant on my soapbox. I really dislike it when people
> > > > claim
> > > > > ownership of common terms. Tom DeMark, Robert Pardo, Van Tharp, and
> > > > others
> > > > > put Service Mark symbols on terms that they think are unique to them,
> > but
> > > > > are not. I appreciate Tharp's enthusiasm over what he calls System
> > > > Quality
> > > > > Number, but I wish he would not put the Service Mark symbol next to
> > every
> > > > > occurrence of it. And trying to Service Mark the term Position Sizing
> > is
> > > > > like a dietician service marking "calorie counting." Robert Pardo
> > claims
> > > > > "Walk Forward." I used exactly that term describing exactly that
> > process
> > > > in
> > > > > research papers I delivered at conferences in the late 1960s. The
> > mark
> > > > has
> > > > > been registered, not by Robert, but by a company I used to work for
> > and
> > > > with
> > > > > which Robert was not associated, over my strong objection. End of
> > rant.
> > > > >
> > > > > System quality number is equivalent to t-test. Systems with SQNs
> > above 2
> > > > > work well for exactly the same reasons that systems with t-test
> > scores
> > > > above
> > > > > 2 work well. In fact, it is possible to create a custom objective
> > > > function
> > > > > that Is the t-test and use it for optimization. Attendees at my
> > workshops
> > > > > in Melbourne later this month will see that demonstrated. Optimizing
> > for
> > > > > the t-test of expectancy is equivalent to optimizing for CAR, so
> > don't
> > > > > bother creating the custom function unless you have a better
> > candidate
> > > > for
> > > > > your objective function than CAR.
> > > > >
> > > > > Back to the topic at hand -----
> > > > >
> > > > > There is No rule of thumb to determine how long the in-sample period
> > > > should
> > > > > be. The Only way to determine that is by testing the model and the
> > data
> > > > > together. And be prepared for that length to change over time. Some
> > > > > writers suggest a relationship between the number of free parameters
> > and
> > > > the
> > > > > number of data points, or some proportional division of the available
> > > > data.
> > > > > Those techniques do work on industrial time-series data which is
> > usually
> > > > > stationary, but they do not work on financial time-series data which
> > is
> > > > > non-stationary and changes as trading systems become better at
> > extracting
> > > > > inefficiencies from it.
> > > > >
> > > > > No matter how good the in-sample results look, no matter how high the
> > > > t-test
> > > > > score is, no matter how many closed trades are represented --
> > in-sample
> > > > > results have no value in estimating the future performance of the
> > system.
> > > > > None. The only information you have that gives any indication of
> > future
> > > > > performance are the out-of-sample results from testing on data that
> > was
> > > > > never used at all -- not even once -- during system development.
> > > > >
> > > > > Tomorrow is out-of-sample. The only way to prepare for real-money
> > trading
> > > > > tomorrow is to be rigorous during the system testing and validation
> > > > > process. Anything less will overestimate the probability of success.
> > > > >
> > > > > Thanks for listening,
> > > > > Howard
> > > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|