|
PureBytes Links
Trading Reference Links
|
Mike -
Understood. That is why I thought that Howard might want to comment on his perspective of the trade-offs. But the bottom line is that for intermediate term trading systems, you have to do this to get an accurate equity representation.
If Fred Tonetti is reading this, he'll probably remember that we sat in a hotel bar in Tampa at a conference years ago and drew on quite a few napkins talking about the correct way to handle this.
For a single ticker, it is relatively straightforward. You have to capture the trades for each OOS CBT pass. That complete trade file is then used to generate a full OOS equity pass and the stat's are then accurate.
BTW, this was what was behind something I posted a year or so ago, that implemented the detection of the IS and OOS phases - since it is not in the Status("ActionEx") codes.
Solving it for a portfolio of tickers is a fair amount tougher...
--- In amibroker@xxxxxxxxxxxxxxx, "Mike" <sfclimbers@xxx> wrote:
>
> Bruce,
>
> Unfortunately, what you are suggesting may negatively impact the solution for working around skewed summary metrics. Using state form will cause the latter portion a trade (that was artificially closed out in the previous period) to appear as a complete trade, starting and finishing, in the current period of the walk forward.
>
> Custom metrics based on closed trades only (to avoid partial trades skewing the metrics) would suffer from these opening boundary trades, since they would now appear in the list of closed trades. This might prove just as problematic to fitness function calculations as the closing boundary open trades.
>
> If the opening boundary effects and the closing boundary effects roughly canceled each other out, we would have the best of both worlds; We would capture a trade's entire lifespan, and we could rely on the summary statistics as is. But, we would have to take a pretty close look to see if that was in fact the case for a given strategy.
>
> As always, it may end up a trade off; Capturing the complete life of a trade vs. precision fitness function calculations.
>
> Mike
>
> --- In amibroker@xxxxxxxxxxxxxxx, "Bruce" <brucer@> wrote:
> >
> > All -
> >
> > I've discussed the following with Howard and am posting mainly for
> > others on the board. He might want to comment on the trade-offs. But,
> > there is an easy way in practice to deal with one of the drawbacks that
> > he mentioned when he said -
> >
> > "What happens with open (or potentially open) trades at the boundaries
> > of walk forward periods is difficult to handle in both theory and
> > practice. Tomasz' implementation is to close all open trades at the end
> > of each WF period; and go into each WF period flat, not taking anew
> > position until there is a new signal. "
> >
> > Closing all trades at the end of a WF period is a compromise of sort to
> > avoid some messy analysis at the boundary. BUT, it is fairly easy to
> > handle going into each WF period and not taking a new position until
> > there is a signal.
> >
> > If the signals are in impulse form (generated with Cross()), then the
> > system will be flat in the WF period until the first signal. But, if
> > the signals are put in state form, the trade in progress will be taken
> > at the beginning of the period. To do this, all that is necessary is
> > to place the following code at the end of the program -
> >
> > Buy = Flip( Buy, Sell);
> > Sell = Not Buy;
> >
> > Finally, a brief example to illustrate. Say you have a simple timing
> > system and the walk-forward analysis is yearly. Assume the walk-forward
> > period for 2005 has a Buy on Nov. 1. In processing that year, AB sill
> > generate a Buy on Nov. 1 and a Sell on Dec.31. Now if the walk-forward
> > for the year 2006 has a Sell on Feb. 1, you would like the system to be
> > on a Buy starting Jan. 2. - even though the individual trade metrics
> > will be skewed, the equity curve and metrics will be almost correct (in
> > certain cases there is one boundary effect flat day).
> >
> > If the signals were left in impulse form, there would not be a Buy for
> > that first month of 2006. If you put the signals in state form, there
> > will be.
> >
> > -- BruceR
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, Howard B <howardbandy@> wrote:
> > >
> > > Hi Mike --
> > >
> > > CAR is compound annual rate of return. Expectancy is the percentage
> > (or
> > > dollar amount, but not for this discussion) gain on the average trade.
> > The
> > > final value of the trading account, Terminal Relative Wealth in some
> > > descriptions, is: (1 + expectancy) raised to the power of the number
> > of
> > > trades. When there are a few large trades included, this relationship
> > is
> > > slightly different, but not enough to worry this discussion. CAR is:
> > (1 +
> > > annual gain) raised to the power of the number of years. These are
> > both
> > > based on geometric means. Aren't they the same, or so close that they
> > act
> > > the same when used as an objective function?
> > >
> > > Van Tharp defines System Quality Number on page 28 of "Definitive
> > Guide to
> > > Position Sizing." SQN = (expectancy / standard deviation) times
> > squareroot
> > > of number of trades. SQN decreases when a median trade is replaced by
> > > either a large win or a large loss. It is possible to redefine the
> > metric
> > > so that large wins are not penalized, such as by using the
> > semi-deviation
> > > instead of the standard deviation as the denominator. But without
> > making
> > > that change, outliers, both good and bad, do affect SQN by reducing
> > it.
> > >
> > > What happens with open (or potentially open) trades at the boundaries
> > of
> > > walk forward periods is difficult to handle in both theory and
> > practice.
> > > Tomasz' implementation is to close all open trades at the end of each
> > WF
> > > period; and go into each WF period flat, not taking a new position
> > until
> > > there is a new signal. This creates a potentially serious distortion
> > of
> > > results when trades are typically held a long time (a large proportion
> > of a
> > > WF period) or when there are a few large trades that comprise the
> > majority
> > > of a system's profit or loss.
> > >
> > > I think this leads to a conclusion that we both agree on -- high
> > quality
> > > trading systems that can benefit from position sizing are based on
> > high
> > > frequency trading with very careful control over losses, and even
> > control
> > > over gains.
> > >
> > > Thanks for listening,
> > > Howard
> > >
> > >
> > > On Wed, Oct 7, 2009 at 2:36 PM, Mike sfclimbers@ wrote:
> > >
> > > >
> > > >
> > > > Howard,
> > > >
> > > > Assuming that SQN is the t-test for expectancy, then optimizing on
> > the
> > > > t-test of expectancy (i.e. SQN) is not the same as optimizing on
> > CAR.
> > > >
> > > > The primary reason that CAR is a poor target for optimization is
> > that
> > > > outliers can significantly improve the calculation. The exact
> > opposite is
> > > > true for SQN.
> > > >
> > > > SQN rewards consistency and punishes outliers. Consistent winners
> > with a
> > > > few large wins will improve CAR but hurt SQN, resulting in different
> > > > parameter combinations being selected during an optimization.
> > > >
> > > > As for writing a custom method, AmiBroker's stats are calculated
> > based on
> > > > the assumption that all open trades are closed out at the backtest
> > boundary
> > > > date. Many open trades, or even just a few large open trades, can
> > skew these
> > > > values.
> > > >
> > > > For high frequency strategies or strategies using heavy position
> > sizing,
> > > > creating a custom function is the only way to get reliable
> > measurements.
> > > >
> > > > Mike
> > > >
> > > >
> > > > --- In amibroker@xxxxxxxxxxxxxxx <amibroker%40yahoogroups.com>,
> > Howard B
> > > > howardbandy@ wrote:
> > > > >
> > > > > Greetings all --
> > > > >
> > > > > There has been a lot of activity on this thread. I'll not respond
> > to each
> > > > > point individually, but will make a couple of general comments.
> > > > >
> > > > > I know David Aronson, speak with him regularly, and collaborate
> > with him
> > > > on
> > > > > projects. I have a copy of his book, "Evidence-Based Technical
> > Analysis."
> > > > > His book is excellent and I highly recommend it. I think David and
> > I are
> > > > in
> > > > > pretty close agreement on most of the modeling, simulation,
> > testing, and
> > > > > validation issues.
> > > > >
> > > > > I have spoken with Robert Pardo and have exchanged several emails
> > and
> > > > forum
> > > > > postings with him. I think his earlier book was very good,
> > particularly
> > > > at
> > > > > the time it was published. And his more recent book is not quite
> > up to
> > > > > those standards. There are several important areas he did not
> > cover and
> > > > > several areas where I see things considerably differently than
> > Robert.
> > > > >
> > > > > I have spoken with and exchanged emails with Van Tharp, and I have
> > copies
> > > > of
> > > > > his books "Trade Your Way to Financial Freedom" and "Definitive
> > Guide to
> > > > > Position Sizing." Both are excellent, and I recommend them both
> > highly.
> > > > Be
> > > > > sure to get the second edition of Trade Your Way to Financial
> > Freedom --
> > > > it
> > > > > has some important corrections and clarifications.
> > > > >
> > > > > Permit me a short rant on my soapbox. I really dislike it when
> > people
> > > > claim
> > > > > ownership of common terms. Tom DeMark, Robert Pardo, Van Tharp,
> > and
> > > > others
> > > > > put Service Mark symbols on terms that they think are unique to
> > them, but
> > > > > are not. I appreciate Tharp's enthusiasm over what he calls System
> > > > Quality
> > > > > Number, but I wish he would not put the Service Mark symbol next
> > to every
> > > > > occurrence of it. And trying to Service Mark the term Position
> > Sizing is
> > > > > like a dietician service marking "calorie counting." Robert Pardo
> > claims
> > > > > "Walk Forward." I used exactly that term describing exactly that
> > process
> > > > in
> > > > > research papers I delivered at conferences in the late 1960s. The
> > mark
> > > > has
> > > > > been registered, not by Robert, but by a company I used to work
> > for and
> > > > with
> > > > > which Robert was not associated, over my strong objection. End of
> > rant.
> > > > >
> > > > > System quality number is equivalent to t-test. Systems with SQNs
> > above 2
> > > > > work well for exactly the same reasons that systems with t-test
> > scores
> > > > above
> > > > > 2 work well. In fact, it is possible to create a custom objective
> > > > function
> > > > > that Is the t-test and use it for optimization. Attendees at my
> > workshops
> > > > > in Melbourne later this month will see that demonstrated.
> > Optimizing for
> > > > > the t-test of expectancy is equivalent to optimizing for CAR, so
> > don't
> > > > > bother creating the custom function unless you have a better
> > candidate
> > > > for
> > > > > your objective function than CAR.
> > > > >
> > > > > Back to the topic at hand -----
> > > > >
> > > > > There is No rule of thumb to determine how long the in-sample
> > period
> > > > should
> > > > > be. The Only way to determine that is by testing the model and the
> > data
> > > > > together. And be prepared for that length to change over time.
> > Some
> > > > > writers suggest a relationship between the number of free
> > parameters and
> > > > the
> > > > > number of data points, or some proportional division of the
> > available
> > > > data.
> > > > > Those techniques do work on industrial time-series data which is
> > usually
> > > > > stationary, but they do not work on financial time-series data
> > which is
> > > > > non-stationary and changes as trading systems become better at
> > extracting
> > > > > inefficiencies from it.
> > > > >
> > > > > No matter how good the in-sample results look, no matter how high
> > the
> > > > t-test
> > > > > score is, no matter how many closed trades are represented --
> > in-sample
> > > > > results have no value in estimating the future performance of the
> > system.
> > > > > None. The only information you have that gives any indication of
> > future
> > > > > performance are the out-of-sample results from testing on data
> > that was
> > > > > never used at all -- not even once -- during system development.
> > > > >
> > > > > Tomorrow is out-of-sample. The only way to prepare for real-money
> > trading
> > > > > tomorrow is to be rigorous during the system testing and
> > validation
> > > > > process. Anything less will overestimate the probability of
> > success.
> > > > >
> > > > > Thanks for listening,
> > > > > Howard
> > > > >
> > > >
> > > >
> > > >
> > >
> >
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|