Re: [amibroker] Re: What is best statistic for straightness of equity curve?, AmiBroker Email List Archive

I like R**2... a bit of programming, but can be done...

Kevin Campbell

In a message dated 3/13/2008 7:15:09 PM Central Daylight Time, brian_z111@xxxxxxxxx writes:

BTW

That is a slightly different discussion to the one on deciding which
ObjectiveFunction is superior (relatively speaking) but if we don't
understand the first point what basis do we have for making a
decision on ObjectiveFunctions?

Where I am at is to dig in further on the mechanics, with the view to
checking the value of our current thinking. The eventual OF that
comes out of that research may well surprize.

brian_z

--- In amibroker@xxxxxxxxxxxxxxx, "brian_z111" <brian_z111@xxx> wrote:
>
> I agree with Howard's (past) comments that the best metric is the
OOS
> metric (that is for those who have used optimization to design the
> system) or better still, several OOS metrics (if we have the data).
>
> The speculative (at this stage) point that I am introducing into
the
> discussion is that foward looking performance can be estimated from
> the root causes (mechanics of the trading system).
>
> By continually focussing on the unknown future we are chasing
> phantasms. While we try to catch one others are popping up
everywhere
> (just like a horror movie).
>
> Better off to find the mother ship, and understand the spawning
> process, if we are to have any hope of dealing with the offspring.
>
> brian_z
>
>
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, "Paul Ho" <paultsho@> wrote:
> >
> > This particular shortcoming of Sharpe ratio as mentioned by
Howard
> has been
> > well flaged by many books. and It make sense when one is
comparing
> PAST
> > performance from one fund manger to another, or from one system
to
> another.
> > However, when one is comparing forward looking performance, such
as
> when one
> > is developing new systems or evaulating new variations of an
> existing
> > system. Then IMHO this criticism is a little unjustified. Reason:
> If there
> > are an equity curve in front of me, one that is with a occasional
> surge of
> > profit (positive deviation) followed by a relatively flat patch.
I
> wouldn't
> > know with a lot of confidence I'm go to experience a flat patch or
> > continuing surge if I trade this system in the future. I have
seen
> a number
> > of systems that have a very quick rise in patches during backtest
> and
> > optimisation, but basically flat during forward testing. If I
have
> a choice,
> > I would prefer a lower return but with less deviation (both
> positive and
> > negative) when I'm developing new system because I'm more
confident
> that it
> > will generate a regular profit for me. I must confess I am a
short
> term
> > trader, my trades last for hours to days. I can apprecriate that
> long term
> > traders, those with trades lasting weeks to years, might have a
> different
> > psychology and can withstand large period of flat patches to wait
> for the
> > big one. Of course, once I have started using a system, I'm all
for
> positive
> > surprises.
> > I personally think the biggest drawback of Sharpe ratio lies with
> the fact
> > that the straightness of an equity curve cannot be adequately
> described by a
> > single Sharpe Ratio, because vastly different equity curves
shares
> similar
> > ratio numbers. A series of Sharpe Ratios measured periodically is
a
> better
> > guide. Tuschar Chande even went as far as suggesting measuring
> a "Sharpe
> > Ratio" over the series of Sharpe Ratio, I think this has merit.
> >
> >
> > _____
> >
> > From: amibroker@xxxxxxxxxxxxxxx
[mailto:amibroker@xxxxxxxxxxxxxxx]
> On Behalf
> > Of Dennis Brown
> > Sent: Friday, 14 March 2008 4:46 AM
> > To: amibroker@xxxxxxxxxxxxxxx
> > Subject: Re: [amibroker] Re: What is best statistic for
> straightness of
> > equity curve?
> >
> >
> >
> > Howard,
> >
> >
> > You make an excellent point. The metrics used to evaluate a
system
> needs to
> > take into consideration the normal "character" of the trading
> systems basic
> > methodology.
> >
> > For instance my system takes small profits and losses many times
a
> day. It
> > is not biased for long or short. It does not hold overnight, It
> only trades
> > broad market futures. It does not compound equity. It is
goodness
> be able
> > to take a consistent draw from a fixed account size.
> >
> > This means that my system will be subject to very different
market
> forces
> > than a system that swing trades stocks for a week or two, and is
> subject to
> > overnight gaps, company earnings announcements, dividends,
interest
> rates
> > (on margin accounts), and other unpredictable events.
> >
> > My system will perform with a much smoother equity curve just
> because of the
> > way it is defined. Commissions and Bid/Ask spreads are the main
> hurdles to
> > profitability, but they are constants.
> >
> > I have a much easier time telling if my system is robust.
> >
> > Best regards,
> > Dennis
> >
> >
> > On Mar 13, 2008, at 1:01 PM, Howard B wrote:
> >
> >
> >
> > Greetings all --
> >
> > Professional money managers are sometimes evaluated based on the
> Sharpe
> > Ratio of their performance, so it has some value. But, in my
> research, I
> > have not found Sharpe Ratio to be a very good metric for use when
> developing
> > systems. Yes, higher Sharpe Ratios will have smaller standard
> deviations
> > than lower Sharpe Ratios, but the standard deviation includes
both
> positive
> > and negative deviations. That is, it penalizes both positive and
> negative
> > performance. If you are designing trend following systems with
> long holding
> > periods, and looking for the infrequent large gains associated
with
> this
> > type of system, Sharpe Ratio penalizes these. When Sharpe Ratio
is
> used as
> > the objective function in an automated walk forward process,
systems
> > selected as the best in-sample often perform much less well out-
of-
> sample
> > than systems selected using K-Ratio, RRR, CAR/MDD, or UPI.
> >
> > Thanks for listening,
> > Howard
> >
> >
> >
> > On Wed, Mar 12, 2008 at 10:33 PM, Paul Ho <paultsho@xxxxxx
> > <mailto:paultsho@> com.au> wrote:
> >
> >
> >
> >
> > Time doesnt permit me to write a long post. But I think Jack
> Schwager in one
> > of his books povides a very good description of what You want.
> Tuschar
> > Chande also has insights.
> > One such parameter is the Sharpe ratio, but you need use it
slightly
> > differently. Firstly, take risk free return as zero, and you are
> obtaining
> > the ratio of mean return to std deviation. Secondly, calculated
> yearly
> > sharpe ratios and compare them from year to year.
> >
> >
> > _____
> >
> > From: amibroker@xxxxxxxxx <mailto:amibroker@xxxxxxxxxxxxxxx>
ps.com
> > [mailto:amibroker@xxxxxxxxx <mailto:amibroker@xxxxxxxxxxxxxxx>
> ps.com] On
> > Behalf OfDennis Brown
> > Sent: Thursday, 13 March 2008 12:24 PM
> > To: amibroker@xxxxxxxxx <mailto:amibroker@xxxxxxxxxxxxxxx> ps.com
> > Subject: Re: [amibroker] Re: What is best statistic for
> straightness of
> > equity curve?
> >
> >
> >
> > Brian,
> >
> > Thanks for your reply.
> >
> > My thinking is that the Std Error will work. I do not need to use
a
> > Log function on my equity curve, because I do not compound my
> results,
> > so they are linear. I also base my work on constant range bars,
so
> > that linearizes the curves even more. Profit potential can only
> come
> > from price movement. The smoothest and straightest equity curves
> come
> > from the most robust systems. Period. You can look at the curve
and
> > judge it, or find a number that is associated with this property.
> >
> > However, step functions get introduced into your nice trading
> system
> > from big news events that change the character of the markets
> > overnight, or in a minute during the day. I consider these things
> > that produce large quick drawdowns will be captured by a Maximum
> > Drawdown metric. The test period needs to have some of these big
> > events in it. The event may be too quick to affect a large
> > statistical function much, giving a false sense of goodness to
the
> > system. Or the perturbation might show up in a way that takes a
> great
> > system and makes the smoothness number look bad due to a one time
> > event. That is the challenge with a single number, so I will have
> to
> > experiment with the right weightings.
> >
> > That is why I say that the absolute judgement comes from
> examination
> > of the equity curve. The goodness numbers are just for ease of
> > relative comparisons of automated parameter optimization for
> candidate
> > systems. It is also nice to have a number or two as a future
point
> of
> > reference rather than going back over equity curves for every
> > comparison.
> >
> > Perhaps an FFT over the equity curve would generate an
interesting
> > signature in the period of the dominant frequency and I also need
> the
> > amplitude. I would have to look into this more, since I have not
> > tried this before.
> >
> > I will start out simple and see how better numbers compare to the
> > curves, then decide where to go from there.
> >
> > > (Why don't you just start posting some of your bits and pieces,
> like
> > > your new PlotShapes PDF, to the UKB - it is a live site - we
don't
> > > have to wait for the big bang moment to become an author - a
lot
> of
> > > my stuff is mundane and/or half finished, but it still has its
> uses).
> >
> > I am buried in work right now, so I wanted to gauge the value to
> > others of some of the things I could post on the UKB. I would
have
> to
> > fight for the time to figure out how to post and fiddle with with
> > formatting issues etc. If it were as easy as sending a PDF email
> > attachment here, I would have done it a month ago. It is the up
> front
> > time investment that is holding me back right now.
> >
> > When I get little feedback or interest from a post, I can't
> prioritize
> > the time to share more of what I am doing. If I were not so busy,
I
> > would do it anyway, but for now I need powerful justification to
> delay
> > some other important work to make time for it. This is not a
spare
> > time hobby for me, because I have no spare time right now. :-(
> >
> > I could use a teammate to get me through the initial stages.
> However,
> > I see that only a few have ventured as far as posting yet, so the
> > field is limited. I do all my content creation on a Mac, and keep
> my
> > virtual PC free of everything but AmiBroker and related support
> > programs. That is why I prefer to generate PDF content as it
works
> > everywhere. And I have exceptionally easy to use and powerful
tools
> > for generating them already.
> >
> > Best regards,
> > Dennis Brown
> >
> > On Mar 12, 2008, at 7:19 PM, brian_z111 wrote:
> >
> > > Dennis,
> > >
> > > So where is your thinking on this now?
> > >
> > >
> > > (I have been following and I am building to some possible input
> but
> > > since I don't understand logs and barely understand standard
> error I
> > > have had to go back to school - it takes quite a while for me
to
> get
> > > my head around that stuff and interpret it into trade talk).
> > >
> > > I have taken a different approach to evaluation (which is still
a
> > > work in progress) and based on that I am inclined to the view
that
> > > evaluations on one equity curve are on rather weak ground - IMO
> > > simulation is required for analysis of 'what counts most'.
> > >
> > > Also I am zeroing in on the root causes of equity curve
profiles
> and
> > > measuring smoothness of a curve is measuring the effect.
> > >
> > > BTW - your pane based analysis is very interesting but I think
> > > ultimately it might prove to have some limitations for good
> > > evaluation (but not if we correctly identify root causes - we
can
> > > just pick them out, add some mathematical antecedents and then
we
> > > will now the answers that simulation will give us and not need
to
> > > bother the processor - I have convinced myself that this is in
my
> > > grasps and later I hope the maths people will connect my
> conceptual
> > > does and bingo, we are there).
> > >
> > > However, I love your question and approach, so over to your
> immediate
> > > problem (I had it in mind to go to town on an equity curve
> smoothness
> > > metric anyway).
> > >
> > > K-ratio is actually a risk reward metric (is that what you
want)?
> > >
> > > It also (to me) gets a little mysterious in its workings
(Klestner
> > > doesn't fully explain one part of it - not from my, lay, point
of
> > > view anyway).
> > >
> > > I am still thinking about it.
> > >
> > > So far I would say StDev is out.
> > > StandardError will do exactly what you say you want to do (as
far
> as
> > > I can tell - once again the stats teachers seem to find it hard
to
> > > put it into trade talk - I see it explained in different ways in
> > > different books).
> > >
> > > I haven't reached a final conclusion but it seems most likely
> that if
> > > you use Standard Error on a compounded equity curve with the
LogN
> > > approach taken by Klestner you are there - no need to go past
> that -
> > > my reservation is based on the fact that I am not sure how to
> handle
> > > standardisation - I only work in relative % change - Klestner
> > > attempts to standardise the K-ratio - he had some trouble with
it
> to
> > > start out and had to add a standardising factor.
> > >
> > >> Everything I do is in indicator mode in realtime. I build all
my
> > >> metrics into my AFL. My charts and numbers always match and all
> > >> my
> > >> settings are stored in my Flexible Parameters scheme for
> different
> > >> test systems. It is a little different approach, but that is
one
> > >> of
> > >> the beauties of AB --that it allows a lot of flexibility of
doing
> > >> your
> > >> own thing if you don't want to use the built-in ways.
> > >
> > > Yes, all of my evaluation methods are home made, or adaptions of
> > > popular methods - works for me.
> > >
> > > As I said - if you want all of your evaluation in one window you
> > > might need a math formula to sum up the transition from root
> cause to
> > > simulation (I naively believe I have the beginning and end in
the
> bag
> > > and conceptually the middle formula seems attainable).
> > >
> > > (Why don't you just start posting some of your bits and pieces,
> like
> > > your new PlotShapes PDF, to the UKB - it is a live site - we
don't
> > > have to wait for the big bang moment to become an author - a
lot
> of
> > > my stuff is mundane and/or half finished, but it still has its
> uses).
> > >
> > > brian_z
> > >
> > >
> > > --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
> ps.com,
> > Dennis Brown <see3d@> wrote:
> > >>
> > >> Howard,
> > >>
> > >> Thanks for the input. I will investigate these some more.
> > >>
> > >> However, I do not use the built-in equity functions, or any of
> the
> > >> built-in trading functions. Tomasz has done a wonderful job
with
> > >> these, but they do not fit well with what I am doing with my
> > > trading.
> > >> I find it easier to understand what I am getting if I write
> > > everything
> > >> myself just for my situation and not the general case.
> > >>
> > >> Everything I do is in indicator mode in realtime. I build all
my
> > >> metrics into my AFL. My charts and numbers always match and all
> > > my
> > >> settings are stored in my Flexible Parameters scheme for
> different
> > >> test systems. It is a little different approach, but that is
one
> > > of
> > >> the beauties of AB --that it allows a lot of flexibility of
doing
> > > your
> > >> own thing if you don't want to use the built-in ways.
> > >>
> > >> Sometimes, you have to march to the beat of a different
drummer
> to
> > >> make money in these markets.
> > >>
> > >> Thanks again,
> > >> Dennis Brown
> > >>
> > >>
> > >> On Mar 12, 2008, at 1:38 PM, Howard B wrote:
> > >>
> > >>> Hi Dennis --
> > >>>
> > >>> There are several metrics already built in to AmiBroker that
> > > measure
> > >>> both the steepness and smoothness of the equity curve. Try
> > >>> generating a few test runs, plot their equity curves, note the
> > >>> values of these metrics, and see which ones best fit your
> > > trading
> > >>> personality. A nice advantage to using these is that they
> > > usually
> > >>> tend to select trading systems that test well out-of-sample,
so
> > > are
> > >>> appropriate for use with the Walk-Forward technique now also
> > > built
> > >>> in to AmiBroker.
> > >>>
> > >>> KRatio
> > >>> CAR/MDD
> > >>> RAR/MDD
> > >>> RRR
> > >>> RecoveryFactor
> > >>> UlcerPerformanceIndex
> > >>>
> > >>> Thanks,
> > >>> Howard
> > >>>
> > >>> On Tue, Mar 11, 2008 at 6:06 PM, Dennis Brown <see3d@>
> > >>> wrote:
> > >>> Hello,
> > >>>
> > >>> I have my system for intraday trading complete enough that I
> need
> > > to
> > >>> start selecting goodness criteria for comparing variations. I
> have
> > >>> selected a number of metrics to display in realtime for an n
day
> > >>> backtest like:
> > >>>
> > >>> total trade count
> > >>> average bars per trade
> > >>> winning trade %
> > >>> trade bars % in green
> > >>> best trade $
> > >>> worst trade $
> > >>> average win $
> > >>> average loss $
> > >>> *total profit $
> > >>> *max draw down $
> > >>> *EDGE (average $ per trade)
> > >>> *I have a graph of the cumulative profit over time and an
> overlaid
> > >>> straight line plot. This is the most powerful tool, because it
> > > lets
> > >>> me see the real character of the system. The straighter the
> line,
> > > the
> > >>> less likely it is over fit to the data and represents a robust
> > > system.
> > >>>
> > >>> I also have a graph of the trade equity on a trade by trade
> > > basis, so
> > >>> I can see how good the entry timing is and how a trade
> progresses
> > > on
> > >>> average or in outlier conditions.
> > >>>
> > >>> The * items are my key metrics for system comparison. This
> simple
> > >>> system runs completely in indicator mode. I test about 1000-
2000
> > >>> trades over a 10 week test period.
> > >>>
> > >>> Because of the type and manner of my trades (1 futures
contract
> > > only
> > >>> traded during market hours), the data is easy to judge for
> > > goodness.
> > >>> Since every day is an island, I could even use interesting
> random
> > > day
> > >>> strategies for in and out of sample data, but so far I just
use
> > >>> various sequential segments.
> > >>>
> > >>> However, when I am spinning my scroll wheel on parameters
while
> > >>> looking at my charts, it would be nice to have a number that
> > >>> represents how straight the equity curve is as a first pass --
> > >>> especially for when I partially automate the optimization
> > > process
> > >>> later.
> > >>>
> > >>> I thought I would just take the standard deviation of the
whole
> > > curve
> > >>> to the straight line. This is easy. But I think some of you
have
> > >>> given this problem a lot of thought and I figured one of you
may
> > > have
> > >>> some additional insights into the best method for getting a
> > > meaningful
> > >>> number for straightness/smoothness of the equity curve. So
here
> I
> > > put
> > >>> the question to you now with an open mind, before I become
set
> in
> > > my
> > >>> ways ;-)
> > >>>
> > >>> Best regards,
> > >>> Dennis Brown
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> > >
> > >
> > >
> > >
> > > Please note that this group is for discussion between users
only.
> > >
> > > To get support from AmiBroker please send an e-mail directly to
> > > SUPPORT {at} amibroker.com
> > >
> > > For NEW RELEASE ANNOUNCEMENTS and other news always check
DEVLOG:
> > > http://www.amibroke <http://www.amibroker.com/devlog/>
> r.com/devlog/
> > >
> > > For other support material please check also:
> > > http://www.amibroke <http://www.amibroker.com/support.html>
> > r.com/support.html
> > >
> > > Yahoo! Groups Links
> > >
> > >
> > >
> >
>

------------------------------------

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
Individual Email | Traditional

<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)

<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/