I agree with Howard's (past) comments that the best metric is the OOS
metric (that is for those who have used optimization to design the
system) or better still, several OOS metrics (if we have the
data).
The speculative (at this stage) point that I am introducing into
the
discussion is that foward looking performance can be estimated from
the root causes (mechanics of the trading system).
By continually
focussing on the unknown future we are chasing
phantasms. While we try to
catch one others are popping up everywhere
(just like a horror
movie).
Better off to find the mother ship, and understand the spawning
process, if we are to have any hope of dealing with the
offspring.
brian_z
--- In amibroker@xxxxxxxxxps.com,
"Paul Ho" <paultsho@xx.> wrote:
>
> This particular
shortcoming of Sharpe ratio as mentioned by Howard
has been
> well
flaged by many books. and It make sense when one is comparing
PAST
>
performance from one fund manger to another, or from one system to
another.
> However, when one is comparing forward looking
performance, such as
when one
> is developing new systems or
evaulating new variations of an
existing
> system. Then IMHO this
criticism is a little unjustified. Reason:
If there
> are an equity
curve in front of me, one that is with a occasional
surge of
>
profit (positive deviation) followed by a relatively flat patch. I
wouldn't
> know with a lot of confidence I'm go to experience a flat
patch or
> continuing surge if I trade this system in the future. I have
seen
a number
> of systems that have a very quick rise in patches
during backtest
and
> optimisation, but basically flat during
forward testing. If I have
a choice,
> I would prefer a lower return
but with less deviation (both
positive and
> negative) when I'm
developing new system because I'm more confident
that it
> will
generate a regular profit for me. I must confess I am a short
term
>
trader, my trades last for hours to days. I can apprecriate that
long
term
> traders, those with trades lasting weeks to years, might have a
different
> psychology and can withstand large period of flat
patches to wait
for the
> big one. Of course, once I have started
using a system, I'm all for
positive
> surprises.
> I
personally think the biggest drawback of Sharpe ratio lies with
the
fact
> that the straightness of an equity curve cannot be adequately
described by a
> single Sharpe Ratio, because vastly different
equity curves shares
similar
> ratio numbers. A series of Sharpe
Ratios measured periodically is a
better
> guide. Tuschar Chande
even went as far as suggesting measuring
a "Sharpe
> Ratio" over the
series of Sharpe Ratio, I think this has merit.
>
>
>
_____
>
> From: amibroker@xxxxxxxxxps.com
[mailto:amibroker@xxxxxxxxxps.com]
On Behalf
> Of Dennis Brown
> Sent: Friday, 14 March 2008 4:46
AM
> To: amibroker@xxxxxxxxxps.com
>
Subject: Re: [amibroker] Re: What is best statistic for
straightness
of
> equity curve?
>
>
>
> Howard,
>
>
> You make an excellent point. The metrics used to evaluate a
system
needs to
> take into consideration the normal "character" of
the trading
systems basic
> methodology.
>
> For
instance my system takes small profits and losses many times a
day.
It
> is not biased for long or short. It does not hold overnight, It
only trades
> broad market futures. It does not compound equity. It
is goodness
be able
> to take a consistent draw from a fixed account
size.
>
> This means that my system will be subject to very
different market
forces
> than a system that swing trades stocks for
a week or two, and is
subject to
> overnight gaps, company earnings
announcements, dividends, interest
rates
> (on margin accounts), and
other unpredictable events.
>
> My system will perform with a
much smoother equity curve just
because of the
> way it is defined.
Commissions and Bid/Ask spreads are the main
hurdles to
>
profitability, but they are constants.
>
> I have a much easier
time telling if my system is robust.
>
> Best regards,
>
Dennis
>
>
> On Mar 13, 2008, at 1:01 PM, Howard B
wrote:
>
>
>
> Greetings all --
>
>
Professional money managers are sometimes evaluated based on the
Sharpe
> Ratio of their performance, so it has some value. But, in
my
research, I
> have not found Sharpe Ratio to be a very good
metric for use when
developing
> systems. Yes, higher Sharpe Ratios
will have smaller standard
deviations
> than lower Sharpe Ratios,
but the standard deviation includes both
positive
> and negative
deviations. That is, it penalizes both positive and
negative
>
performance. If you are designing trend following systems with
long
holding
> periods, and looking for the infrequent large gains associated
with
this
> type of system, Sharpe Ratio penalizes these. When
Sharpe Ratio is
used as
> the objective function in an automated
walk forward process, systems
> selected as the best in-sample often
perform much less well out-of-
sample
> than systems selected using
K-Ratio, RRR, CAR/MDD, or UPI.
>
> Thanks for listening,
>
Howard
>
>
>
> On Wed, Mar 12, 2008 at 10:33 PM,
Paul Ho <paultsho@xxxxxx
> <mailto:paultsho@...>
com.au> wrote:
>
>
>
>
> Time doesnt
permit me to write a long post. But I think Jack
Schwager in one
>
of his books povides a very good description of what You want.
Tuschar
> Chande also has insights.
> One such parameter is
the Sharpe ratio, but you need use it slightly
> differently. Firstly,
take risk free return as zero, and you are
obtaining
> the ratio of
mean return to std deviation. Secondly, calculated
yearly
> sharpe
ratios and compare them from year to year.
>
>
> _____
>
> From: amibroker@xxxxxxxxx <mailto:amibroker@xxxxxxxxxps.com>
ps.com
> [mailto:amibroker@yahoogrou <mailto:amibroker@xxxxxxxxxps.com>
ps.com] On
> Behalf OfDennis Brown
> Sent: Thursday, 13 March
2008 12:24 PM
> To: amibroker@xxxxxxxxx <mailto:amibroker@xxxxxxxxxps.com>
ps.com
> Subject: Re: [amibroker] Re: What is best statistic for
straightness of
> equity curve?
>
>
>
>
Brian,
>
> Thanks for your reply.
>
> My thinking is
that the Std Error will work. I do not need to use a
> Log function on
my equity curve, because I do not compound my
results,
> so they
are linear. I also base my work on constant range bars, so
> that
linearizes the curves even more. Profit potential can only
come
>
from price movement. The smoothest and straightest equity curves
come
> from the most robust systems. Period. You can look at the curve and
> judge it, or find a number that is associated with this
property.
>
> However, step functions get introduced into your
nice trading
system
> from big news events that change the
character of the markets
> overnight, or in a minute during the day. I
consider these things
> that produce large quick drawdowns will be
captured by a Maximum
> Drawdown metric. The test period needs to have
some of these big
> events in it. The event may be too quick to affect
a large
> statistical function much, giving a false sense of goodness
to the
> system. Or the perturbation might show up in a way that takes
a
great
> system and makes the smoothness number look bad due to a
one time
> event. That is the challenge with a single number, so I will
have
to
> experiment with the right weightings.
>
>
That is why I say that the absolute judgement comes from
examination
> of the equity curve. The goodness numbers are just for ease of
> relative comparisons of automated parameter optimization for
candidate
> systems. It is also nice to have a number or two as a
future point
of
> reference rather than going back over equity
curves for every
> comparison.
>
> Perhaps an FFT over the
equity curve would generate an interesting
> signature in the period of
the dominant frequency and I also need
the
> amplitude. I would
have to look into this more, since I have not
> tried this
before.
>
> I will start out simple and see how better numbers
compare to the
> curves, then decide where to go from there.
>
> > (Why don't you just start posting some of your bits and pieces,
like
> > your new PlotShapes PDF, to the UKB - it is a live site
- we don't
> > have to wait for the big bang moment to become an
author - a lot
of
> > my stuff is mundane and/or half finished,
but it still has its
uses).
>
> I am buried in work right
now, so I wanted to gauge the value to
> others of some of the things I
could post on the UKB. I would have
to
> fight for the time to
figure out how to post and fiddle with with
> formatting issues etc. If
it were as easy as sending a PDF email
> attachment here, I would have
done it a month ago. It is the up
front
> time investment that is
holding me back right now.
>
> When I get little feedback or
interest from a post, I can't
prioritize
> the time to share more
of what I am doing. If I were not so busy, I
> would do it anyway, but
for now I need powerful justification to
delay
> some other
important work to make time for it. This is not a spare
> time hobby
for me, because I have no spare time right now. :-(
>
> I could
use a teammate to get me through the initial stages.
However,
> I
see that only a few have ventured as far as posting yet, so the
> field
is limited. I do all my content creation on a Mac, and keep
my
>
virtual PC free of everything but AmiBroker and related support
>
programs. That is why I prefer to generate PDF content as it works
>
everywhere. And I have exceptionally easy to use and powerful tools
>
for generating them already.
>
> Best regards,
> Dennis
Brown
>
> On Mar 12, 2008, at 7:19 PM, brian_z111 wrote:
>
> > Dennis,
> >
> > So where is your thinking on
this now?
> >
> >
> > (I have been following and I
am building to some possible input
but
> > since I don't
understand logs and barely understand standard
error I
> > have
had to go back to school - it takes quite a while for me to
get
>
> my head around that stuff and interpret it into trade talk).
>
>
> > I have taken a different approach to evaluation (which is
still a
> > work in progress) and based on that I am inclined to the
view that
> > evaluations on one equity curve are on rather weak
ground - IMO
> > simulation is required for analysis of 'what counts
most'.
> >
> > Also I am zeroing in on the root causes of
equity curve profiles
and
> > measuring smoothness of a curve is
measuring the effect.
> >
> > BTW - your pane based analysis
is very interesting but I think
> > ultimately it might prove to have
some limitations for good
> > evaluation (but not if we correctly
identify root causes - we can
> > just pick them out, add some
mathematical antecedents and then we
> > will now the answers that
simulation will give us and not need to
> > bother the processor - I
have convinced myself that this is in my
> > grasps and later I hope
the maths people will connect my
conceptual
> > does and bingo,
we are there).
> >
> > However, I love your question and
approach, so over to your
immediate
> > problem (I had it in mind
to go to town on an equity curve
smoothness
> > metric
anyway).
> >
> > K-ratio is actually a risk reward metric
(is that what you want)?
> >
> > It also (to me) gets a
little mysterious in its workings (Klestner
> > doesn't fully explain
one part of it - not from my, lay, point of
> > view anyway).
>
>
> > I am still thinking about it.
> >
> > So
far I would say StDev is out.
> > StandardError will do exactly what
you say you want to do (as far
as
> > I can tell - once again the
stats teachers seem to find it hard to
> > put it into trade talk - I
see it explained in different ways in
> > different books).
>
>
> > I haven't reached a final conclusion but it seems most
likely
that if
> > you use Standard Error on a compounded equity
curve with the LogN
> > approach taken by Klestner you are there - no
need to go past
that -
> > my reservation is based on the fact
that I am not sure how to
handle
> > standardisation - I only
work in relative % change - Klestner
> > attempts to standardise the
K-ratio - he had some trouble with it
to
> > start out and had to
add a standardising factor.
> >
> >> Everything I do is
in indicator mode in realtime. I build all my
> >> metrics into my
AFL. My charts and numbers always match and all
> >> my
>
>> settings are stored in my Flexible Parameters scheme for
different
> >> test systems. It is a little different
approach, but that is one
> >> of
> >> the beauties of
AB --that it allows a lot of flexibility of doing
> >>
your
> >> own thing if you don't want to use the built-in
ways.
> >
> > Yes, all of my evaluation methods are home
made, or adaptions of
> > popular methods - works for me.
>
>
> > As I said - if you want all of your evaluation in one window
you
> > might need a math formula to sum up the transition from root
cause to
> > simulation (I naively believe I have the beginning
and end in the
bag
> > and conceptually the middle formula seems
attainable).
> >
> > (Why don't you just start posting some
of your bits and pieces,
like
> > your new PlotShapes PDF, to the
UKB - it is a live site - we don't
> > have to wait for the big bang
moment to become an author - a lot
of
> > my stuff is mundane
and/or half finished, but it still has its
uses).
> >
>
> brian_z
> >
> >
> > --- In amibroker@xxxxxxxxx
<mailto:amibroker%40yahoogroups.com>
ps.com,
>
Dennis Brown <see3d@> wrote:
> >>
> >>
Howard,
> >>
> >> Thanks for the input. I will
investigate these some more.
> >>
> >> However, I do
not use the built-in equity functions, or any of
the
> >>
built-in trading functions. Tomasz has done a wonderful job with
>
>> these, but they do not fit well with what I am doing with my
>
> trading.
> >> I find it easier to understand what I am
getting if I write
> > everything
> >> myself just for my
situation and not the general case.
> >>
> >>
Everything I do is in indicator mode in realtime. I build all my
>
>> metrics into my AFL. My charts and numbers always match and
all
> > my
> >> settings are stored in my Flexible
Parameters scheme for
different
> >> test systems. It is a
little different approach, but that is one
> > of
> >>
the beauties of AB --that it allows a lot of flexibility of doing
> >
your
> >> own thing if you don't want to use the built-in
ways.
> >>
> >> Sometimes, you have to march to the
beat of a different drummer
to
> >> make money in these
markets.
> >>
> >> Thanks again,
> >>
Dennis Brown
> >>
> >>
> >> On Mar 12,
2008, at 1:38 PM, Howard B wrote:
> >>
> >>> Hi
Dennis --
> >>>
> >>> There are several metrics
already built in to AmiBroker that
> > measure
> >>>
both the steepness and smoothness of the equity curve. Try
>
>>> generating a few test runs, plot their equity curves, note
the
> >>> values of these metrics, and see which ones best fit
your
> > trading
> >>> personality. A nice advantage
to using these is that they
> > usually
> >>> tend to
select trading systems that test well out-of-sample, so
> >
are
> >>> appropriate for use with the Walk-Forward technique
now also
> > built
> >>> in to AmiBroker.
>
>>>
> >>> KRatio
> >>> CAR/MDD
>
>>> RAR/MDD
> >>> RRR
> >>>
RecoveryFactor
> >>> UlcerPerformanceIndex
>
>>>
> >>> Thanks,
> >>> Howard
>
>>>
> >>> On Tue, Mar 11, 2008 at 6:06 PM, Dennis
Brown <see3d@>
> >>> wrote:
> >>>
Hello,
> >>>
> >>> I have my system for intraday
trading complete enough that I
need
> > to
> >>>
start selecting goodness criteria for comparing variations. I
have
>
>>> selected a number of metrics to display in realtime for an n
day
> >>> backtest like:
> >>>
>
>>> total trade count
> >>> average bars per
trade
> >>> winning trade %
> >>> trade bars %
in green
> >>> best trade $
> >>> worst trade
$
> >>> average win $
> >>> average loss
$
> >>> *total profit $
> >>> *max draw down
$
> >>> *EDGE (average $ per trade)
> >>> *I
have a graph of the cumulative profit over time and an
overlaid
>
>>> straight line plot. This is the most powerful tool, because
it
> > lets
> >>> me see the real character of the
system. The straighter the
line,
> > the
> >>>
less likely it is over fit to the data and represents a robust
> >
system.
> >>>
> >>> I also have a graph of the
trade equity on a trade by trade
> > basis, so
> >>> I
can see how good the entry timing is and how a trade
progresses
>
> on
> >>> average or in outlier conditions.
>
>>>
> >>> The * items are my key metrics for system
comparison. This
simple
> >>> system runs completely in
indicator mode. I test about 1000-2000
> >>> trades over a 10
week test period.
> >>>
> >>> Because of the
type and manner of my trades (1 futures contract
> > only
>
>>> traded during market hours), the data is easy to judge
for
> > goodness.
> >>> Since every day is an island,
I could even use interesting
random
> > day
> >>>
strategies for in and out of sample data, but so far I just use
>
>>> various sequential segments.
> >>>
>
>>> However, when I am spinning my scroll wheel on parameters
while
> >>> looking at my charts, it would be nice to have a
number that
> >>> represents how straight the equity curve is
as a first pass --
> >>> especially for when I partially
automate the optimization
> > process
> >>>
later.
> >>>
> >>> I thought I would just take
the standard deviation of the whole
> > curve
> >>> to
the straight line. This is easy. But I think some of you have
>
>>> given this problem a lot of thought and I figured one of you
may
> > have
> >>> some additional insights into the
best method for getting a
> > meaningful
> >>> number
for straightness/smoothness of the equity curve. So here
I
>
> put
> >>> the question to you now with an open mind,
before I become set
in
> > my
> >>> ways
;-)
> >>>
> >>> Best regards,
>
>>> Dennis Brown
> >>>
> >>>
>
>>>
> >>>
> >>
> >
>
>
> >
> >
> > Please note that this group is for
discussion between users only.
> >
> > To get support from
AmiBroker please send an e-mail directly to
> > SUPPORT {at}
amibroker.com
> >
> > For NEW RELEASE ANNOUNCEMENTS and
other news always check DEVLOG:
> > http://www.amibroke <http://www.amibroker.com/devlog/>
r.com/devlog/
> >
> > For other support material please
check also:
> > http://www.amibroke
<http://www.amibroker.com/support.html>
>
r.com/support.html
> >
> > Yahoo! Groups Links
>
>
> >
> >
>