This particular shortcoming of Sharpe ratio as
mentioned by Howard has been well flaged by many books. and It make sense
when one is comparing PAST performance from one fund manger to another, or
from one system to another. However, when one is comparing forward looking
performance, such as when one is developing new systems or evaulating new
variations of an existing system. Then IMHO this criticism is a little
unjustified. Reason: If there are an equity curve in front of me, one that is
with a occasional surge of profit (positive deviation) followed by a
relatively flat patch. I wouldn't know with a lot of confidence I'm go to
experience a flat patch or continuing surge if I trade this system in the
future. I have seen a number of systems that have a very quick rise in patches
during backtest and optimisation, but basically flat during forward testing. If
I have a choice, I would prefer a lower return but with less deviation (both
positive and negative) when I'm developing new system because I'm more confident
that it will generate a regular profit for me. I must confess I am a short term
trader, my trades last for hours to days. I can apprecriate that long term
traders, those with trades lasting weeks to years, might have a different
psychology and can withstand large period of flat patches to wait for the big
one. Of course, once I have started using a system, I'm all for positive
surprises.
I personally think the biggest drawback of
Sharpe ratio lies with the fact that the straightness of an equity curve cannot
be adequately described by a single Sharpe Ratio, because vastly different
equity curves shares similar ratio numbers. A series of Sharpe Ratios measured
periodically is a better guide. Tuschar Chande even went as far as suggesting
measuring a "Sharpe Ratio" over the series of Sharpe Ratio, I think this has
merit.
Howard,
You make an excellent point. The metrics used to evaluate a system
needs to take into consideration the normal "character" of the trading systems
basic methodology.
For instance my system takes small profits and losses many times a day.
It is not biased for long or short. It does not hold overnight, It
only trades broad market futures. It does not compound equity. It
is goodness be able to take a consistent draw from a fixed account
size.
This means that my system will be subject to very different market forces
than a system that swing trades stocks for a week or two, and is subject to
overnight gaps, company earnings announcements, dividends, interest rates (on
margin accounts), and other unpredictable events.
My system will perform with a much smoother equity curve just because of
the way it is defined. Commissions and Bid/Ask spreads are the
main hurdles to profitability, but they are constants.
I have a much easier time telling if my system is robust.
Best regards,
Dennis
On Mar 13, 2008, at 1:01 PM, Howard B wrote:
Greetings all -- Professional money managers are sometimes
evaluated based on the Sharpe Ratio of their performance, so it has some
value. But, in my research, I have not found Sharpe Ratio to be a very
good metric for use when developing systems. Yes, higher Sharpe Ratios
will have smaller standard deviations than lower Sharpe Ratios, but the
standard deviation includes both positive and negative deviations.
That is, it penalizes both positive and negative performance. If you
are designing trend following systems with long holding periods, and looking
for the infrequent large gains associated with this type of system, Sharpe
Ratio penalizes these. When Sharpe Ratio is used as the objective
function in an automated walk forward process, systems selected as the best
in-sample often perform much less well out-of-sample than systems selected
using K-Ratio, RRR, CAR/MDD, or UPI. Thanks for
listening, Howard
On Wed, Mar 12, 2008 at 10:33 PM, Paul Ho < paultsho@xxxxxxcom.au>
wrote:
Time
doesnt permit me to write a long post. But I think Jack Schwager in one of
his books povides a very good description of what You want. Tuschar Chande
also has insights.
One
such parameter is the Sharpe ratio, but you need use it slightly
differently. Firstly, take risk free return as zero, and you are obtaining
the ratio of mean return to std deviation. Secondly, calculated yearly
sharpe ratios and compare them from year to
year.
Brian,
Thanks for your reply.
My thinking is that the
Std Error will work. I do not need to use a Log function on my equity
curve, because I do not compound my results, so they are linear. I also
base my work on constant range bars, so that linearizes the curves
even more. Profit potential can only come from price movement. The
smoothest and straightest equity curves come from the most robust
systems. Period. You can look at the curve and judge it, or find a number
that is associated with this property.
However, step functions
get introduced into your nice trading system from big news events that
change the character of the markets overnight, or in a minute
during the day. I consider these things that produce large quick
drawdowns will be captured by a Maximum Drawdown metric. The test
period needs to have some of these big events in it. The event may
be too quick to affect a large statistical function much,
giving a false sense of goodness to the system. Or the perturbation
might show up in a way that takes a great system and makes the
smoothness number look bad due to a one time event. That is the
challenge with a single number, so I will have to experiment with the right
weightings.
That is why I say that the absolute judgement comes
from examination of
the equity curve. The goodness numbers are just for ease of relative comparisons of
automated parameter optimization for candidate systems. It is also nice to
have a number or two as a future point of reference rather than going
back over equity curves for every comparison.
Perhaps
an FFT over the equity curve would generate an interesting signature in the period of
the dominant frequency and I also need the amplitude. I would have to
look into this more, since I have not tried this before.
I
will start out simple and see how better numbers compare to the curves, then decide where
to go from there.
> (Why don't you just start posting some of
your bits and pieces, like > your new PlotShapes PDF, to the UKB -
it is a live site - we don't > have to wait for the big bang
moment to become an author - a lot of > my stuff is mundane and/or
half finished, but it still has its uses).
I am buried in work
right now, so I wanted to gauge the value to others of some of the
things I could post on the UKB. I would have to fight for the time to
figure out how to post and fiddle with with formatting issues etc. If
it were as easy as sending a PDF email attachment here, I would
have done it a month ago. It is the up front time investment that is
holding me back right now.
When I get little feedback or interest
from a post, I can't prioritize the time to share more of
what I am doing. If I were not so busy, I would do it anyway, but for
now I need powerful justification to delay some other important work
to make time for it. This is not a spare time hobby for me, because
I have no spare time right now. :-(
I could use a teammate to get
me through the initial stages. However, I see that only a few have
ventured as far as posting yet, so the field is limited. I do all
my content creation on a Mac, and keep my virtual PC free of
everything but AmiBroker and related support programs. That is why I
prefer to generate PDF content as it works everywhere. And I have
exceptionally easy to use and powerful tools for generating them
already.
Best regards, Dennis Brown
On Mar 12, 2008, at
7:19 PM, brian_z111 wrote:
> Dennis, > > So where
is your thinking on this now? > > > (I have been
following and I am building to some possible input but > since I
don't understand logs and barely understand standard error I >
have had to go back to school - it takes quite a while for me to
get > my head around that stuff and interpret it into trade
talk). > > I have taken a different approach to evaluation
(which is still a > work in progress) and based on that I am
inclined to the view that > evaluations on one equity curve are on
rather weak ground - IMO > simulation is required for analysis of
'what counts most'. > > Also I am zeroing in on the root
causes of equity curve profiles and > measuring smoothness of a
curve is measuring the effect. > > BTW - your pane based
analysis is very interesting but I think > ultimately it might
prove to have some limitations for good > evaluation (but not if
we correctly identify root causes - we can > just pick them out,
add some mathematical antecedents and then we > will now the
answers that simulation will give us and not need to > bother the
processor - I have convinced myself that this is in my > grasps
and later I hope the maths people will connect my conceptual >
does and bingo, we are there). > > However, I love your
question and approach, so over to your immediate > problem (I had
it in mind to go to town on an equity curve smoothness > metric
anyway). > > K-ratio is actually a risk reward metric (is
that what you want)? > > It also (to me) gets a little
mysterious in its workings (Klestner > doesn't fully explain one
part of it - not from my, lay, point of > view
anyway). > > I am still thinking about it. > >
So far I would say StDev is out. > StandardError will do exactly
what you say you want to do (as far as > I can tell - once again
the stats teachers seem to find it hard to > put it into trade
talk - I see it explained in different ways in > different
books). > > I haven't reached a final conclusion but it
seems most likely that if > you use Standard Error on a compounded
equity curve with the LogN > approach taken by Klestner you are
there - no need to go past that - > my reservation is based on the
fact that I am not sure how to handle > standardisation - I only
work in relative % change - Klestner > attempts to standardise the
K-ratio - he had some trouble with it to > start out and had to
add a standardising factor. > >> Everything I do is in
indicator mode in realtime. I build all my >> metrics into my
AFL. My charts and numbers always match and all >>
my >> settings are stored in my Flexible Parameters scheme for
different >> test systems. It is a little different approach,
but that is one >> of >> the beauties of AB --that it
allows a lot of flexibility of doing >> your >> own
thing if you don't want to use the built-in ways. > > Yes,
all of my evaluation methods are home made, or adaptions of >
popular methods - works for me. > > As I said - if you want
all of your evaluation in one window you > might need a math
formula to sum up the transition from root cause to > simulation
(I naively believe I have the beginning and end in the bag > and
conceptually the middle formula seems attainable). > > (Why
don't you just start posting some of your bits and pieces, like >
your new PlotShapes PDF, to the UKB - it is a live site - we
don't > have to wait for the big bang moment to become an author -
a lot of > my stuff is mundane and/or half finished, but it still
has its uses). > > brian_z > > > ---
In amibroker@xxxxxxxxxps.com, Dennis Brown
<see3d@xxx> wrote: >> >>
Howard, >> >> Thanks for the input. I will investigate
these some more. >> >> However, I do not use the
built-in equity functions, or any of the >> built-in trading
functions. Tomasz has done a wonderful job with >> these, but
they do not fit well with what I am doing with my >
trading. >> I find it easier to understand what I am getting if
I write > everything >> myself just for my situation and
not the general case. >> >> Everything I do is in
indicator mode in realtime. I build all my >> metrics into my
AFL. My charts and numbers always match and all > my >>
settings are stored in my Flexible Parameters scheme for
different >> test systems. It is a little different approach,
but that is one > of >> the beauties of AB --that it
allows a lot of flexibility of doing > your >> own thing
if you don't want to use the built-in ways. >> >>
Sometimes, you have to march to the beat of a different drummer
to >> make money in these markets. >> >>
Thanks again, >> Dennis
Brown >> >> >> On Mar 12, 2008, at 1:38 PM,
Howard B wrote: >> >>> Hi Dennis
-- >>> >>> There are several metrics already
built in to AmiBroker that > measure >>> both the
steepness and smoothness of the equity curve. Try >>>
generating a few test runs, plot their equity curves, note
the >>> values of these metrics, and see which ones best fit
your > trading >>> personality. A nice advantage to
using these is that they > usually >>> tend to select
trading systems that test well out-of-sample, so >
are >>> appropriate for use with the Walk-Forward technique
now also > built >>> in to
AmiBroker. >>> >>> KRatio >>>
CAR/MDD >>> RAR/MDD >>> RRR >>>
RecoveryFactor >>>
UlcerPerformanceIndex >>> >>>
Thanks, >>> Howard >>> >>> On Tue,
Mar 11, 2008 at 6:06 PM, Dennis Brown <see3d@xxx> >>>
wrote: >>> Hello, >>> >>> I have my
system for intraday trading complete enough that I need >
to >>> start selecting goodness criteria for comparing
variations. I have >>> selected a number of metrics to
display in realtime for an n day >>> backtest
like: >>> >>> total trade count >>>
average bars per trade >>> winning trade % >>>
trade bars % in green >>> best trade $ >>> worst
trade $ >>> average win $ >>> average loss
$ >>> *total profit $ >>> *max draw down
$ >>> *EDGE (average $ per trade) >>> *I have a
graph of the cumulative profit over time and an overlaid >>>
straight line plot. This is the most powerful tool, because it >
lets >>> me see the real character of the system. The
straighter the line, > the >>> less likely it is over
fit to the data and represents a robust >
system. >>> >>> I also have a graph of the trade
equity on a trade by trade > basis, so >>> I can see
how good the entry timing is and how a trade progresses >
on >>> average or in outlier
conditions. >>> >>> The * items are my key
metrics for system comparison. This simple >>> system runs
completely in indicator mode. I test about 1000-2000 >>>
trades over a 10 week test period. >>> >>>
Because of the type and manner of my trades (1 futures contract >
only >>> traded during market hours), the data is easy to
judge for > goodness. >>> Since every day is an
island, I could even use interesting random > day >>>
strategies for in and out of sample data, but so far I just
use >>> various sequential
segments. >>> >>> However, when I am spinning my
scroll wheel on parameters while >>> looking at my charts,
it would be nice to have a number that >>> represents how
straight the equity curve is as a first pass -- >>>
especially for when I partially automate the optimization >
process >>> later. >>> >>> I thought
I would just take the standard deviation of the whole >
curve >>> to the straight line. This is easy. But I think
some of you have >>> given this problem a lot of thought and
I figured one of you may > have >>> some additional
insights into the best method for getting a >
meaningful >>> number for straightness/smoothness of
the equity curve. So here I > put >>> the question to
you now with an open mind, before I become set in >
my >>> ways ;-) >>> >>> Best
regards, >>> Dennis
Brown >>> >>> >>> >>> >> > > > > >
Please note that this group is for discussion between users
only. > > To get support from AmiBroker please send an
e-mail directly to > SUPPORT {at} amibroker.com > > For NEW RELEASE
ANNOUNCEMENTS and other news always check DEVLOG: > http://www.amibroker.com/devlog/ > >
For other support material please check also: > http://www.amibroker.com/support.html > >
Yahoo! Groups
Links > > >
__._,_.___
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
__,_._,___
|