Dennis,
Thats is exactly the reason I raised the preceived
shortcoming of Sharpe ratio for discussion. Because I dont think it is
necessarily a shortcoming in itself. I also want to highlight the difference
between system development/system evaluation and real tradiing performance
monitoring, which I agree Sharpe ratio has definite disadvantage for the
latter.
I personally dont use Sharpe ratio per se in its original
form, but I am interested in Tushar Chande's method of modifying Sharper
ratio based on a rolling "3 monthly" computation of Sharpe Ratios, which
reflect quarterly performance. This approach is further enhanced by calculating
the "double" Sharpe ratio, that is the modified Sharpe Ratio of the modified
Sharpe Ratio. The logic is that if quarterly returns are steady and without
sharp gains or losses, then the standard deviation of the modified Sharpe Ratios
will be small. - hence I think it will be a good measure of the straightness of
equity curve - a question that you asked earlier. The measure of straightness of
equity curve should invarably measure the word "straightness" suggests, the
absence of neither sharp gains or losses.
Of couse if one really dislike Sharpe
ratio, one can also use what Schwager proposed: what he called
the Return Retracement Ratio. It is too much to describe it here in detail, but
it is basically the ratio of CAR divided by what he calls average maximum
retracement measure. Schwager describes it in quite detail in his book Schwager
on Futures: technical Analysis. A book I highly
recommend.
I personally use a composite type fitness
criteria which consists of something like
CAR*WinPct*(AvgPlPct/AvgBarsHold)^0.8*DoubleSharpe.... etc This gets changed a
bit with different systems depending the number of trades tath the system in
question naturally generates.
Paul,
Like you said, I always assume that if I start trading a new system that
the first thing I will experience is the worst case drawdown that will shake
my confidence in it. I have to ask myself how I would feel after a few
days of this, and would I continue, or start to question my system development
methods and drop out just when the system turns around. This psychology
happens to traders and investors all the time, and is one of the main causes
of losses.
Brian,
I have put together a lot of systems recently, looking for the most
robust ones. I have profitable systems with a Win% ranging from 30% to
80%. The higher the Win%, the larger the drawdowns. No free lunch.
However, the best systems are in the 45% to 55% range so far. What
does that tell you?
Best regards,
Dennis
On Mar 13, 2008, at 8:27 PM, Paul Ho wrote:
Before One can look at OOS, Once needs to have a
system or several systems Candidate and so you still need to have metrics to
evaluate what basically is forward looking performance estimates. Besides,
just because OOS says it is fine, it doesnt mean it will continue into the
future. And you are still more prone to start off at the flat spot of the
equity curve if you have a system that have more deviation from a
straightness of an equity curve regardless of how much OOS you have
done.
Your other statements seems more like motherhood
statements than looking for the mother ship to
me.
I agree with Howard's (past) comments that the best metric is the OOS
metric (that is for those who have used optimization to design the
system) or better still, several OOS metrics (if we have the
data).
The speculative (at this stage) point that I am introducing
into the discussion is that foward looking performance can be
estimated from the root causes (mechanics of the trading
system).
By continually focussing on the unknown future we are
chasing phantasms. While we try to catch one others are popping up
everywhere (just like a horror movie).
Better off to find the
mother ship, and understand the spawning process, if we are to have
any hope of dealing with the offspring.
brian_z
--- In amibroker@xxxxxxxxxps.com,
"Paul Ho" <paultsho@xx.> wrote: > > This particular
shortcoming of Sharpe ratio as mentioned by Howard has been >
well flaged by many books. and It make sense when one is comparing
PAST > performance from one fund manger to another, or from one
system to another. > However, when one is comparing forward
looking performance, such as when one > is developing new
systems or evaulating new variations of an existing > system.
Then IMHO this criticism is a little unjustified. Reason: If
there > are an equity curve in front of me, one that is with a
occasional surge of > profit (positive deviation) followed by a
relatively flat patch. I wouldn't > know with a lot of
confidence I'm go to experience a flat patch or > continuing surge
if I trade this system in the future. I have seen a number > of
systems that have a very quick rise in patches during backtest
and > optimisation, but basically flat during forward testing.
If I have a choice, > I would prefer a lower return but with
less deviation (both positive and > negative) when I'm
developing new system because I'm more confident that it > will
generate a regular profit for me. I must confess I am a short
term > trader, my trades last for hours to days. I can
apprecriate that long term > traders, those with trades lasting
weeks to years, might have a different > psychology and can
withstand large period of flat patches to wait for the > big
one. Of course, once I have started using a system, I'm all for
positive > surprises. > I personally think the biggest
drawback of Sharpe ratio lies with the fact > that the
straightness of an equity curve cannot be adequately described by
a > single Sharpe Ratio, because vastly different equity curves
shares similar > ratio numbers. A series of Sharpe Ratios
measured periodically is a better > guide. Tuschar Chande even
went as far as suggesting measuring a "Sharpe > Ratio" over the
series of Sharpe Ratio, I think this has merit. > > >
_____ > > From: amibroker@xxxxxxxxxps.com
[mailto:amibroker@xxxxxxxxxps.com]
On Behalf > Of Dennis Brown > Sent: Friday, 14 March 2008
4:46 AM > To: amibroker@xxxxxxxxxps.com >
Subject: Re: [amibroker] Re: What is best statistic for straightness
of > equity curve? > > > > Howard,
> > > You make an excellent point. The metrics used
to evaluate a system needs to > take into consideration the
normal "character" of the trading systems basic >
methodology. > > For instance my system takes small profits
and losses many times a day. It > is not biased for long or
short. It does not hold overnight, It only trades > broad market
futures. It does not compound equity. It is goodness be able >
to take a consistent draw from a fixed account size. > > This
means that my system will be subject to very different market
forces > than a system that swing trades stocks for a week or
two, and is subject to > overnight gaps, company earnings
announcements, dividends, interest rates > (on margin accounts),
and other unpredictable events. > > My system will perform
with a much smoother equity curve just because of the > way it
is defined. Commissions and Bid/Ask spreads are the main hurdles
to > profitability, but they are constants. > > I have
a much easier time telling if my system is robust. > > Best
regards, > Dennis > > > On Mar 13, 2008, at 1:01
PM, Howard B wrote: > > > > Greetings all
-- > > Professional money managers are sometimes evaluated
based on the Sharpe > Ratio of their performance, so it has some
value. But, in my research, I > have not found Sharpe Ratio to
be a very good metric for use when developing > systems. Yes,
higher Sharpe Ratios will have smaller standard deviations >
than lower Sharpe Ratios, but the standard deviation includes both
positive > and negative deviations. That is, it penalizes both
positive and negative > performance. If you are designing trend
following systems with long holding > periods, and looking for
the infrequent large gains associated with this > type of
system, Sharpe Ratio penalizes these. When Sharpe Ratio is used
as > the objective function in an automated walk forward process,
systems > selected as the best in-sample often perform much less
well out-of- sample > than systems selected using K-Ratio, RRR,
CAR/MDD, or UPI. > > Thanks for listening, >
Howard > > > > On Wed, Mar 12, 2008 at 10:33
PM, Paul Ho <paultsho@xxxxxx > <mailto:paultsho@...> com.au>
wrote: > > > > > Time doesnt permit me
to write a long post. But I think Jack Schwager in one > of his
books povides a very good description of what You want.
Tuschar > Chande also has insights. > One such parameter
is the Sharpe ratio, but you need use it slightly > differently.
Firstly, take risk free return as zero, and you are obtaining >
the ratio of mean return to std deviation. Secondly, calculated
yearly > sharpe ratios and compare them from year to
year. > > > _____ > > From:
amibroker@xxxxxxxxx <mailto:amibroker@xxxxxxxxxps.com>
ps.com > [mailto:amibroker@yahoogrou <mailto:amibroker@xxxxxxxxxps.com>
ps.com] On > Behalf OfDennis Brown > Sent: Thursday, 13
March 2008 12:24 PM > To: amibroker@xxxxxxxxx <mailto:amibroker@xxxxxxxxxps.com>
ps.com > Subject: Re: [amibroker] Re: What is best statistic for
straightness of > equity curve? > > >
> Brian, > > Thanks for your reply. > >
My thinking is that the Std Error will work. I do not need to use a
> Log function on my equity curve, because I do not compound my
results, > so they are linear. I also base my work on constant
range bars, so > that linearizes the curves even more. Profit
potential can only come > from price movement. The smoothest
and straightest equity curves come > from the most robust
systems. Period. You can look at the curve and > judge it, or find
a number that is associated with this property. > > However,
step functions get introduced into your nice trading system >
from big news events that change the character of the markets >
overnight, or in a minute during the day. I consider these things >
that produce large quick drawdowns will be captured by a Maximum >
Drawdown metric. The test period needs to have some of these big >
events in it. The event may be too quick to affect a large >
statistical function much, giving a false sense of goodness to the
> system. Or the perturbation might show up in a way that takes a
great > system and makes the smoothness number look bad due to
a one time > event. That is the challenge with a single number, so
I will have to > experiment with the right weightings. >
> That is why I say that the absolute judgement comes from
examination > of the equity curve. The goodness numbers are
just for ease of > relative comparisons of automated parameter
optimization for candidate > systems. It is also nice to have a
number or two as a future point of > reference rather than
going back over equity curves for every > comparison. >
> Perhaps an FFT over the equity curve would generate an
interesting > signature in the period of the dominant frequency and
I also need the > amplitude. I would have to look into this
more, since I have not > tried this before. > > I will
start out simple and see how better numbers compare to the >
curves, then decide where to go from there. > > > (Why
don't you just start posting some of your bits and pieces,
like > > your new PlotShapes PDF, to the UKB - it is a live
site - we don't > > have to wait for the big bang moment to
become an author - a lot of > > my stuff is mundane and/or
half finished, but it still has its uses). > > I am
buried in work right now, so I wanted to gauge the value to >
others of some of the things I could post on the UKB. I would have to
> fight for the time to figure out how to post and fiddle with with
> formatting issues etc. If it were as easy as sending a PDF email
> attachment here, I would have done it a month ago. It is the up
front > time investment that is holding me back right
now. > > When I get little feedback or interest from a post,
I can't prioritize > the time to share more of what I am doing.
If I were not so busy, I > would do it anyway, but for now I need
powerful justification to delay > some other important work to
make time for it. This is not a spare > time hobby for me, because
I have no spare time right now. :-( > > I could use a
teammate to get me through the initial stages. However, > I see
that only a few have ventured as far as posting yet, so the > field
is limited. I do all my content creation on a Mac, and keep my
> virtual PC free of everything but AmiBroker and related support
> programs. That is why I prefer to generate PDF content as it
works > everywhere. And I have exceptionally easy to use and
powerful tools > for generating them already. > > Best
regards, > Dennis Brown > > On Mar 12, 2008, at 7:19
PM, brian_z111 wrote: > > > Dennis, > > >
> So where is your thinking on this now? > > >
> > > (I have been following and I am building to some
possible input but > > since I don't understand logs and
barely understand standard error I > > have had to go back to
school - it takes quite a while for me to get > > my head
around that stuff and interpret it into trade talk). > > >
> I have taken a different approach to evaluation (which is still
a > > work in progress) and based on that I am inclined to the
view that > > evaluations on one equity curve are on rather weak
ground - IMO > > simulation is required for analysis of 'what
counts most'. > > > > Also I am zeroing in on the root
causes of equity curve profiles and > > measuring smoothness
of a curve is measuring the effect. > > > > BTW - your
pane based analysis is very interesting but I think > >
ultimately it might prove to have some limitations for good > >
evaluation (but not if we correctly identify root causes - we can >
> just pick them out, add some mathematical antecedents and then
we > > will now the answers that simulation will give us and not
need to > > bother the processor - I have convinced myself that
this is in my > > grasps and later I hope the maths people will
connect my conceptual > > does and bingo, we are
there). > > > > However, I love your question and
approach, so over to your immediate > > problem (I had it in
mind to go to town on an equity curve smoothness > > metric
anyway). > > > > K-ratio is actually a risk reward
metric (is that what you want)? > > > > It also (to me)
gets a little mysterious in its workings (Klestner > > doesn't
fully explain one part of it - not from my, lay, point of > >
view anyway). > > > > I am still thinking about
it. > > > > So far I would say StDev is out. >
> StandardError will do exactly what you say you want to do (as far
as > > I can tell - once again the stats teachers seem to
find it hard to > > put it into trade talk - I see it explained
in different ways in > > different books). > > >
> I haven't reached a final conclusion but it seems most likely
that if > > you use Standard Error on a compounded equity
curve with the LogN > > approach taken by Klestner you are there
- no need to go past that - > > my reservation is based on
the fact that I am not sure how to handle > > standardisation
- I only work in relative % change - Klestner > > attempts to
standardise the K-ratio - he had some trouble with it to > >
start out and had to add a standardising factor. > > >
>> Everything I do is in indicator mode in realtime. I build all
my > >> metrics into my AFL. My charts and numbers always
match and all > >> my > >> settings are stored in
my Flexible Parameters scheme for different > >> test
systems. It is a little different approach, but that is one >
>> of > >> the beauties of AB --that it allows a lot of
flexibility of doing > >> your > >> own thing if
you don't want to use the built-in ways. > > > > Yes,
all of my evaluation methods are home made, or adaptions of > >
popular methods - works for me. > > > > As I said - if
you want all of your evaluation in one window you > > might need
a math formula to sum up the transition from root cause to >
> simulation (I naively believe I have the beginning and end in the
bag > > and conceptually the middle formula seems
attainable). > > > > (Why don't you just start posting
some of your bits and pieces, like > > your new PlotShapes
PDF, to the UKB - it is a live site - we don't > > have to wait
for the big bang moment to become an author - a lot of > > my
stuff is mundane and/or half finished, but it still has its
uses). > > > > brian_z > > >
> > > --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com>
ps.com, > Dennis Brown <see3d@> wrote: >
>> > >> Howard, > >> > >> Thanks
for the input. I will investigate these some more. >
>> > >> However, I do not use the built-in equity
functions, or any of the > >> built-in trading functions.
Tomasz has done a wonderful job with > >> these, but they do
not fit well with what I am doing with my > > trading. >
>> I find it easier to understand what I am getting if I
write > > everything > >> myself just for my
situation and not the general case. > >> > >>
Everything I do is in indicator mode in realtime. I build all my >
>> metrics into my AFL. My charts and numbers always match and
all > > my > >> settings are stored in my Flexible
Parameters scheme for different > >> test systems. It is a
little different approach, but that is one > > of >
>> the beauties of AB --that it allows a lot of flexibility of
doing > > your > >> own thing if you don't want to
use the built-in ways. > >> > >> Sometimes, you
have to march to the beat of a different drummer to > >>
make money in these markets. > >> > >> Thanks
again, > >> Dennis Brown > >> >
>> > >> On Mar 12, 2008, at 1:38 PM, Howard B
wrote: > >> > >>> Hi Dennis -- >
>>> > >>> There are several metrics already built
in to AmiBroker that > > measure > >>> both the
steepness and smoothness of the equity curve. Try > >>>
generating a few test runs, plot their equity curves, note the >
>>> values of these metrics, and see which ones best fit
your > > trading > >>> personality. A nice
advantage to using these is that they > > usually >
>>> tend to select trading systems that test well out-of-sample,
so > > are > >>> appropriate for use with the
Walk-Forward technique now also > > built > >>> in
to AmiBroker. > >>> > >>> KRatio >
>>> CAR/MDD > >>> RAR/MDD > >>>
RRR > >>> RecoveryFactor > >>>
UlcerPerformanceIndex > >>> > >>>
Thanks, > >>> Howard > >>> >
>>> On Tue, Mar 11, 2008 at 6:06 PM, Dennis Brown
<see3d@> > >>> wrote: > >>>
Hello, > >>> > >>> I have my system for
intraday trading complete enough that I need > > to >
>>> start selecting goodness criteria for comparing variations. I
have > >>> selected a number of metrics to display in
realtime for an n day > >>> backtest like: >
>>> > >>> total trade count > >>>
average bars per trade > >>> winning trade % >
>>> trade bars % in green > >>> best trade
$ > >>> worst trade $ > >>> average win
$ > >>> average loss $ > >>> *total profit
$ > >>> *max draw down $ > >>> *EDGE
(average $ per trade) > >>> *I have a graph of the
cumulative profit over time and an overlaid > >>>
straight line plot. This is the most powerful tool, because it >
> lets > >>> me see the real character of the system.
The straighter the line, > > the > >>> less
likely it is over fit to the data and represents a robust > >
system. > >>> > >>> I also have a graph of
the trade equity on a trade by trade > > basis, so >
>>> I can see how good the entry timing is and how a trade
progresses > > on > >>> average or in outlier
conditions. > >>> > >>> The * items are my
key metrics for system comparison. This simple > >>>
system runs completely in indicator mode. I test about 1000-2000 >
>>> trades over a 10 week test period. >
>>> > >>> Because of the type and manner of my
trades (1 futures contract > > only > >>> traded
during market hours), the data is easy to judge for > >
goodness. > >>> Since every day is an island, I could even
use interesting random > > day > >>>
strategies for in and out of sample data, but so far I just use >
>>> various sequential segments. > >>> >
>>> However, when I am spinning my scroll wheel on parameters
while > >>> looking at my charts, it would be nice to have
a number that > >>> represents how straight the equity
curve is as a first pass -- > >>> especially for when I
partially automate the optimization > > process >
>>> later. > >>> > >>> I thought I
would just take the standard deviation of the whole > >
curve > >>> to the straight line. This is easy. But I think
some of you have > >>> given this problem a lot of thought
and I figured one of you may > > have > >>> some
additional insights into the best method for getting a > >
meaningful > >>> number for straightness/smoothness of
the equity curve. So here I > > put > >>> the
question to you now with an open mind, before I become set in >
> my > >>> ways ;-) > >>> >
>>> Best regards, > >>> Dennis Brown >
>>> > >>> > >>> >
>>> > >> > > > > >
> > > > > Please note that this group is for
discussion between users only. > > > > To get support
from AmiBroker please send an e-mail directly to > > SUPPORT {at}
amibroker.com > > > > For NEW RELEASE ANNOUNCEMENTS and
other news always check DEVLOG: > > http://www.amibroke <http://www.amibroker.com/devlog/>
r.com/devlog/ > > > > For other support material
please check also: > > http://www.amibroke <http://www.amibroker.com/support.html> >
r.com/support.html > > > > Yahoo! Groups
Links > > > > > > >
__._,_.___
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
__,_._,___
|