[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] AB Back-testing Metrics



PureBytes Links

Trading Reference Links

Hi Mike,

 

Thank you for your response.

 

I  am comfortable with the Custom Back-tester and know how to limit trades to a “realistic” maximum.

 

My concern is that once you reach the “clamped effect” as you put it, you would, in real life, take-out the excess (buy the Red Porche?).  However, this effect can’t really be represented in the Back-tester.

 

My concern is about evaluating systems.  If you use a “Clamped” approach and profit grows strongly then Exposure% falls and a number of Metrics reflect the change.  If you use “Compounding” approach you can move into the infeasible range (Howard Bandy put it well in a workshop I attended – 1% a day compounded for a relatively short  period and you own Manhattan).  Most approaches can be a mix of the two extremes (Clamped or Compounding) but in my mind the issue of excess profit affecting the Exposure% metric persists.

 

My fundamental worry is, does/can this issue effect the choice between alternate models (or model parameters), particularly in a Walk-Forward back-test scenario when using a complex Objective function?

 

Regards,

 

Robert  

 

From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Mike
Sent: Friday, 9 May 2008 11:23
To: amibroker@xxxxxxxxxxxxxxx
Subject: [amibroker] Re: AB Back-testing Metrics

 

You can control exactly how much is allocated to each trade, by using
custom backtester logic, without having to manipulate any statistics.

For example; I have a script that calculates position size based on a
number of factors, but then prevents the position size from exceeding
a "realistic" maximum. The end result is that you get the compounded
profits up to a maximum.

If your optimization period is a short to moderate length, the
maximum is not reached and the statistics computed reflect
your "Compounded" scenario and are valid for comparison against any
other strategy.

Though, it is true that this just delays the inevitable, which is
skewed statistics once the maximum position size is hit, at which
point the statistics begin to take on your "Clamped" effect.

I find that this most closely models what I would do in real trading
anyway. If the capital became so plentiful as to exceed a realistic
position size, I would start looking for other vehicules in which to
place the excess. I would not continue increasing the position size
beyond the realistic measure. As such, the backtested statistics
reflect reality.

Mike

--- In amibroker@xxxxxxxxxxxxxxx, "Robert Grigg" <robert@xxx> wrote:
>
> I have been thinking through the process of evaluating
the "goodness" of a
> trading system using AB metrics and have become perplexed. Can
someone who
> has unravelled this issue previously help?
>
> There seem to be two general approaches to portfolio sizing while
doing a
> back-test.
>
> The first is to only back-test using the "Initial Equity" amount.
> Generally, we might start using fixed position sizes and a fixed
maximum
> number of positions. In later developmental iterations we might use
risk
> based position sizing or other processes where we vary position
sizing up to
> the maximum amount of Initial Equity. I generally refer to this
evaluation
> approach as "Clamped Equity". This approach tends to give an
equity curve
> that is linear.
>
> The second approach is to compound profits and place trades up
to "Current
> Equity". (In AB terms our Position size is set to a % of Current
Equity).
> This is referred to as "Compounding Profits". The equity curve can
take on
> an exponential appearance.
>
> In real life trading most people tend to do a bit of both. However
in
> back-testing mode the "Compounding Profits" model (with a
notionally good
> system) can quickly become infeasible. (If only I had this system
in
> 2000...).
>
> So, now to the crux of the problem. The "Clamped Equity" approach,
with a
> notionally good system, produces a profit that is quarantined.
Accumulated
> profit can be used to top-up draw-downs but the amount in trades
never
> exceeds initial equity. In AmiBroker metrics, Exposure % is always
> calculated on a bar by bar basis of mark-to-market holding against
current
> mark-to-market equity. However, in the "Clamped Equity" testing
approach,
> the quarantining of profits is intentional and it seems to me that
it would
> be more useful to look at the Exposure as a % of the "Clamped
Equity" (i.e.
> the "Initial Equity")?
>
> Exposure% is also used as a divisor in other metrics such as Net
Risk
> Adjusted Return %, Risk Adjusted Return %, Max System % Draw-down,
> CAR/MaxDD and RAR/MaxDD and so these metrics also may be less
useful given
> this testing approach.
>
> I can see that comparisons between competing models, with the same
test
> period is valid. However, I do not feel so secure if I am doing
> Walk-Forward back-testing using a complex objective function,
particularly
> if I am using weighted components that contain Exposure% and others
that
> don't.
>
> I know that it is relatively easy to use the Custom Back Tester to
produce
> amended statistics. However, I am concerned that I have not found
any other
> discussions of this issue on this or other forums, so maybe I have
muddled
> thinking and it is not a real issue. Any discussion would be
appreciated.
>
> Robert
>

__._,_.___

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html




Your email settings: Individual Email|Traditional
Change settings via the Web (Yahoo! ID required)
Change settings via email: Switch delivery to Daily Digest | Switch to Fully Featured
Visit Your Group | Yahoo! Groups Terms of Use | Unsubscribe

__,_._,___