RE: [amibroker] Re: On Robustness, Post #1 : TO HOWARD

["Howard Bandy" <howardbandy@xxxxxxxxx>]

Hi Mark –

 

Thanks for the confirmation.  There
is one point in your posting that I’d like to comment on.

 

&#8220;Then there are several Statistical Process Control (SPC)<font
size=2 face="Courier New">
techniques that, given criteria 3-5 results, will
give you insight
into whether or not the system's out of
control. &#8220;

 

<span
>I agree that there are
SPC techniques that can apply to evaluation of trading systems.  But it is
important to keep in mind that we are dealing with non-stationary, time
series.  Many of the SPC techniques were developed for stationary,
non-time series, which are Much easier to model and Much easier to monitor.

<span
> 

<span
>Howard  

 



-----Original Message-----
From: quanttrader714
[mailto:quanttrader714@xxxxxxxxx] 
Sent: Monday, November 10, 2003
9:44 AM
To: amibroker@xxxxxxxxxxxxxxx
Subject: [amibroker] Re: On
Robustness, Post #1 : TO HOWARD

 

Not to be a naysayer, but...  I agree with Howard in principle,
however, the
devil's in the details.  My view: the OOS I'm concerned
with are
actual trades.  I personally consider existing data in sample
so applying
what Howard suggested in this context is reconciling real
time results
with expected results to see if the system is still
performing
acceptably.  And that's more difficult in practice than
developing
robustness criteria and finding robust systems.  IMO the
results from
criteria 3-5 are sufficient to use as expected results if
done keeping
the cautions I posted in 4 & 5 in mind and applying an
appropriate
adjustment to the simulations as I also mentioned in that
post
because, as Howard said, OOS results are almost always less
profitable. 
Then there are several Statistical Process Control (SPC)
techniques
that, given criteria 3-5 results, will give you insight
into whether
or not the system's out of control.  Or, given *a lot* of
experience,
one could even eyeball it.  But it certainly ain't simple.

Several have
commented here and privately that all this seems awfully
complicated. 
So instead of another Edison quote, let me suggest
looking at
(find w/google) some of Caxton's research (Bruce Kovner,
first Market
Wizards book, started by borrowing $3,000 and now #111 on
Forbes 400
list of wealthiest Americans).  

--- In
amibroker@xxxxxxxxxxxxxxx, "Gary A. Serkhoshian"
<serkhoshian777@xxxx>
wrote:
> Thanks
Howard.  Makes sense, and seems simple to implement.  With
Tomasz
adding MCS into AmiBroker, life will only get sweeter : )
>  
> Kind
Regards,
> Gary
> 
> Howard
Bandy <howardbandy@xxxx> wrote:
> 
> Hi Gary
&#8211;
> 
>  
> 
> I was
thinking of looking at the recent trades in the out of sample
period. 
We can get an idea of what the possible distribution of
various
metrics are by looking at the in sample results.  But the out
of sample
results are (almost) always less profitable, have a lower
ratio of
wins to losses, etc than the in sample results.  One
technique I
use is to run a quick and dirty monte carlo program I
wrote in
Basic that gives the likelihood of various metrics &#8211;
such as
the
proportion of winning versus losing trades.  If the out of sample
results
start falling in the area of the distribution that is
"unlikely",
then I have a warning that the system may be
broken.
> 
>  
> 
> Howard
> 
>  <font
color=navy><<<<SNIP>>>>












Yahoo! Groups Sponsor


  ADVERTISEMENT 









Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx 
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html



Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.