|
PureBytes Links
Trading Reference Links
|
Not to be a naysayer, but... I agree with Howard in principle,
however, the devil's in the details. My view: the OOS I'm concerned
with are actual trades. I personally consider existing data in sample
so applying what Howard suggested in this context is reconciling real
time results with expected results to see if the system is still
performing acceptably. And that's more difficult in practice than
developing robustness criteria and finding robust systems. IMO the
results from criteria 3-5 are sufficient to use as expected results if
done keeping the cautions I posted in 4 & 5 in mind and applying an
appropriate adjustment to the simulations as I also mentioned in that
post because, as Howard said, OOS results are almost always less
profitable. Then there are several Statistical Process Control (SPC)
techniques that, given criteria 3-5 results, will give you insight
into whether or not the system's out of control. Or, given *a lot* of
experience, one could even eyeball it. But it certainly ain't simple.
Several have commented here and privately that all this seems awfully
complicated. So instead of another Edison quote, let me suggest
looking at (find w/google) some of Caxton's research (Bruce Kovner,
first Market Wizards book, started by borrowing $3,000 and now #111 on
Forbes 400 list of wealthiest Americans).
--- In amibroker@xxxxxxxxxxxxxxx, "Gary A. Serkhoshian"
<serkhoshian777@xxxx> wrote:
> Thanks Howard. Makes sense, and seems simple to implement. With
Tomasz adding MCS into AmiBroker, life will only get sweeter : )
>
> Kind Regards,
> Gary
>
> Howard Bandy <howardbandy@xxxx> wrote:
>
> Hi Gary –
>
>
>
> I was thinking of looking at the recent trades in the out of sample
period. We can get an idea of what the possible distribution of
various metrics are by looking at the in sample results. But the out
of sample results are (almost) always less profitable, have a lower
ratio of wins to losses, etc than the in sample results. One
technique I use is to run a quick and dirty monte carlo program I
wrote in Basic that gives the likelihood of various metrics –
such as
the proportion of winning versus losing trades. If the out of sample
results start falling in the area of the distribution that is
"unlikely", then I have a warning that the system may be
broken.
>
>
>
> Howard
>
>
>
> -----Original Message-----
> From: Gary A. Serkhoshian [mailto:serkhoshian777@x...]
> Sent: Sunday, November 09, 2003 11:33 AM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: RE: [amibroker] On Robustness, Post #1 : TO HOWARD
>
>
>
> Howard,
>
>
>
> Thanks for the detailed response. Helpful as always.
>
>
>
> Regarding your comment:
>
>
>
> Other techniques could be comparison of various metrics of recent
trades with the probabilities that those results come from a system
that is healthy or broken.
>
>
>
> Can we come to this conclusion by looking at frequency
distributions of the metrics in question during the IS period?
>
>
>
> Thanks again,
>
> Gary
>
>
>
>
>
> Howard Bandy <howardbandy@xxxx> wrote:
>
> Hi Gary –
>
>
>
> One -- Yes, the presidential election cycle has strong biases.
This year, year 3, is traditionally an up year. I mentioned the
presidential cycle as an example to ward off the flames that might
come from suggesting a crystal ball approach to model selection.
>
>
>
> Two -- I have done quite a bit of research into use of analysis of
the equity curve as a feedback mechanism to help determine the health
of a system. Other techniques could be comparison of various metrics
of recent trades with the probabilities that those results come from a
system that is healthy or broken.
>
>
>
> Three -- As I have mentioned in posts to this board and to
HolyGrailSM, I believe several things are true of markets and systems.
Not everyone on this list agrees with me on these points, so
you'll
read some other opinions..
>
> Entries and exits need not be symmetric. In the equity markets,
drops are much steeper in slope than rises, so the parameters used to
recognize them in the same number of bars are different.
> A good system need not trade all, or even a large portion, of
tradables well.
> Markets change dramatically over time. It is very difficult to
design a system that trades profitably over a long time period,
particularly when the market characteristics change within that
period.
> Systems that once worked well, then fail, will probably not work
well again.
>
>
>
> Howard
>
>
>
> -----Original Message-----
> From: Gary A. Serkhoshian [mailto:serkhoshian777@x...]
> Sent: Wednesday, November 05, 2003 9:15 PM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: RE: [amibroker] On Robustness, Post #1 : TO HOWARD
>
>
>
> Howard,
>
>
>
> A few questions regarding your post to Dave
>
>
>
> < One – we design two systems, one for bullish periods, the
other
for bearish periods. Then we look into our crystal ball and use the
system with the upward bias if we have some idea that the near future
will be bullish, and use the system with the downward bias if we have
some idea that the near future will be bearish. This is not wholly a
dream. For example, there are strong seasonalities in the US four
year presidential cycle. >
>
>
>
> Regarding the 4-year cycle, are you specifically referring to year 3
(the year that is coming to an end) ? I've read year 3 since WWII has
been profitable to the tune of 15% on avg. Any other cycles you'd
suggest to follow?
>
>
>
> < Two – we design two systems, one for bullish periods, the
other
for bearish periods, and include failsafe mechanisms in both. Then we
trade both systems and let the system that works make money while the
system that doesn't work recognizes that it doesn't work and
stays flat. >
>
>
>
> Which failsafe mechanisms do you prefer? I've been looking at a DD
"floor" or perhaps a factor of MaxDD to turn off the system. I
believe Dimitris has suggested a downslope in the 100MA of equity
curve which makes sense, too.
>
>
>
> < Three – we design a system that is profitable in both bull and
bear periods. I think this is the hardest to do, since the markets
act so differently that it requires additional parameters to be able
to recognize the additional patterns. In my experience, systems
designed to do well in both bullish and bearish periods do not do
exceptionally well in either period >
>
>
>
> What you're describing is essentially that buys and shorts can not
be symmetrical. Is that right? What are the primary things that
differentiate up moves from down moves that require the need for
asymmetry of signals.
>
>
>
> Thanks for the post, as this subject is scratching where I itch in
my eduction of system development and optmization.
>
>
>
> Kind Regards,
>
> Gary
>
>
>
> Howard Bandy <howardbandy@xxxx> wrote:
>
> Hi Dave –
>
>
>
> Good posting. I'd like to comment on your last paragraph.
>
>
>
> - if one system does better in bull years and another in bear, the
one that
> does better in reality will depend on the proportion of bull and
bear years
> that actually occur. when we weight bull, bear and sideways markets
equally,
> are we matching their proportions in real life? what time frame
would we
> want to base that judgment on?
>
> It seems there are three approaches to take.
>
>
>
> One – we design two systems, one for bullish periods, the other
for
bearish periods. Then we look into our crystal ball and use the
system with the upward bias if we have some idea that the near future
will be bullish, and use the system with the downward bias if we have
some idea that the near future will be bearish. This is not wholly a
dream. For example, there are strong seasonalities in the US four
year presidential cycle.
>
>
>
> Two – we design two systems, one for bullish periods, the other
for
bearish periods, and include failsafe mechanisms in both. Then we
trade both systems and let the system that works make money while the
system that doesn't work recognizes that it doesn't work and
stays flat.
>
>
>
> Three – we design a system that is profitable in both bull and
bear
periods. I think this is the hardest to do, since the markets act so
differently that it requires additional parameters to be able to
recognize the additional patterns. In my experience, systems designed
to do well in both bullish and bearish periods do not do exceptionally
well in either period.
>
>
>
> Howard
>
>
>
> -----Original Message-----
> From: Dave Merrill [mailto:dmerrill@x...]
> Sent: Monday, November 03, 2003 7:06 AM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: RE: [amibroker] On Robustness, Post #1
>
>
>
> some robustness issues that have been rattling around in my head
over the
> weekend...
>
> <<<SNIP>>>
>
>
>
>
>
> Send BUG REPORTS to bugs@xxxx
> Send SUGGESTIONS to suggest@xxxx
> -----------------------------------------
> Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
> (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> --------------------------------------------
> Check group FAQ at:
http://groups.yahoo.com/group/amibroker/files/groupfaq.html
>
> Your use of Yahoo! Groups is subject to the Yahoo! Terms of
Service.
>
>
>
>
>
>
> Howard Bandy <howardbandy@xxxx> wrote:
>
> Hi Dave –
>
>
>
> Good posting. I'd like to comment on your last paragraph.
>
>
>
> - if one system does better in bull years and another in bear, the
one that
> does better in reality will depend on the proportion of bull and
bear years
> that actually occur. when we weight bull, bear and sideways markets
equally,
> are we matching their proportions in real life? what time frame
would we
> want to base that judgment on?
>
> It seems there are three approaches to take.
>
>
>
> One – we design two systems, one for bullish periods, the other
for
bearish periods. Then we look into our crystal ball and use the
system with the upward bias if we have some idea that the near future
will be bullish, and use the system with the downward bias if we have
some idea that the near future will be bearish. This is not wholly a
dream. For example, there are strong seasonalities in the US four
year presidential cycle.
>
>
>
> Two – we design two systems, one for bullish periods, the other
for
bearish periods, and include failsafe mechanisms in both. Then we
trade both systems and let the system that works make money while the
system that doesn't work recognizes that it doesn't work and
stays flat.
>
>
>
> Three – we design a system that is profitable in both bull and
bear
periods. I think this is the hardest to do, since the markets act so
differently that it requires additional parameters to be able to
recognize the additional patterns. In my experience, systems designed
to do well in both bullish and bearish periods do not do exceptionally
well in either period.
>
>
>
> Howard
>
>
>
> -----Original Message-----
> From: Dave Merrill [mailto:dmerrill@x...]
> Sent: Monday, November 03, 2003 7:06 AM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: RE: [amibroker] On Robustness, Post #1
>
>
>
> some robustness issues that have been rattling around in my head
over the
> weekend...
>
> <<<SNIP>>>
>
>
>
>
>
>
> Send BUG REPORTS to bugs@xxxx
> Send SUGGESTIONS to suggest@xxxx
> -----------------------------------------
> Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
> (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> --------------------------------------------
> Check group FAQ at:
http://groups.yahoo.com/group/amibroker/files/groupfaq.html
>
> Your use of Yahoo! Groups is subject to the Yahoo! Terms of
Service.
>
> ---------------------------------
>
>
> Do you Yahoo!?
> Protect your identity with Yahoo! Mail AddressGuard
>
> Send BUG REPORTS to bugs@xxxx
> Send SUGGESTIONS to suggest@xxxx
> -----------------------------------------
> Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
> (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> --------------------------------------------
> Check group FAQ at:
http://groups.yahoo.com/group/amibroker/files/groupfaq.html
>
> Your use of Yahoo! Groups is subject to the Yahoo! Terms of
Service.
>
>
>
>
>
> Send BUG REPORTS to bugs@xxxx
> Send SUGGESTIONS to suggest@xxxx
> -----------------------------------------
> Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
> (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> --------------------------------------------
> Check group FAQ at:
http://groups.yahoo.com/group/amibroker/files/groupfaq.html
>
> Your use of Yahoo! Groups is subject to the Yahoo! Terms of
Service.
>
> ---------------------------------
>
>
> Do you Yahoo!?
> Protect your identity with Yahoo! Mail AddressGuard
>
>
> Send BUG REPORTS to bugs@xxxx
> Send SUGGESTIONS to suggest@xxxx
> -----------------------------------------
> Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
> (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> --------------------------------------------
> Check group FAQ at:
http://groups.yahoo.com/group/amibroker/files/groupfaq.html
>
> Your use of Yahoo! Groups is subject to the Yahoo! Terms of
Service.
>
>
>
>
>
>
>
> Yahoo! Groups SponsorADVERTISEMENT
>
> Send BUG REPORTS to bugs@xxxx
> Send SUGGESTIONS to suggest@xxxx
> -----------------------------------------
> Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
> (Web page: http://groups.yahoo.com/group/amiquote/messages/)
> --------------------------------------------
> Check group FAQ at:
http://groups.yahoo.com/group/amibroker/files/groupfaq.html
>
> Your use of Yahoo! Groups is subject to the Yahoo! Terms of
Service.
>
>
> ---------------------------------
> Do you Yahoo!?
> Protect your identity with Yahoo! Mail AddressGuard
------------------------ Yahoo! Groups Sponsor ---------------------~-->
Buy Ink Cartridges or Refill Kits for your HP, Epson, Canon or Lexmark
Printer at MyInks.com. Free s/h on orders $50 or more to the US & Canada.
http://www.c1tracking.com/l.asp?cid=5511
http://us.click.yahoo.com/mOAaAA/3exGAA/qnsNAA/GHeqlB/TM
---------------------------------------------------------------------~->
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
|