|
PureBytes Links
Trading Reference Links
|
Thank you Brian for your excellent post. If you don't mind sometime do
post your list of other rocks that traders dash their ships in.
Cheers,
AV
--- In amibroker@xxxxxxxxxxxxxxx, "brian_z111" <brian_z111@xxx> wrote:
>
> Now that you have got me thinking about the subject I have decided to
> pencil in some new rules to my 'little green book':
>
> - datamining is a fancy name for 'tuning our system to a dataset'
> - anytime we change one system rule, by any amount, based on data
> feedback, we are tuning, even if that dataset is produced in live
> trading
> - the best test, of a robust system, is when we submit it to a
> dataset, that is unknown to the system, without any changes to the
> rules, and the variance in the outcomes is low, when compared to the
> previous test (providing the test samples are > 3-400 trades at the
> least).
>
> Hope that clarifies it for you.
>
> brian_z
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, "brian_z111" <brian_z111@> wrote:
> >
> > Hello Simon,
> >
> > Great question.
> >
> > I have an interest in Single Sample Testing (SST) and pushing the
> > boundaries there. It is a big NO, NO to the 'defenders of the
> faith'.
> >
> > I also have a strong bias to simple systems. No, or few, indicators
> > with lookback periods etc (I don't use many rules/lose degrees of
> > freedom) hence my interest in the subject.
> >
> > My gut feeling tells me I can do it but I haven't got far with the
> > proof (however that doesn't mean much since there are terabytes of
> > books and academic research, out there, that I am totally unaware
> of).
> >
> > Personally, I think SST only has academic interest.
> > I am following it because I am curious, I learn from the enquiry
> and
> > I love to confound my critics.
> >
> > So, possibly your friend is correct but if s/he is absolutely
> certain
> > about it s/he would be capable of writing a book on evaluation - in
> > fact if that is the case, I wish s/he would, thereby saving me a
> lot
> > of time and trouble.
> >
> > Anyway, over to the here and now.
> >
> > > My question is, does anyone know if the data-mining bias can be
> > > considered irrelvant when the sample size is so large? (in this
> > >case,
> > > the sample size is roughly 8400 trades).
> >
> > Possibly I can ride my motorbike, at 200mph, going the wrong way up
> a
> > 6 lane highway but what is the point if I just want to get from A
> to
> > B - am I going somewhere or thrill seeking?
> >
> > Here are some rules from my notebook:
> >
> > - good data, relevant to current conditions, is scarce. Why waste
> it?
> > - sample error is real
> > - around 300 to 400 trades is the minimum, with no further
> > substantial minimization of sample error beyond, around 10,000
> > - there is a sweet spot around 1,000 - 5,000 trades
> > - if data is short then work with no less than 3-400
> > - if data is in plentiful supply (intraday?) then use more
> > - one sample might be good enough (in exceptional circumstances/for
> > exceptional traders) but why not reduce risk and use more (if you
> > have the data)
> > - 1 IS and 1 OOS is better than 1 IS alone
> > - even though I am interested in SST, and more likely than most to
> > succeed with it, I am actually using several OOS, of optimum
> length,
> > whenever I can.
> >
> > No, 8400 trades, in a single IS test, does not guarantee success
> (it
> > is very easy to find rare cases, on a computer, because we can work
> > our way through such large datasets in a relatively short space of
> > time - 1 in a million chance in real life === 1 in a backtest
> chance
> > on a computer).
> >
> > We can't rely on stats alone - they never give a definitive answer.
> >
> > Different story if your friend has observed a persistent, and
> > predictable, market inefficiency and the stats are just confirming
> > and quantifying that.
> >
> > >Put another way, with so many
> > > observations, how many different rules would have to be back
> tested
> > >in
> > > order for data-mining bias to creep in?
> >
> > I am still mulling over this point.
> >
> > What is the least number of rules that a useful system could be
> > described in? Perhaps three rules would be the least that anyone is
> > successfully using (I don't know - I am wondering how many is the
> > least possible).
> >
> > Say I have a system with only three rules - if I test it IS and
> > change 1 rule a little bit I am still tuning the system to that
> data,
> > aren't I?
> >
> > If I have a system with only three rules, test in IS, and it is
> > successful, then test it OS and it is successful, all I am doing is
> > confirming that the system is tuned to those two particular
> datasets,
> > aren't I .
> >
> > Based on those observations I would say that, since we can't avoid
> > data mining, even with simplistic methods, then we are always 'data
> > mining' when we use historical data.
> >
> > The only time we are not datamining is when we are live trading.
> >
> > OOS testing is the historical surrogate for live trading, in that
> at
> > least the data is unknown, to the system, prior to walkforward or
> OOS.
> >
> > The only thing about datamining that varies, when we are using
> > historical data, is the degree.
> >
> > The more rules + the greater the range of adjustble parameters
> within
> > the rules == the more likely we are to be 'fooled by randomness'.
> >
> > In short - no matter what we do we can never achieve 100% certainty
> > but OOS and live paper trading will minimize the risk compared to
> SST
> > alone.
> >
> > Some food for thought:
> >
> > Data mining, per se, is not the only thing on the list of 'rocks
> that
> > traders dash their ships on' - there's more on the same list (most
> of
> > them receive a lot less publicity).
> >
> > brian_z
> >
> > brian_z
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "si00si00" <si00si00@> wrote:
> > >
> > > Hi all,
> > >
> > > I have a friend who has developed a trading system. It is an
> > intraday
> > > system that makes on average around 5 futures trades per day. We
> > were
> > > discussing it the other day and a point of disagreement arose
> > between
> > > us. He claims that there is no necessity for him to test the
> > strategy
> > > on out of sample data because he has back tested it using over 8
> > years
> > > of historical intraday data, and the patterns the strategy
> predicts
> > > occur 70% of the time or more.
> > >
> > > My question is, does anyone know if the data-mining bias can be
> > > considered irrelvant when the sample size is so large? (in this
> > case,
> > > the sample size is roughly 8400 trades). Put another way, with so
> > many
> > > observations, how many different rules would have to be back
> tested
> > in
> > > order for data-mining bias to creep in?
> > >
> > > Thanks in advance for any thoughts you might have!
> > >
> > > Simon
> > >
> >
>
------------------------------------
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|