|
PureBytes Links
Trading Reference Links
|
Data management isn't very exciting and so doesn't find its way into
the textbooks but it is a big part of trading.
Ticks are the natural data points of the markets - electronically
recorded - matched to regulated transactions -everything else is a
construct of convenience.
Volume is going through the roof and the exchanges can barely keep up.
Some stocks will have no ticks per second and some many.
Data vendors take the ticks live from the exchanges and pass them on
to us or third party vendors.
No ticks create a 'hole' in the data - the data also contains errors -
missing or incorrect ticks OR missing/incorrect bars at the EOD
level.
At the EOD the exchanges release a daily tick file with corrections
etc - (this is the ASX but the format is similar around the world
AFAIK) - brokers/regulators etc can reverse or correct trades up
ntil the start of trade the next day or sometimes beyond that - the
change file for the ASX is appended to the tick file for the next day
in the premarket time slot, which mucks up the next day 24 hours vols
etc, so vendors have to 'correct' stuff like that before we get it.
Corrections are handled in different ways - say an EOD provider might
wait until 4 hours after close to release the EOD data - in some
cases the exchanges might release a daily file or the vendors have to
compile that themselves, from the corrected tick files - some vendors
put more effort into corrections etc than others e.g. eSignal have
intraday corrections and a tick filter (removes bad ticks) - EOD
vendors don't necessarily have to take the live ticks - tahn can wait
for the daily tick file that comes after the close.
All other timeframes are based on the ticks - they are compressed
into bars (from 1 sec up) by the exchanges or the vendors.
The min timeframe varies from vendor to vendor e.g. IB has 5 secs as
the minimum, or something like that - so they don't do ticks.
IB gives free RT data to account holders who trade a bit - the main
attraction of IB, for AB users, is to AutoTrade or for nonUS traders
to get access to the US market (foreigners aren't allowed to open on
shore accounts) or to get access to the international markets, which
some US vendors/brokers don't have.
What is the best intraday timeframe?
There is no correct answer - some traders use ticks, some use other
timeframes.
Best not to download a complete database at first - take some tick
data for a few stocks and play with the timeframes until you find out
what you like - theoretically ticks are the best because you can go
up or down to any level you want - limiting the database to say, 1
min bars, is a trade off to limit the size and contain management
issues.
If you slide the timeframe around the point at which the holes in the
data becomes visible to the naked eye changes, depending on how
lightly the stock is traded.
I am basically a 'chart reader' so I am fussy about my view -
sometimes I want to see the holes but most of the time I don't - I
will use the lowest timeframe that doesn't produce gaps in the bars -
it is like zooming in and out with a magnifying glass - you see more
detail at a finer grain but can you interpret it?
Around 3-5 mins is probably the most popular.
Is it reasonable to use 2 * 500 downloads == 1000?
Some people use extreme amounts of symbols/historic data.
I am not sure about the logic of the symbol limits, as set by the
providers, or how they police it.
In the RT Quotes window you can only use the limit but there are
programs like QCollector that can download RT from the IQ or eS
server and if the downloader only takes one symbol at a time, and
walks through a long ticker list, how does their server respond to
that - perhaps it varies?
www.mechtrading.com
I am not an extreme data user so I haven't pushed eS to the limits
and don't know the answer to that question.
I have:
created a new 1 min database - enable local
ASCII imported thousands of symbols (don't click on them or they will
start to backfill the selected symbol)
put 200 at a time into watchlists (by importing ticker lists ---> WL)
opened AA and run a scan using
- filter defined and pick a watchlist
- n quotes == 1
- formula is buy = sell = 1;//to process all of the symbols
- checkbox wait for RT backfill
- scan
it takes approx 30 mins to backfill 200 symbols around the eS limit
of say 40000 at 1 min bars
(when you create the database AB gives you a hint about
memory/barcount when you enter bars to download)
Note with local enabled you can disconnect the plugin from the server
and still read the data from your disc.
I batch 200 at a time but I think it will actually do mega lists
since it is downloading one symbol at a time.
You can plot barindex() as an indicator to give you an idea about how
many bars you got for each symbol - it varies depending on how long
the stock has been around - and how heavily it is traded.
How many bars for backtest?
My criteria is that the backtest needs to produce a statistically
valid number of closed trades InSample and then leave enough data for
a the same OS but everyhone does it differently - so it varies from
system to system etc
Whats a good strategy?
No trader will handover their good strategies in public.
The most I will do is give veiled hints about strategies that work,
or are likely to work, and that I am no longer use personally
(because I have some that are better).
Books are the best place to start - value for money - not many actual
winners there but still more bang for your buck than buying systems
or attending seminars/training.
RT and 'massive data' are honeypots for newcomers.
RT brings additional headaches.
I found it easier to learn in EOD for the first year or two.
You can save money on data for a while too because realistically it
takes a while to learn the basics.
brian_z
--- In amibroker@xxxxxxxxxxxxxxx, "tpellaton" <tod@xxx> wrote:
>
> Thanks for the insight. Less confusion now.
>
> It appears that minute or other data can be accumulated just the
same
> as EOD data. With IQ feed they claim 500 intraday symbols with 8
> months backfill. So to get an 8 month intraday (1 minute) data base
> to back test with a stategy might be to reduce my universe of
stocks
> to maybe 1000 stocks and then split that into 2 watchlists of 500
> tickers each so it can be downloaded and then updated. Does this
> sound like a reasonable approach to using intra day data?
>
> Wow. Lots of data:
>
> 6.5 hours/day x 60 min bars/hr =390bars /day/ticker
>
> 390 bars/day/ ticker x 1000 tickers = 390,000 bars/day
>
> 390,000 bars/day x 10 bytes/bar( estimate?) = 3,900,000 bytes/day
=
> 4mb /day
>
> 4mb /day x 250 day/yr = 1 gig/yr of data with 1000 tickers
>
> So 8 months of 1 minute data is a big download and would make for
> very long back tests. It would be nice to have the one minute data
> but maybe 30 or 60 minute bars would be a more reasonable time
> selection for back testing.
>
> Is 8 months reasonable for 30 minute intraday bar back tests? I use
8
> years for EOD back testing which will show performance through full
> bear and bull market cycles. What types intra day data / time
frame
> strategies are some of you using?
>
> Tod Pellaton
>
>
>
>
> --- In amibroker@xxxxxxxxxxxxxxx, "brian_z111" <brian_z111@> wrote:
> >
> > Hello Tod,
> >
> > > Looking at IQ feed or starting an account at IB. Both of these
> data
> > > suppliers have limits on simultaneous data you can have and
> > >backfill data
> > > days? Not sure what this means.
> >
> > Basically - every bar == a set amount of bytes download adn
memory
> to
> > store.
> > 1 EOD bar == 60 * 1 minute bars (it has OHLCV just the same)
> > therefore 60 times the download.
> >
> > The providers allow so many symbols * so many bars == a set
> download
> > (if you use a smaller timeframe then you can't go back so far
> before
> > exhausting the limit).
> >
> > The details vay but this is the theme.
> >
> > If you want more symbols then you pay more.
> >
> >
> > > I guess I just do not have a good understanding of how RT data
is
> > >gathered
> > > and used as compared to EOD data
> >
> > Basically every transaction is recorded by the exchanges = 10000
> > bought at $30 == a tick - in the leading stocks the ticks fly -
the
> > ticks are 'compressed' to create, say a one minute bar (AB can
> handle
> > tick data and you can view any timeframe you want up to the daily
> bar
> > which is then compressed to weekly or monthly.
> >
> > I have eSignal RT - I don't use below one minute so I set that as
> my
> > database timeframe.
> >
> > If I look at a 24 hour chart of a leading stock it will trade
> slowly
> > after hours and premarket.
> >
> > If I look at the opening bar, based on the open time of the
> exchange
> > where the stock trades and the same for the closing one minute
bar
> > (measured by time) i.e. say tge 4pm or 1600 hour one minute bar
> then
> > the close of this bar will be the close of the daily bar.
> >
> > The daily bar == the one minute bars from market open
> > hours 'compressed' into 1 bar.
> >
> > RE backtesting:
> >
> > With eSignal I have a 200 symbol limit so I can take the max
> history
> > for the timeframe I want - say 6 months for 1 minute - and it is
> > saved locally by AB.
> >
> > Then I can do another 200 symbols etc.
> >
> > I am not a big user of back data compared to others and this is
> > enough for me but everyone is different - some do go through a
lot
> > more symbols and depth.
> >
> > I bought a CD with ASX tick data history on it and manually
> imported
> > it into AB to use for backtesxting - it was a PITA to import -
very
> > slow and I had to manually clean out data errors - not
recommended.
> >
> > I use eSignal because I live outside the US and they are the best
> > choice for multiple international markets.
> >
> > IB also have international market data but I have held off
opening
> an
> > account there for personal reasons - their data isn't quite as
good
> > as eSignal.
> >
> > eSignal is just a matter of following the help manual - the
plugin
> > works like a dream as per the notes.
> >
> > Plenty of US citzs seem happy with their RT data from the other
> main
> > supported providers (IQ or QP).
> >
> > Pity you don't live in Aus because locals can ring the eSignal
Aus
> > desk and get the 30 day trial for free (except exchange fees)-
> AFAIK
> > if you continue on with eSignal US the 30 day trial is back
charged
> > once you sign up.
> >
> > I didn't answer all of your questions - others might help with
the
> > rest.
> >
> > brian_z
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "Tod Pellaton" <tod@> wrote:
> > >
> > > I am looking for help in clarifying how to set up and use Real
> time
> > data.
> > > Currently using worden EOD data and can program very simple
> systems
> > and back
> > > tests, etc.
> > >
> > >
> > >
> > > I guess I just do not have a good understanding of how RT data
is
> > gathered
> > > and used as compared to EOD data. I have looked through the
> > documentation
> > > but am confused how it all works. It looks easy enough to set
up
> a
> > data
> > > supplier but confusing how to best utilize RT data.
> > >
> > >
> > >
> > > Looking at IQ feed or starting an account at IB. Both of these
> data
> > > suppliers have limits on simultaneous data you can have and
> > backfill data
> > > days? Not sure what this means. With EOD data I just download
> data
> > for all
> > > US stocks every day.
> > >
> > >
> > >
> > > Is it realistic to download 1 minute data for all stocks
everyday
> > to build a
> > > database for back testing? Are these data suppliers set up to
> > provide us
> > > with this much data every day? Should a smaller universe of
> stocks
> > be
> > > specified to work with to reduce the amount of data? Is it
> > possible to
> > > obtain a few years of data for a universe of stocks for back
> > testing systems
> > > with real time data? How does AB manage all this data
gathering?
> Is
> > it
> > > realistic to think I will need to build a RT database for a
> > universe of
> > > stocks in AB over time? Is there a special group of AFL
commands
> > in AB I
> > > should learn about to work with RT data? What methods are
> currently
> > being
> > > used to gather and mange RT Data?
> > >
> > >
> > >
> > > Any clarification or insight on this topic would be
appreciated.
> > >
> > >
> > >
> > >
> > >
> > > Tod Pellaton
> > >
> >
>
------------------------------------
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|