|
PureBytes Links
Trading Reference Links
|
Hello Tomasz,
Unbelievable!!!!
What took me hours (and even days), your method has reduced to
minutes!!!!
(For anyone who is reading this, in order to concatenate many files
into one, I suggest using the dos Copy command, with * as wild card;
also use the Dos copy command to sort that huge file - it does it in
seconds.)
Thank you again - another happy customer!
--- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@xxx>
wrote:
>
> Hello,
>
> Your response confirms my guesses: you have separate files per
each day
> which is the "worst case".
>
> To speed import up you should join all your files into SINGLE file
> and then sort by ticker. If it ticker is first column of your data
file
> - it would be just alphabetic sorting.
>
> The other way to speed it up (but less) is to go to Tools-
>Preferences
> "Data" and make sure that in memory cache is greater than
> maximum number of symbols you import.
>
> Otherwise it needs to swap to the disk all symbols for all days
you are importing.
>
> Best regards,
> Tomasz Janeczko
> amibroker.com
> ----- Original Message -----
> From: "jacklweinberg" <jacklweinberg@xxx>
> To: <amibroker@xxxxxxxxxxxxxxx>
> Sent: Tuesday, May 15, 2007 5:26 PM
> Subject: [amibroker] Re: Massive data loading?
>
>
> > Tomasz,
> > I think that when "Import Files" is sued, it sorts my selection
(I
> > select an entire years worth of data files at once). My file
names
> > are like this: "nasdaq_20010201.txt" (that is with yyyymmdd in
the
> > name). Is AmiBroker sorting those file names? or does it import
in
> > the order that was selected?
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@>
> > wrote:
> >>
> >> Hello,
> >>
> >> 5 years of data on my end imports in *minutes* not hours.
> >>
> >> a) you have small size of "in-memory" cache so it requires a
lot
> > of disk mangling
> >> b) the data you import are not sorted or sorted improperly (in
> > reverse order) and AB needs to re-sort.
> >>
> >> For fast import It is advised to use data that is sorted in
OLDEST
> > QUOTES FIRST - NEWEST LAST
> >> and to have one file per symbol.
> >>
> >> The worst possible case (it's probably yours) is having sorted
in
> > opposite direction
> >> (newest first) and different symbol in every line.
> >>
> >> Best regards,
> >> Tomasz Janeczko
> >> amibroker.com
> >> ----- Original Message -----
> >> From: "jacklweinberg" <jacklweinberg@>
> >> To: <amibroker@xxxxxxxxxxxxxxx>
> >> Sent: Tuesday, May 15, 2007 2:59 PM
> >> Subject: [amibroker] Re: Massive data loading?
> >>
> >>
> >> > Tomasz,
> >> > Perhaps my original message wasn't clear.
> >> > I have the data in txt files, and am using AmiBroker "Import
> > Files"
> >> > to load them into the database. That is the process that is
> > taking
> >> > over 32 hours!
> >> >
> >> > --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@>
> >> > wrote:
> >> >>
> >> >> Hello,
> >> >>
> >> >> It is MUCH quicker if you change the settings.
> >> >> Default settings are fail-safe. You can however
> >> >> change "number of simultaneous downloads" to 5 or more.
> >> >> BY setting it say to 10 you can get 10x speed up,
> >> >> however Yahoo may block you after some excessive downloading
> >> >> and you will need to revert to defaults and wait a couple of
> >> > minutes
> >> >> for Yahoo to unblock you.
> >> >>
> >> >> Best regards,
> >> >> Tomasz Janeczko
> >> >> amibroker.com
> >> >> ----- Original Message -----
> >> >> From: "jacklweinberg" <jacklweinberg@>
> >> >> To: <amibroker@xxxxxxxxxxxxxxx>
> >> >> Sent: Monday, May 14, 2007 7:33 PM
> >> >> Subject: [amibroker] Massive data loading?
> >> >>
> >> >>
> >> >> > Does anyone have experience at loading up an entirely new
> >> > database
> >> >> > with AmiBroker?
> >> >> > For example, loading AB with all of Nasdaq with 5 years of
> > data?
> >> >> > I used the file import command, and selected a series
of .txt
> >> > files,
> >> >> > and it takes around 32 hours of elapsed time per year!!!
(On
> > a
> >> > fast
> >> >> > machine, no less!)
> >> >> > Does anyone know of a quicker method?
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > Please note that this group is for discussion between
users
> > only.
> >> >> >
> >> >> > To get support from AmiBroker please send an e-mail
directly
> > to
> >> >> > SUPPORT {at} amibroker.com
> >> >> >
> >> >> > For NEW RELEASE ANNOUNCEMENTS and other news always check
> > DEVLOG:
> >> >> > http://www.amibroker.com/devlog/
> >> >> >
> >> >> > For other support material please check also:
> >> >> > http://www.amibroker.com/support.html
> >> >> >
> >> >> > Yahoo! Groups Links
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> >
> >> > Please note that this group is for discussion between users
only.
> >> >
> >> > To get support from AmiBroker please send an e-mail directly
to
> >> > SUPPORT {at} amibroker.com
> >> >
> >> > For NEW RELEASE ANNOUNCEMENTS and other news always check
DEVLOG:
> >> > http://www.amibroker.com/devlog/
> >> >
> >> > For other support material please check also:
> >> > http://www.amibroker.com/support.html
> >> >
> >> > Yahoo! Groups Links
> >> >
> >> >
> >> >
> >> >
> >> >
> >>
> >
> >
> >
> >
> > Please note that this group is for discussion between users only.
> >
> > To get support from AmiBroker please send an e-mail directly to
> > SUPPORT {at} amibroker.com
> >
> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> > http://www.amibroker.com/devlog/
> >
> > For other support material please check also:
> > http://www.amibroker.com/support.html
> >
> > Yahoo! Groups Links
> >
> >
> >
> >
> >
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|