|
PureBytes Links
Trading Reference Links
|
Hello,
Your response confirms my guesses: you have separate files per each day
which is the "worst case".
To speed import up you should join all your files into SINGLE file
and then sort by ticker. If it ticker is first column of your data file
- it would be just alphabetic sorting.
The other way to speed it up (but less) is to go to Tools->Preferences
"Data" and make sure that in memory cache is greater than
maximum number of symbols you import.
Otherwise it needs to swap to the disk all symbols for all days you are importing.
Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: "jacklweinberg" <jacklweinberg@xxxxxxxxx>
To: <amibroker@xxxxxxxxxxxxxxx>
Sent: Tuesday, May 15, 2007 5:26 PM
Subject: [amibroker] Re: Massive data loading?
> Tomasz,
> I think that when "Import Files" is sued, it sorts my selection (I
> select an entire years worth of data files at once). My file names
> are like this: "nasdaq_20010201.txt" (that is with yyyymmdd in the
> name). Is AmiBroker sorting those file names? or does it import in
> the order that was selected?
>
> --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@xxx>
> wrote:
>>
>> Hello,
>>
>> 5 years of data on my end imports in *minutes* not hours.
>>
>> a) you have small size of "in-memory" cache so it requires a lot
> of disk mangling
>> b) the data you import are not sorted or sorted improperly (in
> reverse order) and AB needs to re-sort.
>>
>> For fast import It is advised to use data that is sorted in OLDEST
> QUOTES FIRST - NEWEST LAST
>> and to have one file per symbol.
>>
>> The worst possible case (it's probably yours) is having sorted in
> opposite direction
>> (newest first) and different symbol in every line.
>>
>> Best regards,
>> Tomasz Janeczko
>> amibroker.com
>> ----- Original Message -----
>> From: "jacklweinberg" <jacklweinberg@xxx>
>> To: <amibroker@xxxxxxxxxxxxxxx>
>> Sent: Tuesday, May 15, 2007 2:59 PM
>> Subject: [amibroker] Re: Massive data loading?
>>
>>
>> > Tomasz,
>> > Perhaps my original message wasn't clear.
>> > I have the data in txt files, and am using AmiBroker "Import
> Files"
>> > to load them into the database. That is the process that is
> taking
>> > over 32 hours!
>> >
>> > --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> It is MUCH quicker if you change the settings.
>> >> Default settings are fail-safe. You can however
>> >> change "number of simultaneous downloads" to 5 or more.
>> >> BY setting it say to 10 you can get 10x speed up,
>> >> however Yahoo may block you after some excessive downloading
>> >> and you will need to revert to defaults and wait a couple of
>> > minutes
>> >> for Yahoo to unblock you.
>> >>
>> >> Best regards,
>> >> Tomasz Janeczko
>> >> amibroker.com
>> >> ----- Original Message -----
>> >> From: "jacklweinberg" <jacklweinberg@>
>> >> To: <amibroker@xxxxxxxxxxxxxxx>
>> >> Sent: Monday, May 14, 2007 7:33 PM
>> >> Subject: [amibroker] Massive data loading?
>> >>
>> >>
>> >> > Does anyone have experience at loading up an entirely new
>> > database
>> >> > with AmiBroker?
>> >> > For example, loading AB with all of Nasdaq with 5 years of
> data?
>> >> > I used the file import command, and selected a series of .txt
>> > files,
>> >> > and it takes around 32 hours of elapsed time per year!!! (On
> a
>> > fast
>> >> > machine, no less!)
>> >> > Does anyone know of a quicker method?
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > Please note that this group is for discussion between users
> only.
>> >> >
>> >> > To get support from AmiBroker please send an e-mail directly
> to
>> >> > SUPPORT {at} amibroker.com
>> >> >
>> >> > For NEW RELEASE ANNOUNCEMENTS and other news always check
> DEVLOG:
>> >> > http://www.amibroker.com/devlog/
>> >> >
>> >> > For other support material please check also:
>> >> > http://www.amibroker.com/support.html
>> >> >
>> >> > Yahoo! Groups Links
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >>
>> >
>> >
>> >
>> >
>> > Please note that this group is for discussion between users only.
>> >
>> > To get support from AmiBroker please send an e-mail directly to
>> > SUPPORT {at} amibroker.com
>> >
>> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
>> > http://www.amibroker.com/devlog/
>> >
>> > For other support material please check also:
>> > http://www.amibroker.com/support.html
>> >
>> > Yahoo! Groups Links
>> >
>> >
>> >
>> >
>> >
>>
>
>
>
>
> Please note that this group is for discussion between users only.
>
> To get support from AmiBroker please send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> For other support material please check also:
> http://www.amibroker.com/support.html
>
> Yahoo! Groups Links
>
>
>
>
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|