|
PureBytes Links
Trading Reference Links
|
Hello,
5 years of data on my end imports in *minutes* not hours.
a) you have small size of "in-memory" cache so it requires a lot of disk mangling
b) the data you import are not sorted or sorted improperly (in reverse order) and AB needs to re-sort.
For fast import It is advised to use data that is sorted in OLDEST QUOTES FIRST - NEWEST LAST
and to have one file per symbol.
The worst possible case (it's probably yours) is having sorted in opposite direction
(newest first) and different symbol in every line.
Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message -----
From: "jacklweinberg" <jacklweinberg@xxxxxxxxx>
To: <amibroker@xxxxxxxxxxxxxxx>
Sent: Tuesday, May 15, 2007 2:59 PM
Subject: [amibroker] Re: Massive data loading?
> Tomasz,
> Perhaps my original message wasn't clear.
> I have the data in txt files, and am using AmiBroker "Import Files"
> to load them into the database. That is the process that is taking
> over 32 hours!
>
> --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@xxx>
> wrote:
>>
>> Hello,
>>
>> It is MUCH quicker if you change the settings.
>> Default settings are fail-safe. You can however
>> change "number of simultaneous downloads" to 5 or more.
>> BY setting it say to 10 you can get 10x speed up,
>> however Yahoo may block you after some excessive downloading
>> and you will need to revert to defaults and wait a couple of
> minutes
>> for Yahoo to unblock you.
>>
>> Best regards,
>> Tomasz Janeczko
>> amibroker.com
>> ----- Original Message -----
>> From: "jacklweinberg" <jacklweinberg@xxx>
>> To: <amibroker@xxxxxxxxxxxxxxx>
>> Sent: Monday, May 14, 2007 7:33 PM
>> Subject: [amibroker] Massive data loading?
>>
>>
>> > Does anyone have experience at loading up an entirely new
> database
>> > with AmiBroker?
>> > For example, loading AB with all of Nasdaq with 5 years of data?
>> > I used the file import command, and selected a series of .txt
> files,
>> > and it takes around 32 hours of elapsed time per year!!! (On a
> fast
>> > machine, no less!)
>> > Does anyone know of a quicker method?
>> >
>> >
>> >
>> >
>> > Please note that this group is for discussion between users only.
>> >
>> > To get support from AmiBroker please send an e-mail directly to
>> > SUPPORT {at} amibroker.com
>> >
>> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
>> > http://www.amibroker.com/devlog/
>> >
>> > For other support material please check also:
>> > http://www.amibroker.com/support.html
>> >
>> > Yahoo! Groups Links
>> >
>> >
>> >
>> >
>> >
>>
>
>
>
>
> Please note that this group is for discussion between users only.
>
> To get support from AmiBroker please send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> For other support material please check also:
> http://www.amibroker.com/support.html
>
> Yahoo! Groups Links
>
>
>
>
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|