Title: Re: [amibroker] Re: Managing large Databases (PremiumData and IQFeed)
Thank you!
I looked up the Ram Drive but am not sure that, since i might be switching to Win7 soon, I want to spend the money at this time.
My IQFeed DB is about 20 Gbyte - probably overkill but it is nice for research and play. I wish there was a way to draw from this master DB "as needed", from within AB - or at startup. I play a lot and most of my applications have unique Ticker selection and history requirements. Sometimes I need current/RT data, at other times I want to run only historical segments. If AB could extract only the data that I need, copy this to a temp DB, and work from that, this would result in optimum use of system resources. But there are probably a zillion problems why this is not possible.
btw, am I correct that the local DB is the only copy of the IQFeed data? I can't find any DTN IQFEED DB outside the AB folder (Like Premium Data does).
Thanks for feedback, you gave me something to think about :-)
herman
Sunday, August 16, 2009, 9:59:00 AM, you wrote:
> Herman,
> I too have been battling with maintaining a large database.
> I currently use IQFeed and maintain a 500 symbol db at 1 minute resolution.
> IQFeed currently has about 2 years of intraday data but I keep 10 years of
> mixed day/intraday data. My DB is about 3.75GB.
> Finally these past few weeks I have been able to get the desired performance
> that I have been looking for. These are the steps I have done to acheive this:
> 1) I have 8GB of RAM running on Windows XP OS. I created a RamDisk with 4GB
> of the RAM and copied the entire AB folder + data into this disk. This alone
> has increased my performance between 4X and 6X. A good utility you can
> purchase to create a Ram Disk is RAMDisk Plus by Superspeed.
> 2) I created scripts which Save the DB, unload AB, copy the contents of the
> Ram Disk to a location on the HDD, load AB and run an exploration with the
> "wait for backfill" option. This will get all the 500 symbols updated and
> "online" hence all being updated in realtime. I used the standard Windows XP
> scheduler to schedule this task to run at night, and gives me a backup which I
> can reaload in case my machine crashes during market hours. (Note: I was never
> able to do this with the full 500 symbols without AB crashing at some point until I did the RAM disk thing).
> 3) Cache size max symbols in AB set to 11, and Max Megabytes set to 850. This
> causes AB to interact heavily with the HDD, but since the HDD is in fact in
> RAM (through the RAM Disk) the performance does not degrade at all.
> 4) With these mods I find I no longer need to run separate DB's, but if you
> still do, then I found the most reliable is to run AB as a separate instance
> as a separate user. For this you can first create several logins then create a
> script which launches AB using the "RUNAS" command.
> I am confident that all of these problems are resolved if we could move to AB
> 64bit, but unfortunately the IQFeed client does not work on a 64bit system,
> hence it is not an option for the time being.
> hope this helps,
> eToke
> ------------------------------------
> **** IMPORTANT PLEASE READ ****
> This group is for the discussion between users only.
> This is *NOT* technical support channel.
> TO GET TECHNICAL SUPPORT send an e-mail directly to
> SUPPORT {at} amibroker.com
> TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> http://www.amibroker.com/feedback/
> (submissions sent via other channels won't be considered)
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
> Yahoo! Groups Links
> <*> To visit your group on the web, go to:
> http://groups.yahoo.com/group/amibroker/
> <*> Your email settings:
> Individual Email | Traditional
> <*> To change settings online go to:
> http://groups.yahoo.com/group/amibroker/join
> (Yahoo! ID required)
> <*> To change settings via email:
> mailto:amibroker-digest@xxxxxxxxxxxxxxx
> mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
> <*> To unsubscribe from this group, send an email to:
> amibroker-unsubscribe@xxxxxxxxxxxxxxx
> <*> Your use of Yahoo! Groups is subject to:
> http://docs.yahoo.com/info/terms/
__._,_.___
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
![]()
__,_._,___
|