|
PureBytes Links
Trading Reference Links
|
All,
I was able to drastically improve the performance of the exploration during market hours from 6 minutes to just over a minute!! That is an incredible 6X improvement!
Outside of market hours I was able to get over 2X improvement.
I have 8GB of RAM on my 64bit machine yet I have to run 32bit AB due to the fact that IQFeed do not support a 64bit plugin - hence half my RAM is practically unusable.
My AB database is currently approx 3.75GB. This is what I did: I created a 4GB RAMDRIVE (this will take 4GB of RAM and create a virtual HDD), then copied the entire AB folder into it, and ran the program from there. The result was a fascinating improvement: everything runs just so much faster.
Since I am running a large 1 minute DB I keep my cache symbols low to 11. This I suspect causes a lot of HDD interaction under normal circumstances, but since the HDD is now in RAM, this isn't an issue any more.
Also, the physical HDD is now silent and hardly active any longer. Over all I think this saves some life out of it as a tick-by-tick update every single market day cannot do it much good.
There are various free/cheap RAM Drive utilities out there. Most importantly one would need to pick up a 64bit compatible one. I have not done any research whatsoever on these programs... I just picked up the first one I found.
There are downsides of course: most notably is that you would lose all the data if the power goes down, however since I am planning to use it this way throughout market hours, it is not an issue for me. I can take the latest backup and update it with a backfill run.
I am curious to see whether anybody has experimented with this before and can compare results.
Also, anybody aware of other potential downsides when using this method?
cheers,
eToke
--- In amibroker@xxxxxxxxxxxxxxx, "Paul Ho" <paul.tsho@xxx> wrote:
>
> The following test would help you determine which "elements" are critical to improving exporation speed
> 1.
> Temporary change your IQFeed database from IQFeed to local, make sure local storage is enabled. run your exploration, That is the speed your hardware can do with your afl and with no limitation on connection bandwidth.
> 2. I suspect that IQFeed connection at application level, not at adsl level, is your bottle neck. To improve it you have to consider:
> 3. Backfilling is far more time consuming than using RTGetData
> 4. The more array data your afl uses, the slower it is.
> 5. There is only certain amount improvement you can make with new computer hardware. most of the improvements can be gained by reorganising your afl so things that doesnt change change intraday are scanned overnight, and use RTGETDATA as a replacement to array data as much as possible.
>
> --- In amibroker@xxxxxxxxxxxxxxx, "etoketrader" <etoke@> wrote:
> >
> > Hi Tomasz,
> >
> > Agreed on load imposed by the rapid update of 500 symbols. No question about it.
> >
> > What I would like to understand is what elements (hardware, or system optimisation) would help reduce the duration of a scan.
> >
> > Unfortunately running the 64bit version is out of the question because of the realtime plugin issue. This rules out the possibility of adding a large amount of RAM although I don't know whether that alone would help.
> >
> > Are there any other ways I might be able to increase the speed? Would copying the entire db on several GB of a RAM disk before a market session be worth trying?
> >
> > Best regards,
> > eToke
> >
> >
> >
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@> wrote:
> > >
> > > Hello,
> > >
> > > You need to understand that there can be even 800 ticks PER SECOND on one symbol
> > > If you are for example following 500 symbols, it presents the stream of 400000 updates per second.
> > > Even when processing took as little as 2 microseconds per update (1 millionth part of the second)
> > > this would mean 80% CPU utilisation. So yes, if there are lots of quotes streaming
> > > on lots of symbols you will see higher CPU usage and possible also slower exploration.
> > >
> > > Best regards,
> > > Tomasz Janeczko
> > > amibroker.com
> > > ----- Original Message -----
> > > From: "etoketrader" <etoke@>
> > > To: <amibroker@xxxxxxxxxxxxxxx>
> > > Sent: Tuesday, August 04, 2009 6:09 PM
> > > Subject: [amibroker] Re: Speeding exporation intraday
> > >
> > >
> > > > hi Tomasz,
> > > >
> > > > that makes sense.
> > > >
> > > > However in my case all my symbols are already active both on and off market hours.
> > > >
> > > > When quotes are being updated constantly during market hours, a test exploration code which displays the closing price takes
> > > > approx. 6 - 10 times the duration of when the quotes are also streaming but outside of market hours (i.e. hardly any updates).
> > > >
> > > > My question is if there is anything I can do to speed this up.
> > > >
> > > > BR
> > > >
> > > > eToke
> > > >
> > > >
> > > >
> > > > --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@> wrote:
> > > >>
> > > >> Hello,
> > > >>
> > > >> That depends if given symbol is in the "active" (i.e. streaming) list or not.
> > > >> If it is - then turning on/off "wait for backfill" does not make difference.
> > > >>
> > > >> If it is not, then with "wait for backfill" turned on it will of course wait for "missing" bars
> > > >> before proceeding to next symbol.
> > > >>
> > > >> Charts on the other hand, always refresh immediatelly with data that are present right now.
> > > >> If symbol is not present in the active list it is automatically added and backfill request
> > > >> may be sent but this does not block chart from refreshing - it will refresh immediatelly,
> > > >> and once backfill data eventually arrives it will refresh once more.
> > > >> In other words - charts are asynchronous with regards to backfill. AA can be synchronous
> > > >> ("wait for backfill" turned ON) or asynchronous ("wait for backfill" turned OFF).
> > > >>
> > > >> That's the only difference. AFL execution IS IDENTICAL regardless where it occurs.
> > > >> Of course with different zoom levels / different AA range you may be using different number of bars,
> > > >> but that should be obvious.
> > > >>
> > > >> Best regards,
> > > >> Tomasz Janeczko
> > > >> amibroker.com
> > > >> ----- Original Message -----
> > > >> From: "sidhartha70" <sidhartha70@>
> > > >> To: <amibroker@xxxxxxxxxxxxxxx>
> > > >> Sent: Tuesday, August 04, 2009 5:14 PM
> > > >> Subject: [amibroker] Re: Speeding exporation intraday
> > > >>
> > > >>
> > > >> > I'd be interested to hear TJ's answer on this eToke...
> > > >> >
> > > >> > Because I was asking from very similar questions of support & TJ recently... namely that equivalent AFL on charts seems to
> > > >> > execute
> > > >> > much quicker than a scan or exploration. Which obviously leads one to think that other operations that occur in a scan or
> > > >> > exploration, outside of the AFL execution, are what take the time...
> > > >> >
> > > >> > Why this time is greater with a streaming feed I don't know...
> > > >> >
> > > >> > --- In amibroker@xxxxxxxxxxxxxxx, "etoketrader" <etoke@> wrote:
> > > >> >>
> > > >> >> Hi, for testing I am currently running:
> > > >> >>
> > > >> >> n Last quotations = 1
> > > >> >> A very basic AFL code which lists closing price.
> > > >> >> QuickAFL turned on
> > > >> >>
> > > >> >> I am concerned about the comparison between how it runs outside of market hours vs how it runs during.
> > > >> >>
> > > >> >> BR,
> > > >> >> eToke
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> --- In amibroker@xxxxxxxxxxxxxxx, "sidhartha70" <sidhartha70@> wrote:
> > > >> >> >
> > > >> >> > How many quotations are you running the explorations over...?
> > > >> >> >
> > > >> >> > --- In amibroker@xxxxxxxxxxxxxxx, "etoketrader" <etoke@> wrote:
> > > >> >> > >
> > > >> >> > > Tried that. I always run the first backfill before market hours. After that, backfill setting does not seem to make any
> > > >> >> > > difference as long as everything has already been backfilled and all symbols are streaming.
> > > >> >> > >
> > > >> >> > > BR
> > > >> >> > > eToke
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > --- In amibroker@xxxxxxxxxxxxxxx, "murthysuresh" <murthysuresh@> wrote:
> > > >> >> > > >
> > > >> >> > > > backfill before the start of the market hours. turn off backfill as you only need to do it at the beginning of the
> > > >> >> > > > session.
> > > >> >> > > >
> > > >> >> > > > --- In amibroker@xxxxxxxxxxxxxxx, "etoketrader" <etoke@> wrote:
> > > >> >> > > > >
> > > >> >> > > > > Hi,
> > > >> >> > > > >
> > > >> >> > > > > I currently run IQFeed on 500 symbols. Exploration outside of market hours run under 1 minute but slow down to 6
> > > >> >> > > > > minutes
> > > >> >> > > > > during market hours.
> > > >> >> > > > >
> > > >> >> > > > > Can anybody recommend some ideal settings to speed up intraday explorations while streaming?
> > > >> >> > > > >
> > > >> >> > > > > thanks,
> > > >> >> > > > > eToke
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > ------------------------------------
> > > >> >
> > > >> > **** IMPORTANT PLEASE READ ****
> > > >> > This group is for the discussion between users only.
> > > >> > This is *NOT* technical support channel.
> > > >> >
> > > >> > TO GET TECHNICAL SUPPORT send an e-mail directly to
> > > >> > SUPPORT {at} amibroker.com
> > > >> >
> > > >> > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> > > >> > http://www.amibroker.com/feedback/
> > > >> > (submissions sent via other channels won't be considered)
> > > >> >
> > > >> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> > > >> > http://www.amibroker.com/devlog/
> > > >> >
> > > >> > Yahoo! Groups Links
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------------
> > > >
> > > > **** IMPORTANT PLEASE READ ****
> > > > This group is for the discussion between users only.
> > > > This is *NOT* technical support channel.
> > > >
> > > > TO GET TECHNICAL SUPPORT send an e-mail directly to
> > > > SUPPORT {at} amibroker.com
> > > >
> > > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> > > > http://www.amibroker.com/feedback/
> > > > (submissions sent via other channels won't be considered)
> > > >
> > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> > > > http://www.amibroker.com/devlog/
> > > >
> > > > Yahoo! Groups Links
> > > >
> > > >
> > > >
> > >
> >
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|