|
PureBytes Links
Trading Reference Links
|
On the contrary Tomasz,
I feel that Amibroker really takes memory management to the full and what I am hitting here is the 32bit memory limit.
I have now tried AB 64 bit and can confirm that indeed the capacity is way superior.
The issue now lies with the data providers. Hopefully the eventual transition to 64bit only Windows will move them to come out with a 64bit product and these workarounds would be a thing of the past.
I have read on a forum at DTN that at some point they will be delivering a 64bit "wrapper" on their 32bit client. Not sure whether that will work as intended (i'm not really technical) but at least there seems to be some pressure on for them to deliver.
cheers,
eToke
--- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@xxx> wrote:
>
> Hello,
>
> This kind of improvement may suggest that AB own cache settings were sub-optimum.
> Under normal circumstances AB would not touch disk unless hitting cache size limit.
> If cache size limit was lower than number of symbols under scan, then it would access the disk
> and then RAM vs HDD will make the difference.
> Tools->Preferences, "Data" tab "In-memory cache". Note that 32bit version won't allow
> you to use more than approx. 1GB of RAM for the quotation cache alone, because of Windows limits on addressable
> memory for 32 bit applications.
> To use more RAM directly one needs to run 64 bit version of AmiBroker.
> For this reason, if one has 8GB of RAM and uses 32 bit app, using ram-disk is reasonable idea.
> Since data vendors are extremely slow in moving to 64 bit and they don't offer APIs that are 64 bit compatible,
> I am investigating various workarounds.
> I am quite surprised that data vendors do not care about going 64 bit knowning that
> Windows 7 is the LAST Windows OS that will be offered as 32 bit and 64 bit.
> Next Windows OS will be 64-bit only.
>
> Best regards,
> Tomasz Janeczko
> amibroker.com
> ----- Original Message -----
> From: "sidhartha70" <sidhartha70@xxx>
> To: <amibroker@xxxxxxxxxxxxxxx>
> Sent: Wednesday, August 05, 2009 7:15 PM
> Subject: [amibroker] Re: Speeding exporation intraday - Update: Good News!
>
>
> > Thanks.
> >
> > But what did you use to create the RAM drive HDD on your machine currently...?
> >
> > TJ, anything to be aware of using this approach...?
> >
> > TIA
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "etoketrader" <etoke@> wrote:
> >>
> >> I am not sure about the technical details, but I can say that until now on today's current market volume, what was taking 6-7
> >> minutes yesterday is taking around 1m 20sec... that's an incredible increase in performance.
> >>
> >> Everything else seems stable and responsive.
> >>
> >> I am not sure about the 64bit RAM drive, but I am running 64bit windows Xp and 32b ones I downloaded didn't work. I am not sure
> >> whether a 32bit system is capable of creating a RAM drive beyond the 4GB limit... cannot test that as I haven't got a 32bit OS
> >> here.
> >>
> >> cheers,
> >> eToke
> >>
> >> --- In amibroker@xxxxxxxxxxxxxxx, "sidhartha70" <sidhartha70@> wrote:
> >> >
> >> > Very interesting. So what's taking the time here...?? Streaming the data onto your HD...? Hence you remove that bottleneck by
> >> > moving your HD into RAM...?
> >> >
> >> > Why would you want to a 64 bit compatible RAM Drive utility...? You are running 32 bit AB right...? How did you create your
> >> > current 4GB RAM HDD...?
> >> >
> >> > Thanks
> >> >
> >> > --- In amibroker@xxxxxxxxxxxxxxx, "etoketrader" <etoke@> wrote:
> >> > >
> >> > > All,
> >> > >
> >> > > I was able to drastically improve the performance of the exploration during market hours from 6 minutes to just over a
> >> > > minute!! That is an incredible 6X improvement!
> >> > >
> >> > > Outside of market hours I was able to get over 2X improvement.
> >> > >
> >> > > I have 8GB of RAM on my 64bit machine yet I have to run 32bit AB due to the fact that IQFeed do not support a 64bit plugin -
> >> > > hence half my RAM is practically unusable.
> >> > >
> >> > > My AB database is currently approx 3.75GB. This is what I did: I created a 4GB RAMDRIVE (this will take 4GB of RAM and create
> >> > > a virtual HDD), then copied the entire AB folder into it, and ran the program from there. The result was a fascinating
> >> > > improvement: everything runs just so much faster.
> >> > >
> >> > > Since I am running a large 1 minute DB I keep my cache symbols low to 11. This I suspect causes a lot of HDD interaction
> >> > > under normal circumstances, but since the HDD is now in RAM, this isn't an issue any more.
> >> > >
> >> > > Also, the physical HDD is now silent and hardly active any longer. Over all I think this saves some life out of it as a
> >> > > tick-by-tick update every single market day cannot do it much good.
> >> > >
> >> > > There are various free/cheap RAM Drive utilities out there. Most importantly one would need to pick up a 64bit compatible
> >> > > one. I have not done any research whatsoever on these programs... I just picked up the first one I found.
> >> > >
> >> > > There are downsides of course: most notably is that you would lose all the data if the power goes down, however since I am
> >> > > planning to use it this way throughout market hours, it is not an issue for me. I can take the latest backup and update it
> >> > > with a backfill run.
> >> > >
> >> > > I am curious to see whether anybody has experimented with this before and can compare results.
> >> > >
> >> > > Also, anybody aware of other potential downsides when using this method?
> >> > >
> >> > > cheers,
> >> > > eToke
> >> > >
> >> > > --- In amibroker@xxxxxxxxxxxxxxx, "Paul Ho" <paul.tsho@> wrote:
> >> > > >
> >> > > > The following test would help you determine which "elements" are critical to improving exporation speed
> >> > > > 1.
> >> > > > Temporary change your IQFeed database from IQFeed to local, make sure local storage is enabled. run your exploration, That
> >> > > > is the speed your hardware can do with your afl and with no limitation on connection bandwidth.
> >> > > > 2. I suspect that IQFeed connection at application level, not at adsl level, is your bottle neck. To improve it you have to
> >> > > > consider:
> >> > > > 3. Backfilling is far more time consuming than using RTGetData
> >> > > > 4. The more array data your afl uses, the slower it is.
> >> > > > 5. There is only certain amount improvement you can make with new computer hardware. most of the improvements can be gained
> >> > > > by reorganising your afl so things that doesnt change change intraday are scanned overnight, and use RTGETDATA as a
> >> > > > replacement to array data as much as possible.
> >> > > >
> >> > > > --- In amibroker@xxxxxxxxxxxxxxx, "etoketrader" <etoke@> wrote:
> >> > > > >
> >> > > > > Hi Tomasz,
> >> > > > >
> >> > > > > Agreed on load imposed by the rapid update of 500 symbols. No question about it.
> >> > > > >
> >> > > > > What I would like to understand is what elements (hardware, or system optimisation) would help reduce the duration of a
> >> > > > > scan.
> >> > > > >
> >> > > > > Unfortunately running the 64bit version is out of the question because of the realtime plugin issue. This rules out the
> >> > > > > possibility of adding a large amount of RAM although I don't know whether that alone would help.
> >> > > > >
> >> > > > > Are there any other ways I might be able to increase the speed? Would copying the entire db on several GB of a RAM disk
> >> > > > > before a market session be worth trying?
> >> > > > >
> >> > > > > Best regards,
> >> > > > > eToke
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@> wrote:
> >> > > > > >
> >> > > > > > Hello,
> >> > > > > >
> >> > > > > > You need to understand that there can be even 800 ticks PER SECOND on one symbol
> >> > > > > > If you are for example following 500 symbols, it presents the stream of 400000 updates per second.
> >> > > > > > Even when processing took as little as 2 microseconds per update (1 millionth part of the second)
> >> > > > > > this would mean 80% CPU utilisation. So yes, if there are lots of quotes streaming
> >> > > > > > on lots of symbols you will see higher CPU usage and possible also slower exploration.
> >> > > > > >
> >> > > > > > Best regards,
> >> > > > > > Tomasz Janeczko
> >> > > > > > amibroker.com
> >> > > > > > ----- Original Message -----
> >> > > > > > From: "etoketrader" <etoke@>
> >> > > > > > To: <amibroker@xxxxxxxxxxxxxxx>
> >> > > > > > Sent: Tuesday, August 04, 2009 6:09 PM
> >> > > > > > Subject: [amibroker] Re: Speeding exporation intraday
> >> > > > > >
> >> > > > > >
> >> > > > > > > hi Tomasz,
> >> > > > > > >
> >> > > > > > > that makes sense.
> >> > > > > > >
> >> > > > > > > However in my case all my symbols are already active both on and off market hours.
> >> > > > > > >
> >> > > > > > > When quotes are being updated constantly during market hours, a test exploration code which displays the closing
> >> > > > > > > price takes
> >> > > > > > > approx. 6 - 10 times the duration of when the quotes are also streaming but outside of market hours (i.e. hardly any
> >> > > > > > > updates).
> >> > > > > > >
> >> > > > > > > My question is if there is anything I can do to speed this up.
> >> > > > > > >
> >> > > > > > > BR
> >> > > > > > >
> >> > > > > > > eToke
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@> wrote:
> >> > > > > > >>
> >> > > > > > >> Hello,
> >> > > > > > >>
> >> > > > > > >> That depends if given symbol is in the "active" (i.e. streaming) list or not.
> >> > > > > > >> If it is - then turning on/off "wait for backfill" does not make difference.
> >> > > > > > >>
> >> > > > > > >> If it is not, then with "wait for backfill" turned on it will of course wait for "missing" bars
> >> > > > > > >> before proceeding to next symbol.
> >> > > > > > >>
> >> > > > > > >> Charts on the other hand, always refresh immediatelly with data that are present right now.
> >> > > > > > >> If symbol is not present in the active list it is automatically added and backfill request
> >> > > > > > >> may be sent but this does not block chart from refreshing - it will refresh immediatelly,
> >> > > > > > >> and once backfill data eventually arrives it will refresh once more.
> >> > > > > > >> In other words - charts are asynchronous with regards to backfill. AA can be synchronous
> >> > > > > > >> ("wait for backfill" turned ON) or asynchronous ("wait for backfill" turned OFF).
> >> > > > > > >>
> >> > > > > > >> That's the only difference. AFL execution IS IDENTICAL regardless where it occurs.
> >> > > > > > >> Of course with different zoom levels / different AA range you may be using different number of bars,
> >> > > > > > >> but that should be obvious.
> >> > > > > > >>
> >> > > > > > >> Best regards,
> >> > > > > > >> Tomasz Janeczko
> >> > > > > > >> amibroker.com
> >> > > > > > >> ----- Original Message -----
> >> > > > > > >> From: "sidhartha70" <sidhartha70@>
> >> > > > > > >> To: <amibroker@xxxxxxxxxxxxxxx>
> >> > > > > > >> Sent: Tuesday, August 04, 2009 5:14 PM
> >> > > > > > >> Subject: [amibroker] Re: Speeding exporation intraday
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > > >> > I'd be interested to hear TJ's answer on this eToke...
> >> > > > > > >> >
> >> > > > > > >> > Because I was asking from very similar questions of support & TJ recently... namely that equivalent AFL on charts
> >> > > > > > >> > seems to
> >> > > > > > >> > execute
> >> > > > > > >> > much quicker than a scan or exploration. Which obviously leads one to think that other operations that occur in a
> >> > > > > > >> > scan or
> >> > > > > > >> > exploration, outside of the AFL execution, are what take the time...
> >> > > > > > >> >
> >> > > > > > >> > Why this time is greater with a streaming feed I don't know...
> >> > > > > > >> >
> >> > > > > > >> > --- In amibroker@xxxxxxxxxxxxxxx, "etoketrader" <etoke@> wrote:
> >> > > > > > >> >>
> >> > > > > > >> >> Hi, for testing I am currently running:
> >> > > > > > >> >>
> >> > > > > > >> >> n Last quotations = 1
> >> > > > > > >> >> A very basic AFL code which lists closing price.
> >> > > > > > >> >> QuickAFL turned on
> >> > > > > > >> >>
> >> > > > > > >> >> I am concerned about the comparison between how it runs outside of market hours vs how it runs during.
> >> > > > > > >> >>
> >> > > > > > >> >> BR,
> >> > > > > > >> >> eToke
> >> > > > > > >> >>
> >> > > > > > >> >>
> >> > > > > > >> >>
> >> > > > > > >> >> --- In amibroker@xxxxxxxxxxxxxxx, "sidhartha70" <sidhartha70@> wrote:
> >> > > > > > >> >> >
> >> > > > > > >> >> > How many quotations are you running the explorations over...?
> >> > > > > > >> >> >
> >> > > > > > >> >> > --- In amibroker@xxxxxxxxxxxxxxx, "etoketrader" <etoke@> wrote:
> >> > > > > > >> >> > >
> >> > > > > > >> >> > > Tried that. I always run the first backfill before market hours. After that, backfill setting does not seem
> >> > > > > > >> >> > > to make any
> >> > > > > > >> >> > > difference as long as everything has already been backfilled and all symbols are streaming.
> >> > > > > > >> >> > >
> >> > > > > > >> >> > > BR
> >> > > > > > >> >> > > eToke
> >> > > > > > >> >> > >
> >> > > > > > >> >> > >
> >> > > > > > >> >> > > --- In amibroker@xxxxxxxxxxxxxxx, "murthysuresh" <murthysuresh@> wrote:
> >> > > > > > >> >> > > >
> >> > > > > > >> >> > > > backfill before the start of the market hours. turn off backfill as you only need to do it at the beginning
> >> > > > > > >> >> > > > of the
> >> > > > > > >> >> > > > session.
> >> > > > > > >> >> > > >
> >> > > > > > >> >> > > > --- In amibroker@xxxxxxxxxxxxxxx, "etoketrader" <etoke@> wrote:
> >> > > > > > >> >> > > > >
> >> > > > > > >> >> > > > > Hi,
> >> > > > > > >> >> > > > >
> >> > > > > > >> >> > > > > I currently run IQFeed on 500 symbols. Exploration outside of market hours run under 1 minute but slow
> >> > > > > > >> >> > > > > down to 6
> >> > > > > > >> >> > > > > minutes
> >> > > > > > >> >> > > > > during market hours.
> >> > > > > > >> >> > > > >
> >> > > > > > >> >> > > > > Can anybody recommend some ideal settings to speed up intraday explorations while streaming?
> >> > > > > > >> >> > > > >
> >> > > > > > >> >> > > > > thanks,
> >> > > > > > >> >> > > > > eToke
> >> > > > > > >> >> > > > >
> >> > > > > > >> >> > > >
> >> > > > > > >> >> > >
> >> > > > > > >> >> >
> >> > > > > > >> >>
> >> > > > > > >> >
> >> > > > > > >> >
> >> > > > > > >> >
> >> > > > > > >> >
> >> > > > > > >> > ------------------------------------
> >> > > > > > >> >
> >> > > > > > >> > **** IMPORTANT PLEASE READ ****
> >> > > > > > >> > This group is for the discussion between users only.
> >> > > > > > >> > This is *NOT* technical support channel.
> >> > > > > > >> >
> >> > > > > > >> > TO GET TECHNICAL SUPPORT send an e-mail directly to
> >> > > > > > >> > SUPPORT {at} amibroker.com
> >> > > > > > >> >
> >> > > > > > >> > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> >> > > > > > >> > http://www.amibroker.com/feedback/
> >> > > > > > >> > (submissions sent via other channels won't be considered)
> >> > > > > > >> >
> >> > > > > > >> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> >> > > > > > >> > http://www.amibroker.com/devlog/
> >> > > > > > >> >
> >> > > > > > >> > Yahoo! Groups Links
> >> > > > > > >> >
> >> > > > > > >> >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > ------------------------------------
> >> > > > > > >
> >> > > > > > > **** IMPORTANT PLEASE READ ****
> >> > > > > > > This group is for the discussion between users only.
> >> > > > > > > This is *NOT* technical support channel.
> >> > > > > > >
> >> > > > > > > TO GET TECHNICAL SUPPORT send an e-mail directly to
> >> > > > > > > SUPPORT {at} amibroker.com
> >> > > > > > >
> >> > > > > > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> >> > > > > > > http://www.amibroker.com/feedback/
> >> > > > > > > (submissions sent via other channels won't be considered)
> >> > > > > > >
> >> > > > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> >> > > > > > > http://www.amibroker.com/devlog/
> >> > > > > > >
> >> > > > > > > Yahoo! Groups Links
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
> >
> >
> > ------------------------------------
> >
> > **** IMPORTANT PLEASE READ ****
> > This group is for the discussion between users only.
> > This is *NOT* technical support channel.
> >
> > TO GET TECHNICAL SUPPORT send an e-mail directly to
> > SUPPORT {at} amibroker.com
> >
> > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> > http://www.amibroker.com/feedback/
> > (submissions sent via other channels won't be considered)
> >
> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> > http://www.amibroker.com/devlog/
> >
> > Yahoo! Groups Links
> >
> >
> >
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|