|
PureBytes Links
Trading Reference Links
|
Just wanted to post an update. I have been using 32 bit AB on 64 bit XP with success. The 32 bit AB seems limited to 2GB cache, despite previous posters claiming 4GB cache. However, I am able to load many large datasets one after another without AB crashing. This is probably due to the process having a much larger virtual memory space, allowing for contiguous memory blocks being allocated with ease for a 2GB cache. Anyway, this seems to be an acceptable solution for now.
BTW, thanks for the info about Windows virtual memory, Tomasz. I had no idea things were that bad. This is useful information that will help me at work as well. Hopefully everything will switch to 64 bit processing soon, and all these problems will be a fading memory.
Thanks,
Chris
--- In amibroker@xxxxxxxxxxxxxxx, Tomasz Janeczko <groups@xxx> wrote:
>
> Hello,
>
> One note about something that you may not be aware of. Virtual memory
> space available to the process in Windows
> is NOT contiguous. Just in the middle of the address space there are
> USER32.DLL, GDI32.DLL and such
> core DLLs loaded by Windows, then in the quarter of address space
> Microsoft decided to load URLMON.DLL
> and similar and a bunch of other things at other random locations just
> in the middle of free block.
> If you look at address space available to the process using tool like VMMap
> http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx you will see
> that address space in Windows is scattered very badly and even though
> you may seem to have 3GB free
> it is full of holes like Swiss cheese and largest memory block is BELOW
> 1GB.
> I am speaking about "do-nothing" application that does NOT allocate ANY
> memory. Such application
> at the entry point sees largest memory block of about 1GB or less,
> regardless of amount of RAM.
> That is the reason why applications on Windows reach out-of-memory -
> the remaining "free" virtual memory is just scattered so much it can't
> be used for any larger allocation.
>
> See this article on MATLAB site:
> http://www.mathworks.com/support/tech-notes/1100/1106.html
>
> Best regards,
> Tomasz Janeczko
> amibroker.com
>
> On 2010-01-26 09:06, Christopher wrote:
> > Hello,
> >
> > I just wanted to provide some final notes about my experience with this whole thing. I tried the /3G switch and it was only a slightly better solution for my situation. Using this switch made the entire operating system very slow and at times completely unusable. I even tried adjusting the amount of user address space with the /userva switch, and that helped a little, but the OS was never really normal with these switches on my machine.
> >
> > I actually had 4GB of RAM all along, however, one of my graphics cards had something called TurboCache which uses upto 1GB of system RAM. Removing this card increased the amount of usable physical RAM as reported in the System Properties General Tab from 2.5 to 3.5 GB. Interestingly, removing this graphics card seemed to have no effect on the amount of RAM AmiBroker required before crashing.
> >
> > I did several tests on my system to see how much virtual ram AB would be able to use before crashing. It didn't matter if I was using IQFeed or a local database. When Amibroker's VM (virtual ram) got over a certain size, it crashed.
> >
> > If I did not use the /3GB switch, AB would crash whenever it's VM went over 1GB. It didn't matter if I had my second graphics card in or not. It didn't matter if Windows System Information said I had 2GB of VM space available.
> >
> > If I use the /3GB switch, AB would be able to use more VM before crashing, and it seemed to be proportional to the amount of extra RAM alloted to the user space, but it was always around 1GB less than the amount of total VM available, as reported in System Information. It didn't seem like my other processes were using up that other 1GB, but maybe one of these other processes reserved that address space.
> >
> > I also tried decreasing the maximum memory size in AB to the smallest number allowed and setting the number of symbols to 11. This helped in some situations, but I still had to do a lot of manual flushing of the cache to prevent losing data that I was downloading from IQFeed. I really don't understand why AB has these arbitrary min/max memory and number of symbol limits. Also, having a symbol number of 10 mean something special is strange, especially since it isn't documented. Seems like another checkbox would have been more appropriate for the special meaning.
> >
> > Wouldn't it be better to have a mechanism to prevent AB to crash despite all these settings? Or is there no way to predict how much data is coming down the pipe when retrieving data?
> >
> > I know AmiBroker is very powerful, I just wish this tool were a little more user friendly. I realize that there may not be too many people trying to do what I am doing right now, but I know there have been a few in the past, and there probably will be many more to come. Hopefully someone will glean something useful from these posts, but I don't think users should have to spend their time dredging through thousands of forum posts to figure out that we should use the number 11 instead of the number 10.
> >
> > Thanks,
> > Chris
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, Tomasz Janeczko<groups@> wrote:
> >
> >> Some extra reading about /3G Windows OS switch:
> >> http://blogs.technet.com/askperf/archive/2007/03/23/memory-management-demystifying-3gb.aspx
> >>
> >> On 2010-01-14 11:03, Tomasz Janeczko wrote:
> >>
> >>> Hello,
> >>>
> >>> In fact you should DECREASE maximum megabytes and set number of symbols
> >>> to 11. 10 has special meaning "adjust automatically". Also it was
> >>> discussed in the past that using 32 bit operating system you are NOT able
> >>> to use more than 2GB of memory per application (or 3GB with /3G BOOT.INI
> >>> switch). This is limitation of 32-bit operating system and 32-bit
> >>> addressing. For AmiBroker capabilities regarding addressable memory see:
> >>> http://www.amibroker.com/guide/compat.html
> >>>
> >>> So first thing you should do is to adjust your BOOT.INI setting by
> >>> adding /3G switch in order to enable Windows to actually see more
> >>> memory. Even with that 32-bit Windows is not capable of using more than
> >>> 3.5GB RAM (without PAE) in total because 0.5 GB of address space is
> >>> usually reserved for graphic card, and other I/O cards, etc.
> >>>
> >>> The /3GB switch allocates 3 GB of virtual address space to an
> >>> application that uses IMAGE_FILE_LARGE_ADDRESS_AWARE in the process
> >>> header (AmiBroker has it). This switch allows applications to address 1
> >>> GB of additional virtual address space above 2 GB.
> >>>
> >>> The following example shows how to add the /3GB parameter in the
> >>> Boot.ini file to enable application memory tuning:
> >>> [boot loader]
> >>> timeout=30
> >>> default=multi(0)disk(0)rdisk(0)partition(2)\WINNT
> >>> [operating systems]
> >>> multi(0)disk(0)rdisk(0)partition(2)\WINNT="Your Windows with 3GB
> >>> support" /3GB
> >>>
> >>> Best regards,
> >>> Tomasz Janeczko
> >>> amibroker.com
> >>>
> >>> On 2010-01-14 07:34, Christopher wrote:
> >>>
> >>>
> >>>> Thanks Marc and ReefBreak,
> >>>>
> >>>> I used Task Manager to determine that the memory limit in AmiBroker is causing my problem. I tried adjusting the memory settings in the Data Preferences tab, but they seem insufficient for my purposes. I cannot set number of symbols to something less than 10, and I cannot increase maximum megabytes over 1023. Are there hidden registry flags to modify these restrictions?
> >>>>
> >>>> I am working with large histories for a very small number of symbols, and it seems that I could avoid the crashes and avoid having to manually flush the cache if I could only set the number of symbols to something in the 3-5 symbol range, or if I could increase the max cache memory to make better use of my RAM.
> >>>>
> >>>> Actually, I thought I had 4GB RAM on this machine, but it turns out that I only have 2.5GB right now. Does the max cache size increase if I add more RAM? Does it also increase if I go to x64 platform?
> >>>>
> >>>> It would be nice if AmiBroker (or the plugin for IQFeed) automatically flushed the cache instead of crashing. Seems like this shouldn't be happening. Is there at least a keyboard shortcut to flush the cache, so that I don't have to keep going to database settings dialog?
> >>>>
> >>>> I wanted to thank everyone who responded to my post. It has been frustrating getting all setup for using AmiBroker to backtest my ideas (I just want to trade!)
> >>>>
> >>>> Good Trading,
> >>>> Chris
> >>>>
> >>>>
> >>>> --- In amibroker@xxxxxxxxxxxxxxx, "de_techneut"<twistedcharts@> wrote:
> >>>>
> >>>>
> >>>>
> >>>>> Hi Chris,
> >>>>>
> >>>>> in order to check memory in detail you can use:
> >>>>> CTRL+ALT+DELETE and then select task manager.
> >>>>>
> >>>>> there you'll be able to look at the processes running on your computer.
> >>>>> it is possible to add columns to the viewing screen.
> >>>>> one of the columns is peak memory usage.
> >>>>> it'l tell you if one application is using excessive memory.
> >>>>>
> >>>>> btw: it doesn't seem logical that IQfeed plugin is the problem as it is an official amibroker release( althou i don't use it myself ).
> >>>>>
> >>>>> it is more likely that an other faulty plugin or dll is causing this.
> >>>>> i have wrecked computers because a program i was using was not handling memory like it should with frequent BSOD's as a result.
> >>>>> when that happens often enough the whole computer will suffer...
> >>>>>
> >>>>> hope this helps
> >>>>>
> >>>>> Marc
> >>>>>
> >>>>>
> >>>>>
> >>>> ------------------------------------
> >>>>
> >>>> **** IMPORTANT PLEASE READ ****
> >>>> This group is for the discussion between users only.
> >>>> This is *NOT* technical support channel.
> >>>>
> >>>> TO GET TECHNICAL SUPPORT send an e-mail directly to
> >>>> SUPPORT {at} amibroker.com
> >>>>
> >>>> TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> >>>> http://www.amibroker.com/feedback/
> >>>> (submissions sent via other channels won't be considered)
> >>>>
> >>>> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> >>>> http://www.amibroker.com/devlog/
> >>>>
> >>>> Yahoo! Groups Links
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>> ------------------------------------
> >>>
> >>> **** IMPORTANT PLEASE READ ****
> >>> This group is for the discussion between users only.
> >>> This is *NOT* technical support channel.
> >>>
> >>> TO GET TECHNICAL SUPPORT send an e-mail directly to
> >>> SUPPORT {at} amibroker.com
> >>>
> >>> TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> >>> http://www.amibroker.com/feedback/
> >>> (submissions sent via other channels won't be considered)
> >>>
> >>> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> >>> http://www.amibroker.com/devlog/
> >>>
> >>> Yahoo! Groups Links
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >
> >
> >
> > ------------------------------------
> >
> > **** IMPORTANT PLEASE READ ****
> > This group is for the discussion between users only.
> > This is *NOT* technical support channel.
> >
> > TO GET TECHNICAL SUPPORT send an e-mail directly to
> > SUPPORT {at} amibroker.com
> >
> > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> > http://www.amibroker.com/feedback/
> > (submissions sent via other channels won't be considered)
> >
> > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> > http://www.amibroker.com/devlog/
> >
> > Yahoo! Groups Links
> >
> >
> >
> >
> >
>
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
amibroker-digest@xxxxxxxxxxxxxxx
amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|