RE: [amibroker] Quad-core test results, AmiBroker Email List Archive

Thanks TJ,

This is very interesting. I will try that.

You are saying that even when having 4GB of DRAM, and despite window XP (32bit) limitation of maximum usage of 4GB it is advisable to use an additional swapfile?

That should be an easy change, I have a small (2GB) partition reserved just for that purpose which is formatted Fat32 (which is supposedly faster for small partitions).

Joseph Biran
____________________________________________

From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Tomasz Janeczko
Sent: Saturday, May 24, 2008 12:21 AM
To: amibroker@xxxxxxxxxxxxxxx
Subject: Re: [amibroker] Quad-core test results

Hello,

Try setting refresh time to zero and use recent betas. This automatically and dynamically adjusts the refresh time based

on actual timing so charts are refreshed as often as possible but not more than that.

As to having no swap file - it is actually bad idea. Surprisingly and counter-intuitively Windows runs SLOWER without swap file.

Yes, if you disable swap in runs slower than with minimum non-growable swap (say 10 MB).

Programs that use VirtualAlloc functions to reserve pages in front will run better with swap file enabled

as reserved pages do not need to be in RAM until the program request to "commit" the page.

Memory allocation in systems like Windows is not as simple as people usually tend to think. "Common sense" does not work.

Programs can reserve large pools of continuous addresses without actually using RAM. That works fine only if swap file extists.

Best regards,
Tomasz Janeczko
amibroker.com

----- Original Message -----

From: J. Biran

To: amibroker@xxxxxxxxxxxxxxx

Sent: Saturday, May 24, 2008 2:21 AM

Subject: RE: [amibroker] Quad-core test results

Thanks Steve and TJ.

Yes, I got the idea.

As far as 2. Is concerned:

I use NO swap file on my HDD at all. I have 4GB of DRAM on my computer, which is the maximum size of memory that WinXP 32bit can handle.

This is an conclusion from my observations of AB behavior.

i.e. if I reduce the Realtime Chart Refresh time too much I will see on Process Explorer (see my prior post) that CPU utilization is pegged at 50%, and the system stops responding (unfortunately undoing such a mistake is VERY time consuming).

A similar example is if I try to change a symbol of a chart and a set of linked charts use range bars that are inappropriate for the new symbol (i.e. 1.0R which is right for ER2 and the new symbol is YM (where 10R is the equivalent) I can go and have lunch before I get a chance to correct that error. (the fact that I need to use intraday tick database with 1E6 bars to load does not help here)

Joseph Biran
____________________________________________

From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Steve Dugas
Sent: Friday, May 23, 2008 12:21 PM
To: amibroker@xxxxxxxxxxxxxxx
Subject: Re: [amibroker] Quad-core test results

Hi Joseph - looks like TJ has answered your questions and perhaps I touched on the answers in my other e-mails. Let me know if you still have a question!

Steve

----- Original Message -----

From: Tomasz Janeczko

To: amibroker@xxxxxxxxxxxxxxx

Sent: Friday, May 23, 2008 5:20 AM

Subject: Re: [amibroker] Quad-core test results

1. FWIW: You can run as many instances as you wish, even 100. As long as you don't update anything it will be just fine.

But if you start saving any data the "last" write counts.

2. Machine should never grind to a halt with 50% CPU utilization on dual core. If it does it means that it is SWAPPING excessively (then system locks the machine).

Excessive swapping usually means that your config is not correct.

Best regards,
Tomasz Janeczko
amibroker.com

----- Original Message -----

From: J. Biran

To: amibroker@xxxxxxxxxxxxxxx

Sent: Friday, May 23, 2008 10:13 AM

Subject: RE: [amibroker] Quad-core test results

I am a bit confused:

1.

You were able to run multiple instances of AB all using same database on one machine? I thought that is not possible.

2.

25% CPU utilization on a quad core CPU means 1 core is fully occupied etc. i.e. on my dual core AMD CPU AB never consumes more than 50%. If utilization is stuck at 50% the machine grinds to a halt.

3.

If the application itself cannot utilize multiple cores (I believe few exist now), the maximum “utilization” you could get from a quad core CPU should be 25% of the theoretical total computing power. Am I missing something?

Joseph Biran
____________________________________________

From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] On Behalf Of Steve Dugas
Sent: Thursday, May 22, 2008 5:50 PM
To: Yahoo - AmiBroker
Subject: [amibroker] Quad-core test results

Hi All,

I finished setting up the new quad-core machine and ran my first tests today, so am posting the promised test results. All I can say is Wow! Really nice improvements, even better than I expected to see in my highest hopes. I am happy as a pig in sh*t !!     8 - )

For background, the test was an optimization run on a single ticker, which is how I will be running all my other tests ( ticker was QID, 468 EOD data bars ). All tests used the same code with different param settings, all tests had about 46,000 opt steps.

To put things in perspective and show why I am so happy, the original code was about 2,400 lines. The first time I ran it, on my old backup computer which I was using as a dedicated optimization machine, it took 7 1/2 hours to run. So then I copied the code and created a shorter version, removing everything which wasn't absolutley necessary for the optimizer, and that reduced the run time to 2 1/2 hours. Then I ran this short version on my faster primary machine and that took 1 1/4 hours, which is about what I was expecting to see on the new machine.

So today I started by running just ran one instance on the quad-core - that took only 30 mins and Task Manager showed it was using just 25% of the total processing power!  Well to make a long story short, I kept adding one instance at a time, all instances ran in 30 mins and each used up an additional 25% of the CPU power. In the end, I was easily running 4 simultaneous instances. This pretty much kept the CPU tached at 100% but all instances ran fine, all finished in 30 mins and I didn't experience any problems at all. I was even saving the first ones to spreadsheets while the final ones were still finishing, wow everything just worked flawlessly! So, the quad-core can run 4 seperate opts simultaneously in 30 mins, which averages out to 7 1/2 minutes per opt, which =

10X improvement over running the short code on the fast machine...

20X improvement over running the short code on the old dedicated optimization machine...

60X improvement over running the original code on the old optimization machine...  Awesome !!!

To those who were wondering what is the best machine to get for running AB, it looks to me like quad is the way to go. ( My machine has an Intel processor, which TJ mentioned should probably work better than AMD for this stuff )....

Steve