Perl scripts hogging CPU

Question

I am running several instances of a script and everything works fine. Each terminal I load builds CPU higher though.

I am running 12 instances and throttling 99% of a dual core CPU. I fully stacked my ram 8gb of DDR3. At one point during the project, I used mysql but found heavy bottlenecks there and some random disconnect error (which googling told me could be the NIC).

Anyways..So writing to disc was a lot quicker straight from shell. The module used is WWW:Mechanize

As far as per instance though,the profile isn't terrible. Bounces up-down to 24%. But I reach a limit in execution to 100% load with 8 or 9 terminal windows running.. Most likely it is a result of bandwidth and just download time.. Could I speed this up at all? Let's say stop downloading after a # of kilobytes ?

Thanks, Peter

Answer 1

Unfortunately I do not believe there is a way to affect the perl interpreter behaviour that way. You can set up cpu limits for all your perl scripts on your box, but as far as I know, there is no way to do that in perl, as the interpreter manages all memory allocation etc. on its own.

You may want to check out this link: How to limit CPU usage in perl

Also you may want to look into perl's threads, as that will definitely be more efficient then running the interpreter for 19 times.

Answer 2

But I reach a limit in execution to 100% load with 8 or 9 terminal windows running.. Most likely it is a result of bandwidth and just download time..

Umm... No. If you hit a wall due to bandwidth constraints/download time, then your processes will block on I/O (process status D in top/ps) and effectively go to sleep, consuming next-to-no CPU cycles until the I/O request completes.

As for what your current problem is, that's impossible to say without more details about what your code is doing, ideally including source code, but a general description of your algorithms may well be sufficient.

Do note, though, that with 8 or 9 processes running, you're guaranteed to peg the CPU at 100% if a single instance averages any more than 11-12% CPU usage. You haven't said anything about average per-process CPU consumption, but, if it regularly bounces up to 24%, then I wouldn't be at all surprised to find that it averages half that or higher.