High %wa CPU load when running PHP as CLI

Question

Sorry for the vague question, but I've just written some php code that executes itself as CLI, and I'm pretty sure it's misbehaving. When I run "top" on the command line it's showing very little resources given to any individual process, but between 40-98% to iowait time (%wa). I usually have about .7% distributed between %us and %sy, with the remaining resources going to idle processes (somewhere between 20-50% usually).

This server is executing MySQL queries in, easily, 300x the time it takes other servers to run the same query, and it even takes what seems like forever to log on via SSH... so despite there being some idle cpu time left over, it seems clear that something very bad is happening. Whatever scripts are running, are updating my MySQL database, but it seems to be exponentially slower then when they started.

I need some ideas to serve as launch points for me to diagnose what's going on.

Some things that I would like to know are:

1. How I can confirm how many scripts are actually running
2. Is there anyway to confirm that these scripts are actually shutting down when they are through, and not just "hanging around" taking up CPU time and memory?
3. What kind of bottlenecks should I be checking to make sure I don't create too many instances of this script so this doesn't happen again.

I realize this is probably a huge question, but I'm more then willing to follow any links provided and read up on this... I just need to know where to start looking.

Answer 1

High iowait may mean you have a slow/defective disk. Try checking it out with a S.M.A.R.T. disk monitor.

http://www.linuxjournal.com/magazine/monitoring-hard-disks-smart

1. ps auxww | grep SCRIPTNAME
2. same.
3. Why are you running more than one instance of your script to begin with?

Answer 2

High iowait means that your disk bandwidth is saturated. This might be just because you're flooding your MySQL server with too many queries, and it's maxing out the disk trying to load the data to execute them.

Alternatively, you might be running low on physical memory, causing large amounts of disk IO for swapping.

To start diagnosing, run vmstat 60 for 5 minutes and check the output - the si and so columns show swap-in and swap-out, and the bi and bo lines show other IO. (Edit your question and paste the output in for more assistance).