How to tell which file Apache is waiting for when WCHAN = flock_lock_file_wait

Question

We're having an issue with our webservers in which more and more Apache processes are getting stuck waiting for file lock (caused by PHP flock()) to somehow resolve, but never does. Eventually the site gets slower and slower, and finally stops completely, until Apache is restarted.

I'm working on the theory that somehow bad code is behind this, perhaps in our homegrown disk cache mechanism. But I can't track it down. (We are running on Ubuntu.)

My question is, how can I tell what file or files it's getting stuck on? If I could see where the hangup is, I could much more easily figure out where the bad code is. When I do this command:

ps -o pid,tt,user,fname,wchan -C apache2 

I get this:

  730 ?        www-data apache2  flock_lock_file_wait  
 3085 ?        www-data apache2  flock_lock_file_wait  
 5393 ?        www-data apache2  flock_lock_file_wait  
 5397 ?        www-data apache2  flock_lock_file_wait  
11181 ?        www-data apache2  flock_lock_file_wait  
30280 ?        www-data apache2  flock_lock_file_wait   

Is there any way for me to go from this to seeing which exact file Apache is locked on / waiting for?

Answer 1

You'll have to use the lsof utiliy:

apt-get install lsof

lsof mean list open files. A lot of things can be done with it, lsof -ni will list opened network connections for example. For your apache processes you'll have to use the -p option (PID):

lsof -p 730

And you'll get all the opened files and libraries (eveything is a file on linux), so obviously you'll get your lock file.