apache/php hangs: semop?

Question

I am writing a wordpress plugin to include a custom made single sign-on.

I have a problem I can not reproduce but it happens often: Sometimes you login, click a another page and this page keeps loading for ever. Also all other request from the same session will stuck from this time. Restarting apache fixes it but is not a real solution.

The server has not a lot traffic so only a few apache processes are running. If I dig into them I see many stucked with semop:

(gdb) bt
#0  0x00002af60c22b2d7 in semop () from /lib64/libc.so.6
#1  0x00002af60bbf612c in ?? () from /usr/lib64/libapr-1.so.0
#2  0x000055555559e614 in ?? () from /usr/sbin/httpd2-prefork
#3  0x000055555559e9ea in ?? () from /usr/sbin/httpd2-prefork
#4  0x000055555559f25d in ap_mpm_run () from /usr/sbin/httpd2-prefork
#5  0x000055555557a080 in main () from /usr/sbin/httpd2-prefork

looks like they are waiting for a file:

strace -p 3069
....
read(7, 0x7fff16a04df7, 1)              = -1 EAGAIN (Resource temporarily unavailable)
semop(286162952, 0x2af60bd07dc0, 1 <unfinished ...>

read(7 ,..) points to a pipe:

# ls -la /proc/3069/fd/7
lr-x------ 1 root   root 64 Nov  7 17:24 7 -> pipe:[157329520]

That pipe is used by all the apache processes:

# lsof | grep 157329520
httpd2-pr  2430       root    7r     FIFO                0,5             157329520 pipe
httpd2-pr  2430       root    8w     FIFO                0,5             157329520 pipe
httpd2-pr  3061     wwwrun    7r     FIFO                0,5             157329520 pipe
httpd2-pr  3061     wwwrun    8w     FIFO                0,5             157329520 pipe
...

It might be not related to the problem but I still wounder what kind of pipe this is.

People here are guessing it's not a server problem after all but from what I see in the backtrace PHP is not even called yet. No libphp5.so at all.

I would love to post some PHP code for you to look at, but I just don't know which part... My script makes a curl call to another script in the same session. Maybe this is creating some kind of blocking? Or could it be https problem?

I really can't tell if it is PHP or Apache but the key to the problem might be to understand what apache is doing here.

Any idea how to fix or debug this?

Answer 1

In the strace output it looks like it is waiting for file-pointer number seven. You can find out which file this is by looking for a line like

open("/foo/bar", ......)     = 7

somewhere before the read call.