Determine what's blocking a process on OSX using command line tools?

Question

The title says it most, really. On Linux it would be easy with strace and possibly lsof or /proc, and it used to be easy on OSX until truss was removed from OSX Leopard, along with underlying system calls (afaik).

The obvious approach is to tackle this problem with dtrace, but alas as far as I understand dtrace won't do because it catches events as they happen - and in my case, the blocking system call has already started. I'd love to stand corrected if this can be solved with dtrace, by the way.

I saw Xcode's Instruments has a monitor which achieves something similar by taking periodic samples of a process' stack (not sure what system calls it relies on to do that!), maybe something similar on the command line could be good enough (as it would display the stack all the way to the library call that wraps the system call). To be useful for my usecase, this "sampling commandline tool" would have to find and parse the arguments it finds on the stack to be useful to determine what file/file descriptor are we blocked on.

One last thing - on Linux, you could usually do this as a regular user (assuming no ptrace_scope tricks). It would be great if the OSX solution won't require root, either.

Answer 1

You can use sample: sample PID -e

E.g. for the nc -l localhost 5999 you'd get a file with call graph:

Call graph:
    9046 Thread_242504   DispatchQueue_1: com.apple.main-thread  (serial)
      9046 start  (in libdyld.dylib) + 1  [0x7fff90a847e1]
        9046 ???  (in nc)  load address 0x102453000 + 0x166c  [0x10245466c]
          9046 __accept  (in libsystem_kernel.dylib) + 10  [0x7fff94445996]

And other useful information like loaded Binary Images.

Answer 2

I'm proposing a solution that assumes the following:

1. the process can be signaled.
2. the process is single threaded (a signal might not be handled in the blocking thread).
3. you have root access.

dtruss supports printing a complete stack trace for every system call (the -s argument).

example:

terminal 1:

$ python
>>> import socket
>>> s = socket.socket()
>>> s.bind(('', 1234))
>>> s.listen(1)
>>> s.accept() # blocking!

terminal 2:

$ dtruss -s -p `pgrep python` # or your python pid if you don't have pgrep (port install proctools)

terimal 1:

press Ctrl-C

terminal 2:

sigreturn(0x7FFF5FBFD660, 0x1E, 0x10031B3D0)             = 0 Err#-2
          libsystem_kernel.dylib`__accept+0xa # HERE IT IS!
          libpython2.7.dylib`PyEval_EvalFrameEx+0x42bf
          libpython2.7.dylib`fast_function+0xb3
...

the first frame in the stack if sigreturn (teardown code for signal handlers: http://linux.die.net/man/2/sigreturn).

the second frame is the standard library wrapper for our system call: accept.