How to avoid zombie processes when killing rake tasks?

Question

I have a bash script that calls a few rake tasks in succession. This script is called by a continuous integration server. The problem is that if the integration server kills the task (by sending the script a SIGTERM), I can't get the entire process tree to get killed. For instance, if Rake is running a set of rspec tests using the Parallel feature, the script and top level rake command can be killed, but inevitably some threads continue.

So far, I've tried many different ways of using the 'trap' command inside the bash script. I've tried finding all subprocess with pstree, using "kill -9 0" to kill the current process group, etc. If the script is killed while a rake tasks is running, however, it seems to bypass the trap. As far as I can tell Rake, or perhaps Ruby's threading library, does something odd to the process group or possibly interferes with the trap.

If the script is not in the middle of a rake task, it will trap the SIGTERM correctly, echo "Received SIGTERM", then kill all descendants.

How is it possible for Rake to interfere with a trap statement in the parent process (bash script)?

Answer 1

Bash assumes that the entire process group received the signal, so it will wait for the currently executing command to finish to determine what to do (more info). The trap will not run until the command has finished.

If you don't want to wait for the command to finish, you can instead use

trap 'kill $!' TERM
job &
wait $!

wait will exit immediately when the shell receives SIGTERM, causing your trap to run at once.