Does foreman restart processes gracefully?

Question

I'm about to use foreman with upstart export in production to start some infinite rake tasks.

My humble "daemons" are as simple as this:

task magic: :environment do
  loop do
    make_stuff_happen

    sleep 10
  end
end

I'd like to know if foreman doesn't just kill processes when asked to restart (eg. capistrano deploy), since I don't want any corrupt operations to occur because of this.

And if so how to prevent it?

Answer 1

if you do the export, you may have a file on the configuration like app-jekyll-1.conf which contain something like this

start on starting app-jekyll                                                                                               
stop on stopping app-jekyll
respawn

exec su - username -c 'cd /some/directory; export PORT=5000; bundle exec jekyll serve -w >> /tmp/a    pp.log/jekyll-1.log 2>&1'

which is basically the configuration of the ubuntu upstart.

now when you restart a service with upstart, it actually will call initctl restart

http://upstart.ubuntu.com/cookbook/#restart

calling initctl restart will send SIGTERM signal to the job

https://serverfault.com/questions/189780/what-signal-does-upstart-initctl-use-to-restart-a-job

Now to cleaning up the resource is depending whether your program do a clean-up when it receive a SIGTERM signal.

The question if foreman will kill the rake process when you asked to restart, then the answer is yes it will kill the process by sending SIGTERM to the rake process.

By default when rake received SIGTEM, it will exit the process.

for example, if this is your Rakefile

def do_some_magic
  puts "doing some magic.."
  sleep 5
end

task :magic do
  puts "Running the task wit PID #{$$}"
  loop do
    do_some_magic

    sleep 5
  end
end

and then if you run it

$ rake magic
Running the task wit PID 29527
doing some magic..

On Separate terminal, you can send SIGTERM to the rake process with the given PID. and it will throw an exception

$ kill 29527

and on the previous terminal you will get

...                         
doing some magic..                                            
rake aborted!                                                 
SIGTERM                                                       
/tmp/foo/Rakefile:11:in `sleep'                               
/tmp/foo/Rakefile:11:in `block (2 levels) in <top (required)>'
/tmp/foo/Rakefile:8:in `loop'                                 
/tmp/foo/Rakefile:8:in `block in <top (required)>'            
Tasks: TOP => magic                                           
(See full trace by running task with --trace)

So that's basically rake died. It will immediately stop everything. You may leave inconsistent state if do_some_magic is currently performing DB operation for example.

You can of course handle this signal to perform some clean up. This is how you can handle it if you want to wait until do_some_magic finished

def do_some_magic                          
  puts "doing some magic.."                
  sleep 5                                  
end                                        

task :magic do                             
  puts "Running the task wit PID #{$$}"    
  loop do                                  
    do_some_magic                          

    if $shutdown                           
      puts "shutting down..., exiting loop"
      break                                
    end                                    
  end                                      
end                                        

trap "TERM" do                             
  puts "SIG TERM received."                
  $shutdown = true                         
end                                        

and again run it..

$ rake magic
Running the task wit PID 30150                     
doing some magic..                                 

and also on the separater terminal you kill the process with the pid again

$ kill 30150

on you previous terminal the process will be something like this

....
doing some magic..             
SIG TERM received.             
shutting down..., exiting loop 

Conclusion

• foreman export will just create an ubuntu upstart script.
• Restart mechanism will be handled by the OS. Which is sending SIGTERM to the process.
• It depend on the actual process to maintain the state of the system.
• Rake by default doesn't handle SIGTERM. you have to do it your self to make sure if the task is interrupted, you do a clean-up to make sure the state of the system is not corrupted

Hopes it help