Gracefully shutdown UNIX-socket server on NodeJS running under Forever

Question

I have an NodeJS application which sets up a UNIX-socket to expose some interprocess communication channel (some kind of monitoring stuff). UNIX-socket file is placed in os.tmpdir() folder (i.e. /tmp/app-monitor.sock).

var net = require('net');
var server = net.createServer(...);
server.listen('/tmp/app-monitor.sock', ...);

I use a signal handling (SIGINT, SITERM, etc...) to gracefully shutdown my server and remove a socket file.

function shutdown() {
    server.close(); // socket file is automatically removed here
    process.exit();
}

process.on('SIGINT', shutdown);
// and so on

My application is running with forever start ... to monitor it's lifecycle.

I have a problem with forever restartall command. When forever doing restartall it's using a SIGKILL to terminate all child processes. SIGKILL can't be handled by a process so my app dies without any shutdown procedures.

The problem is a socket file which is not removed when SIGKILL is used. After the child process is restarted, new server can't be started cause' a listen call will cause a EADDRINUSE error.

I can't remove a existing socket file during an app startup cause' I don't know if it a real working socket or some traces of a previous unclean shutdown.

So, the question is... What is the better way to handle such situation (SIGKILL and UNIX-socket server)?

Answer 1

As other people have mentioned, you cannot do anything in response to SIGKILL, which is in general why forever (and everybody else) should not be using SIGKILL except in extreme circumstances. So the best you can do is clean up in another process.

I suggest you clean up on start. When you get EADDRINUSE, try to connect to the socket. If the socket connection succeeds, another server is running and so this instance should exit. If the connection fails then it is safe to unlink the socket file and create a new one.

var fs = require('fs');
var net = require('net');
var server = net.createServer(function(c) { //'connection' listener
    console.log('server connected');
    c.on('end', function() {
        console.log('server disconnected');
    });
    c.write('hello\r\n');
    c.pipe(c);
});

server.on('error', function (e) {
    if (e.code == 'EADDRINUSE') {
        var clientSocket = new net.Socket();
        clientSocket.on('error', function(e) { // handle error trying to talk to server
            if (e.code == 'ECONNREFUSED') {  // No other server listening
                fs.unlinkSync('/tmp/app-monitor.sock');
                server.listen('/tmp/app-monitor.sock', function() { //'listening' listener
                    console.log('server recovered');
                });
            }
        });
        clientSocket.connect({path: '/tmp/app-monitor.sock'}, function() { 
            console.log('Server running, giving up...');
            process.exit();
        });
    }
});

server.listen('/tmp/app-monitor.sock', function() { //'listening' listener
    console.log('server bound');
});

Answer 2

you should be able to use SIGTERM to do what you want: process.on('SIGTERM', shutdown)

Answer 3

server.on('error', function (e) {
  if (e.code == 'EADDRINUSE') {
    console.log('Address in use, retrying...');
    setTimeout(function () {
      server.close();
      server.listen(PORT, HOST);
    }, 1000);
  }
});

http://nodejs.org/api/net.html#net_server_listen_port_host_backlog_callback

UPD

you cant handle SIGKILL, then you must cleanup socket manual

this example work fine with forever

var fs = require('fs');
var net = require('net');
var server = net.createServer(function(c) {});
server.listen('./app-monitor.sock', function() {
  console.log('server bound');
});

server.on('error', function (e) {
  if (e.code == 'EADDRINUSE') {
    console.log('Address in use, retrying...');
    setTimeout(function () {
      fs.unlink('./app-monitor.sock');
    }, 1000);
  }
});

Answer 4

Since you cannot handle SIGKILL and you want to use forever (which uses SIGKILL) you will have to use a workaround.

For example, first send a signal to shutdown your server and then do the forever restart:

kill -s SIGUSR1 $pid
# $pid contains the pid of your node process, or use
# killall -SIGUSR1 node

sleep 2
forever restart test.js

In your js handle SIGUSR1:

process.on('SIGUSR1', gracefullyShutdownMyServer);

Answer 5

you must choose

1. change SIGKILL to another signal in forever-monitor to handle in app
2. take care of your app in this case, using fs.unlink
3. stop using forever