Listening Twisted TCP connection with python-daemon gives bad file descriptor

Question

I'm trying to create a program which:

• fork at start using multiprocessing
• the forked process uses python-daemon to fork again in the background
• opening a twisted listening TCP port in the resulting background process

The reason I need to fork the process before launching python-daemon is because I want the staring process to stay alive (by default python-daemon kills the father process).

So far my code is:

from twisted.web import xmlrpc, server
from twisted.internet import reactor
from daemon import daemon
import multiprocessing
import os
import logging

class RemotePart(object):

    def setup(self):
        self.commands = CommandPart()
        reactor.listenTCP(9091, server.Site(self.commands))

class CommandPart(xmlrpc.XMLRPC, object):

    def __init__(self):
        super(CommandPart, self).__init__()

    def xmlrpc_stop(self):
        return True


class ServerPart(object):

    def __init__(self):
        self.logger = logging.getLogger("server")
        self.logger.info("ServerPart.__init__()")

    def start_second_daemon(self):
        self.logger.info("start_second_daemon()")        
        daemon_context = daemon.DaemonContext(detach_process=True)
        daemon_context.stdout = open(
            name="log.txt", 
            mode='w+', 
            buffering=0
        )
        daemon_context.stderr = open(
            name="log.txt", 
            mode='w+', 
            buffering=0
        )
        daemon_context.working_directory = os.getcwd()

        daemon_context.open()
        self.inside_daemon()

    def inside_daemon(self):
        self.logger.setLevel(0)
        self.logger.info("inside daemon")
        self.remote = RemotePart()
        self.remote.setup()
        reactor.run()

class ClientPart(object):

    def __init__(self):
        logging.basicConfig(level=0)
        self.logger = logging.getLogger("client")
        self.logger.info("ClientPart.__init__()")

    def start_daemon(self):
        self.logger.info("start_daemon()")
        start_second_daemon()

    def launch_daemon(self):
        self.logger.info("launch_daemon()")
        server = ServerPart()
        p = multiprocessing.Process(target=server.start_second_daemon())
        p.start()
        p.join()

if __name__ == '__main__':
    client = ClientPart()
    client.launch_daemon()

Starting the process seems to work:

INFO:client:ClientPart.__init__()
INFO:client:launch_daemon()
INFO:server:ServerPart.__init__()
INFO:server:start_second_daemon()

But looking to the log file of the background process, twisted cannot open the TCP port:

INFO:server:inside daemon
Traceback (most recent call last):
  File "forking_test.py", line 74, in <module>
    client.launch_daemon()
  File "forking_test.py", line 68, in launch_daemon
    p = multiprocessing.Process(target=server.start_second_daemon())
  File "forking_test.py", line 45, in start_second_daemon
    self.inside_daemon()
  File "forking_test.py", line 51, in inside_daemon
    self.remote.setup()
  File "forking_test.py", line 12, in setup
    reactor.listenTCP(9091, server.Site(self.commands))
  File "/usr/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 482, in listenTCP
    p.startListening()
  File "/usr/lib/python2.7/site-packages/twisted/internet/tcp.py", line 1004, in startListening
    self.startReading()
  File "/usr/lib/python2.7/site-packages/twisted/internet/abstract.py", line 429, in startReading
    self.reactor.addReader(self)
  File "/usr/lib/python2.7/site-packages/twisted/internet/epollreactor.py", line 247, in addReader
    EPOLLIN, EPOLLOUT)
  File "/usr/lib/python2.7/site-packages/twisted/internet/epollreactor.py", line 233, in _add
    self._poller.register(fd, flags)
IOError: [Errno 9] Bad file descriptor

Any idea ? It seems python-daemon is closing all the file descriptors of the background process when this one starts, could it be coming from this behavior ?

Answer 1

There are lots of reasons that running fork and then running some arbitary library code that doesn't work. It would be hard to list them all here, but generally it's not cool to do. My guess as to what's specifically happening here is that something within multiprocessing is closing the "waker" file descriptor that lets Twisted communicate with its thread pool, but I can't be completely sure.

If you were to re-write this to:

1. Use spawnProcess instead of multiprocessing
2. Use twistd instead of python-daemonize

the interactions would be far less surprising, because you'd be using process-spawning and daemonization code specifically designed to work with Twisted, instead of two things with lots of accidental platform interactions (calling fork, serializing things over pipes with pickle, calling setsid and setuid and changing controlling terminal and session leader at various times).

(And actually I would recommend integrating with your platform's daemon management tools, like upstart or launchd or systemd or a cross-platform one like runit rather than depending on any daemonization code, including that in twistd, but I would need to know more about your application to know what to recommend.)