Python 3 concurrent.futures socket server works with ThreadPoolExecutor but not ProcessPoolExecutor

Question

I am trying to create a simple socket server using the new concurrent.futures classes. I can get it to work fine with ThreadPoolExecutor but it just hangs when I use ProcessPoolExecutor and I can't understand why. Given the circumstance, I thought it might have something to do with trying to pass something to the child process that wasn't pickle-able but I don't that is the case. A simplified version of my code is below. I would appreciate any advice.

import concurrent.futures
import socket, os

HOST = ''
PORT = 9001

def request_handler(conn, addr):
    pid = os.getpid()
    print('PID', pid, 'handling connection from', addr)
    while True:
        data = conn.recv(1024)
        if not data:
            print('PID', pid, 'end connection')
            break
        print('PID', pid, 'received:', bytes.decode(data))
        conn.send(data)
    conn.close()

def main():
    with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
            sock.bind((HOST, PORT))
            sock.listen(10)
            print("Server listening on port", PORT)
            while True:
                conn, addr = sock.accept()
                executor.submit(request_handler, conn, addr)
                conn.close()
            print("Server shutting down")


if __name__ == '__main__':
    main()

Answer 1

Good call Donkopotamus, you should have posted that as the answer.

I wrapped the handler up in a try-block and when it attempts to call conn.recv() I got the exception: [Errno 10038] An operation was attempted on something that is not a socket. Since the worker process was dying badly, it deadlocked. With the try-block in place catching the error, I can continue to connect as much as I want without any deadlock.

I do not get this error if I change the code to use a ThreadPoolExecutor. I have tried several options and it looks like there is no way to properly pass a socket to the worker process in a ProcessPoolExecutor. I have found other references to this around the internet but they say it can be done on some platforms but not others. I have tried it on Windows, Linux and MacOS X and it doesn't work on any of them. This seems like a significant limitation.

Answer 2

are you sure it is not caused by 'conn.close()' after you run 'executor.submit(...)'? I just remove the line and try on mac10.12.6 and python 3.6.3 it works well.