Bad File descriptor with uWSGI native async websockets and redis

Question

Hi I have a simple websocket server which is pushing messages to clients, the code is as follows

    uwsgi.websocket_handshake(env['HTTP_SEC_WEBSOCKET_KEY'], env.get('HTTP_ORIGIN', ''))
    print("websockets...")
    r = redis.StrictRedis(host='localhost', port=6379, db=0)
    channel = r.pubsub()
    channel.subscribe('backchannel')

    websocket_fd = uwsgi.connection_fd()
    redis_fd = channel.connection._sock.fileno()

    while True:
        uwsgi.wait_fd_read(websocket_fd, 3)
        uwsgi.wait_fd_read(redis_fd)
        uwsgi.suspend()
        fd = uwsgi.ready_fd()
        if fd > -1:
            if fd == websocket_fd:
                msg = uwsgi.websocket_recv_nb()
                if msg:
                    r.publish('backchannel', msg)
            elif fd == redis_fd:
                msg = channel.parse_response() 
                print(msg)
                # only interested in user messages
                t = 'message'
                if sys.version_info[0] > 2:
                    t = b'message'
                if msg[0] == t:
                    uwsgi.websocket_send("[%s] %s" % (time.time(), msg))
        else:
            # on timeout call websocket_recv_nb again to manage ping/pong
            msg = uwsgi.websocket_recv_nb()
            if msg:
                r.publish('backchannel', msg)

        r.publish('backchannel', "Resistence is Futile!")

This code when executed results in the following error after pushing about 500 or so messages.

epoll_ctl(): Bad file descriptor [core/event.c line 520]
Traceback (most recent call last):
  File "SocketServer.py", line 71, in application
    uwsgi.wait_fd_read(redis_fd)
IOError: unable to fd 9 to the event queue
epoll_ctl(): Bad file descriptor [core/event.c line 635]

I know I am sending the last message in an infinite loop but I am doing this to test the limits of the system. What I would like to know is the cause of failure, and if there is anything I can do to get the system to push more messages before failing.

I am running this code on ubuntu 12.04 with the following command uwsgi --http :8080 --http-websockets --async=1000 --ugreen --wsgi-file SocketServer.py

runing the sample async chat client app https://github.com/unbit/uwsgi/blob/master/tests/websockets_chat_async.py results in the same error when run under heavy user load.

Answer 1

Strace and Lsof are your friends. Strace will give you a list of system calls made by your running process. Attach it to the running process using "strace -p " to see the list of system calls, then try to reproduce the situation. Hopefully you'll find something useful in the stream of events occured. Lsof will give you a map of file descriptors to actual files.