Python asyncore: “filedescriptor out of range in select()”

Question

I have some python script with more than 3k outgoing socket connections, based on asyncore lib. I can't use select(..) due to connections limit (1024), but poll(..) not working properly too:

asyncore.loop(use_poll=True)

With this invocation my app ignores any socket events. Note that select() on less than 1024 sockets works fine.

Where is my problem?

Answer 1

First, a minor correction. use_poll=True does not cause asyncore to use epoll(2). It causes it to use poll(2) (moreover, the way asyncore works, there is little reason to try to use epoll(2), because the Python-level overhead of asyncore overwhelms any cost of the socket event notification API being used).

Second, a major shortcoming of asyncore is the degree to which it exposes you to platform-specific quirks. For a library that handles more of the differences between select(2) and poll(2) for you, and which actually supports epoll(2), and which is better in a number of other ways too, check out Twisted.

If you can add more details to your question, perhaps the specific problem you're encountering on asyncore can be discovered, but based on the information available now, there's no way to be sure. The very basic features of asyncore's poll(2) support do work - ie, it can deliver read, write, and close notifications in the trivial case. How does your case differ from the trivial case?