multiple threads doing poll() or select() on a single socket or pipe

Question

What do POSIX and other standards say about the situation where multiple threads are doing poll() or select() calls on a single socket or pipe handle at the same time?

If any data arrives, does only one of the waiting threads get woken up or do all of the waiting threads get woken up?

Answer 1

Interesting question ... I read through the current POSIX and did not find a specific answer, i.e., no specification about concurrent invocations. So I'll explain why I think the standard means all will wake up.

The relevant part of the text for select / pselect is:

Upon successful completion, the pselect() or select() function shall modify the objects pointed to by the readfds, writefds, and errorfds arguments to indicate which file descriptors are ready for reading, ready for writing, or have an error condition pending, respectively, [...]

and later

A descriptor shall be considered ready for reading when a call to an input function with O_NONBLOCK clear would not block, whether or not the function would transfer data successfully. (The function might return data, an end-of-file indication, or an error other than one indicating that it is blocked, and in each of these cases the descriptor shall be considered ready for reading.)

In short (the reading case only), we can understand this as:

select does not block this means that the next call to an input function with O_NONBLOCK would not return an error with errno==EWOULDBLOCK. [Note that the "next" is my interpretation of the above.]

If one admits to this interpretation then two concurrent select calls could both return the same FD as readable. In fact even if they are not concurrent, but a first thread calls select with some FD being readable and later e.g., read, a second thread calling select between the two could return the FD as readable for the second thread.

Now the relevant part for the "waking up" part of the question is this:

If none of the selected descriptors are ready for the requested operation, the pselect() or select() function shall block until at least one of the requested operations becomes ready, until the timeout occurs, or until interrupted by a signal.

Here clearly the above interpretation suggests that concurrently waiting calls will all return.

Answer 2

I just found a bug because of this question: I have two threads selecting on the same socket, and will call accept when the fd comes back as isset(). In fact the select comes back for both threads, the fd isset() for that fd in both threads, and both threads call accept(), one wins and the other blocks waiting for another connection to come in.

So in fact select will return in all threads that it is blocking on for the same fd.

Answer 3

They should all wake up, all return the same result value, and all all do the same thing to the FD sets. They are all asking the same question, so they should all get the same answer.

What select() is supposed to do, according to the POSIX documentation which has been cited here, and my mere 25 years' exerience with it, is to return the number of FDs that are readable, writable, etc., at that instant. It would therefore be completely incorrect for all the concurrent select() calls not to all return the same thing.

The select() function can't predict the future, i.e. which thread is actually going to do a read or write, and therefore which thread will succeed in that. They contend. It's a thundering-herd problem.

Answer 4

To avoid system overloading, I did the work on kernel to implement EPOLLEXCLUSIVE for Linux epoll. This flag set to resource will ensure that only one listener will receive event even in case if several threads or processes are listening on give file descriptor via epoll() (linux version of poll/select). This is very useful feature. As for example Enduro/X middleware is multi-process based middleware where several load balanced executables monitor same set of file descriptors (queues) by using epoll. Thus when event arrives without EPOLLEXCLUSIVE, lot of processes gets false wake ups (where some first wake up did already remove the event from FD - thundering herd issue) and others get empty notifications. And that empty processing cost a CPU processing time, if having say 500 binaries waiting for events...

IBM AIX 7.3 introduced new poll() flag - POLLEXCL which functions in similar way as EPOLLEXCLUSIVE.