epoll with edge triggering, one shot and multithreading

Question

This is a question regarding this answer: https://stackoverflow.com/a/14241095/2332808 (would comment it but newly created accounts apparently can't, sorry for the noise. Ressources on epollet/multithreading are hard to find...)

It suggests using epoll as the following:

1. epoll_ctl() to activate notifications (and reactivate if EPOLLONESHOT is used).
2. system input: read()/recv()/accept() in a loop until error EAGAIN.
3. epoll_wait() to receive notifications.

But, assuming there are multiple threads in epoll_wait() on the same epollfd, wouldn't this risk having another thread being woken up on the same fd if it receives more data before you're done reading (e.g. ending up with two threads processing the same fd)

Even if you turn things around and read() till EAGAIN, epoll_ctl() and then call read() again to check there's still nothing (to avoid the race where you'd receive something between the last read and the epoll_ctl())...

BUT there is still no guarantee that you wouldn't have actually received something after the epoll_ctl() and would have both the last read() check and another thread woken up working on the same fd again...

I guess having a lock per fd would be an acceptable solution, but is that the "approved" use of epoll in edge-triggering mode with multiple threads polling on the same epollfd ?

Answer 1

Yes, you still need to do proper locking to guard against those cases you describe - and using a lock per fd is the most sensible approach to do that.