Multi Threaded Server design

Question

I am trying to implement a TCP server which is a part of a larger project. Basically the server should be able to maintain a TCP connection with any number of clients (a minimum of 32) and service any client that requests servicing. In our scenario the thing is that it will be assumed that once the client is connected to the server, it will never close the connection unless some sort of failure occurs (e-g the machine running the client breaks down ) and it will repeatedly request service from the server. Same is the case with all the other clients i-e each will maintain a connection with the server and perform transactions. so to sum up the server will be at the same time maintaining the connection with the clients while simultaneously serving each client as needed and should also have the ability to accept any other client connections that want to connect to the server.

Now I implemented the above functionality using the select() system call of the berkely socket API and it works fine when we have a small number of clients (say 10). But the server needs to be scaled to the highest possible level as we are implementing it on a 16 core machine. For that I looked through various multi threading design techniques e-g one thread per client etc and the best one in my opinion would be a thread pool design. Now As I was about to implement that I ran into some problems: If I designate the main thread to accept any number of incoming connections and save each connections File descriptor in a data structure, and I have a pool of threads, how would I get the threads to poll that whether a particular client is requesting for service or not. The design is simple enough for scenarios in which client contacts the server and after getting the service it closes the connection so that we can pick a thread from a pool, service the client and then push it back into the pool for future connection handling. But when we have to service a set of clients that maintain a connection and request services intermittently, what would be the best approach to do this. All help will be much appreciated as I am really stuck in this. Thanks.

Answer 1

Use pthreads, with one thread per CPU plus one extra thread.

The extra thread (the main thread) listens for new connections with the listen() system call, accepts the new connections with accept(), then determines which worker thread currently has the least number of connections, acquires a lock/mutex for that worker thread's "pending connections" FIFO queue, places the descriptor for the accepted connection onto the worker thread's "pending connections" FIFO queue, and sends a "check your queue" notification (e.g. using a pipe) to the worker thread.

The worker threads use "select()", and send/receive data to whatever connections they've accepted. If/when a worker thread receives a "check your queue" notification from the main thread it would acquire the lock/mutex for its "pending connections" FIFO queue and add any newly accepted connections to its "fd_set" list.

For 1024 connections and 16 CPUs; you might end up with one main thread waiting for new connections (but doing almost nothing as you wouldn't be expecting many new connections), and 16 worker threads handling an average of 64 connections each.

Answer 2

One thread per client is almost certainly the best design. Make sure you always have at least one thread blocked in accept waiting for a new connection - this means that after accept succeeds, you might need to create a new thread before proceeding if it was the last one. I've found semaphores to be a great primitive for keeping track of the need to spawn new listening threads.