How to change TCP Server In C from Blocking Mode to Non-Blocking Mode when it's already blocking Or How to shutdown a blocking TCP Server properly?

Question

I have no problems with running the TCP Server and I like the fact that it's in blocking to avoid useless loops and sleeping code and useless cpu cycles.

The problem happens when shutting it down in Linux environment, it stays on, until the connected user sends something, then it turns off.

I figured it's because it's blocking even though the endless while loop is set to exit. But when it's blocking changing the socket id's to NON_BLOCKING doesn't help at all, most likely has to be set to NON_BLOCKING before the block occurs.

#include <pthread.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <errno.h>
#include <fcntl.h> /* Added for the nonblocking socket */

#define LISTEN_MAX 10       /* Maximum clients that can queue up */
#define LISTEN_PORT 32000
#define MAX_COMMANDS_AT_ONCE 4000
#define NANO_SECOND_MULTIPLIER  1000000  // 1 millisecond = 1,000,000 Nanoseconds

//Global so I can access these where I shut the threads off.
int listenfd, connfd; //sockets that must be set to non-blocking before exiting

int needQuit(pthread_mutex_t *mtx)
{
    switch(pthread_mutex_trylock(mtx)) {
        case 0: /* if we got the lock, unlock and return 1 (true) */
        pthread_mutex_unlock(mtx);
        return 1;
        case EBUSY: /* return 0 (false) if the mutex was locked */
        return 0;
    }
    return 1;
}

/* this is run on it's own thread */
void *tcplistener(void *arg)
{
    pthread_mutex_t *mx = arg;
    //keyboard event.

    SDLKey key_used;
    struct timespec ts;
    //int listenfd,connfd,
    int n,i, ans;
    struct sockaddr_in servaddr,cliaddr;
    socklen_t clilen;
    pid_t     childpid;
    char mesg[MAX_COMMANDS_AT_ONCE];

    listenfd=socket(AF_INET,SOCK_STREAM,0);

    bzero(&servaddr,sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_addr.s_addr=htonl(INADDR_ANY);
    servaddr.sin_port=htons(LISTEN_PORT);
    int option = 1;
    if(setsockopt(listenfd, SOL_SOCKET,SO_REUSEADDR,(char*)&option,sizeof(option)) < 0)
    {
        printf("setsockopt failed\n");
        close(listenfd);
    }
    bind(listenfd,(struct sockaddr *)&servaddr,sizeof(servaddr));

    listen(listenfd,LISTEN_MAX);

    while( !needQuit(mx) )
    {
        clilen=sizeof(cliaddr);
        connfd = accept(listenfd,(struct sockaddr *)&cliaddr,&clilen);

        while( !needQuit(mx) )
        {
            n = recv(connfd,mesg,MAX_COMMANDS_AT_ONCE,0);

            if(n == 0 || n == -1) break;
            //...Do Stuff here with mesg...
        }
    }
    close(connfd);
}

close(connfd);
close(listenfd);
return NULL;
}

int main(int argc, char *argv[])
{
    /* this variable is our reference to the thread */
    pthread_t tcp_listener_thread;
    pthread_mutex_t mxq; /* mutex used as quit flag */

    /* init and lock the mutex before creating the thread.  As long as the
    mutex stays locked, the thread should keep running.  A pointer to the
    mutex is passed as the argument to the thread function. */
    pthread_mutex_init(&mxq,NULL);
    pthread_mutex_lock(&mxq);

    /* create a hread which executes tcplistener(&x) */
    if(pthread_create(&tcp_listener_thread, NULL, tcplistener, &mxq)) {
        fprintf(stderr, "Error creating TCP Listener thread\n");
        //clear thread for tcp listener on exit.
        /* unlock mxq to tell the thread to terminate, then join the thread */
        fcntl(listenfd, F_SETFL, O_NONBLOCK); /* Change the socket into non-blocking state  */
        fcntl(connfd, F_SETFL, O_NONBLOCK); /* Change the socket into non-blocking state    */

        pthread_mutex_unlock(&mxq);
        pthread_join(tcp_listener_thread,NULL);
        pthread_cancel(tcp_listener_thread);
        pthread_exit(NULL);
        return 0;
    }

    //End of the TCP Listener thread.


    // Waits 500 milliseconds before shutting down
    struct timespec ts;
    ts.tv_sec = 0;
    ts.tv_nsec = 500 * NANO_SECOND_MULTIPLIER;
    nanosleep((&ts, NULL);

    //Forces a shutdown of the program and thread.
    //clear thread for tcp listener on exit.
    /* unlock mxq to tell the thread to terminate, then join the thread */
    fcntl(listenfd, F_SETFL, O_NONBLOCK); /* Change the socket into non-blocking state  */
    fcntl(connfd, F_SETFL, O_NONBLOCK); /* Change the socket into non-blocking state    */
    pthread_mutex_unlock(&mxq);
    pthread_join(tcp_listener_thread,NULL);
    pthread_cancel(tcp_listener_thread);
    pthread_exit(NULL);
    return 0;
}

I tried the fix EJP suggested like so, still hanging.. I've made connfd and listernfd both global scope

pthread_mutex_unlock(&mxq); 
    close(connfd); //<- this
    close(listenfd); //<-- this
pthread_join(tcp_listener_thread,NULL);
pthread_cancel(tcp_listener_thread);
pthread_exit(NULL);

Answer 1

To unblock accept(), just close the listening socket. Make sure the code around accept() handles is correctly.

To unblock recv(), shutdown the receiving socket for input. That will cause recv() to return zero, which again must be handled correctly. Or else just close the socket as above, which might be better if you want the receive code to know that you're closing the application.

Answer 2

Your listener thread will indeed block in the accept().

The nasty way to fix this (almost) is to send a signal to the listener thread with pthread_kill(). This causes accept() to return with errno == EINTR, which you test for and then return.

However, that has a race condition: if the signal is received between testing the while (!needQuit(mx)) condition and entering the accept() then it'll be lost and the accept() will block again.

One correct way to solve this is to use something like select() and a pipe. You select for read over the pipe and the socket. When the main thread wants the listener thread to exit it writes a byte to the pipe. The listener thread's select() call returns either when a byte is readable from the pipe (in which case it exits) and/or when a client can be accepted.

Non-blocking sockets are primarily used to multiplex lots of sockets into one event loop (i.e. thread). That's a good idea for server scalability, but not necessary here.