setsockopt on “accepted” fd on Linux

Question

I have had a rather strange observation about behavior of setsockopt on Linux for SO_REUSEADDR. In one line: if I apply the sockopt to an fd returned by accept on a "listening socket" the socketoption is reflected on the port held by the listening socket.

Ok some code.

Server : Opens a socket, applies SO_REUSEADDR to be true. Accepts a connection and then applies SO_REUSEADDR to be false on the fd on the fd returned by accept.

#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <arpa/inet.h>
#include <string.h>

int main(void)
{
        int s,  len;
        int sin_size;
        int reuse = 1;
        int ret;
        struct sockaddr_in my_addr;

        memset(&my_addr, 0, sizeof(my_addr));
        my_addr.sin_family = AF_INET;
        my_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
        my_addr.sin_port = htons(33235);

        if( (s = socket(AF_INET, SOCK_STREAM, 0)) < 0)
        {
               printf("Socket Error\n");
               return -1;
        }

        setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(int));

        if( bind(s, (struct sockaddr*)&my_addr, sizeof(struct sockaddr))  < 0)
        {
               printf("Bind Error\n");
               return -1;
        }

        listen(s, 6);

        reuse = 0;
        memset(&my_addr, 0, sizeof(my_addr));
        while(1) {
           ret = accept(s, (struct sockaddr*)&my_addr, &len);
           if (ret<0) {
              printf("Accept failed\n");
           } else {
              printf("Accepted a client setting reuse add to 0\n");
              setsockopt(ret, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(int));
           }
        }

        printf("Server exiting\n");

        return 0;
}

Client : Client connects to the server, and doesn't do anything after that ensuring that the server socket stays in TIME_WAIT state.

#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <arpa/inet.h>
#include <string.h>
#include <errno.h>

int main(void)
{
        int s,  len;
        int sin_size;
        struct sockaddr_in my_addr;

        memset(&my_addr, 0, sizeof(my_addr));
        my_addr.sin_family = AF_INET;
        my_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
        my_addr.sin_port = htons(33235);

        if( (s = socket(AF_INET, SOCK_STREAM, 0)) < 0)
        {
               printf("Socket Error\n");
               return -1;
        }

        if (!connect(s,(struct sockaddr*)&my_addr, sizeof(struct sockaddr)))
        {
           printf("Client Connected successfully\n");
        }
        else
        {
           printf("%s\n",strerror(errno));
        }

        while(1) sleep(1);

        return 0;
}

Steps that I do reproduce the issue.

1. Run server.
2. Connect client.
3. Kill and restart server. The server fails with Bind Failure

I tested this on mac os. And the bind didn't fail. I have digged up all Posix specifications and none of them say that this code is undefined.

Question:

Can someone with more experience on this share their understanding of the issue?

Answer 1

One way to think about it is that SO_REUSEADDR determines if you can have another socket bound to that same address. It's a property of any socket (listen or connection), but very commonly inherited from listen via accept. In linux it's mapped to the struct sock "sk_reuse" flag.

If you clear this flag on a FD you "accepted" then from that point on the IP/Port pair is considered busy-and-non-reusable. The SO_REUSEADDR flag on the listen socket does not change, but the flag on the accepted socket affects bind logic. You could probably check this with getsockopt.

If you want to know more you can try to read the inet_csk_get_port function: http://lxr.free-electrons.com/source/net/ipv4/inet_connection_sock.c#L100. This is where the actual "binding" takes place.