FD_ISSET() after select() not detecting received data

Question

I have a program which I've designed to run on a bunch of different nodes to transfer files amongst themselves, based on the instructions they get from a process running on a master node. Each process serves as a sender and a receiver both.

My strategy is this :

//Setup a listener socket on port 63000(say) to listen to commands from the master    
//socket. This listener will only listen for commands and read these commands

/* 1) Setup a listener socket on port 63001(say) to listen to connection requests from 
   (NUMNODES-1) other processes on the rest of the nodes
   2) Accept each of the connections from the rest of the (NUMNODES-1) nodes and fill  
   these conneced descriptors into an array of integers  */

for(i=1;i<NUMNODES;i++){
   connfd=accept(...,&node_addr,...);
   index=findNodeIndex(node_addr); //To find the node number, corresp to this address
   connections[index]=connfd; 
}

/* Skipping all details aside, assuming i have a process running on node number 4 (out  
   of a total of 10 nodes, say) connections[4] will be -1, and connections[1..3, 5..10] 
   will hold connection descriptors to each of the other nodes, ready to read any file 
   that they decide to transfer */

fd_set read_set;
for(i=1;i<=NUMNODES;i++){
    if(i==thisNodeNum())   //For nodenum 4, the connections[4] will be -1
        continue;          //So we don't want to put it into the read fd_set

     FD_SET(connections[i],&read_set);
}

fd_set temp_rset;

//A select() call ready to listen to any data that comes in from any of the nodes
for(;;){
    temp_rset=read_set;
    select(maxfdp1,&temp_rset,NULL,NULL,NULL);

    //Listening for commands from master goes here
    if(FD_ISSET(commandListener,&temp_rset){
        ... listen to command and send file to a particular node...
        ... File will be sent to port 63001 on that node...
        ... commandLIstener listens for commands on port 63000(just a reminder)...
    }

    //Listening for data that has come in from the other nodes
    for(i=1;i<=NUMNODES;i++){
        if(FD_ISSET(connections[i],&temp_rset){
            ... write file to the local hard disk, byte by byte...
        }
    }//End of connections[1..NUMNODES] testing for data to read

}//End of infinite for loop

My problem is that my master sends commands to whichever node it pleases on port number 63001 and the command is received and acted upon. The file gets sent byte by byte to the appropriate node (Say, the master commands node 5 to send a file to node 9.. the process on node 5 will utilize connections[9] to send the file to the process on node 9... the process on node 9 will receive the data on connections[5]... atleast that's what I want to happen)

The file gets sent to the appropriate port on the correct node (node 9 @ port 63001), but the FD_ISSET(connections[i], &temp_rset) condition on node 9 never detects any data sent. I have checked using tshark and tcpdump, the data does get sent to node 9, but the select() call never picks up anything.

What am I doing wrong ?

Answer 1

Your accept loop accepts connections from NUMNODES-1 other nodes. So if you have NUMNODES nodes all running this code, this implies that all of them must also do a connect to every other node. Which means you'll end up with TWO connections between each pair of nodes, one initiated from each direction.

Now when you set up read_set for your select loop, it looks like you're only looking at the connections you accepted, not the ones you called connect for. If you're also sending the data on the accepted connections (rather than the connected connection), the process on the other end won't notice it, as it is waiting on its accepted connection(s) and not its connected connections.

Answer 2

Your code should look like this:

fd_set read_set; // set of file descriptors (in this case sockets)
int result;

for(;;)
{
    FD_ZERO(&read_set); // you need to clear the set first!
    FD_SET(commandListener, &read_set); // add command socket to the set 
    for(i=1; i<=NUMNODES; i++) // add node sockets to the set
    {
        if(i==thisNodeNum()) continue;
        FD_SET(connections[i], &read_set);
    }

    // check status of all sockets from the set
    result = select(maxfdp1, &read_set, NULL, NULL, NULL); // instead of tracking maxfdp1 you can use FD_SETSIZE (maximum allowed size of set)

    if(result == -1) // error
    {
        perror("select() error");
    }
    else if(result > 0) // there is new data
    {
        if(FD_ISSET(commandListener, &read_set) // there is new command data
        {
            // (...) handling your tasks here
        }

        for(i=1; i<=NUMNODES; i++) 
        {
            if(FD_ISSET(connections[i], &read_set) // there is new data on i-th node 
            {
                // (...) handling your tasks here
            }
        }
    }

}//End of infinite for loop

Notes:

1. You forgot to clear and rebuild the set of file descriptors (sockets) in your code (select modifies it - notice that you check it later with FD_ISSET).

2. You can put all sockets in one set and check them all at once with select and later identify which have new data (if any).

3. I'm not sure how you index table "connections", as loops in your code are 1..NUMNODES instead of natural C/C++ table indexing of 0..NUMNODES-1.