Can't listen on multiple sockets when using BINDTODEVICE?

Question

I have two network links to the Internet, and I have two default routes set up:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway0        0.0.0.0         UG    0      0      eth0
default         gateway1        0.0.0.0         UG    0      0      eth1
...

I created two sockets with BINDTODEVICE, so that I can send data out either eth0 or eth1. I am also trying to listen on both sockets using recvfrom (UDP data only), but I can only successfully read data from whichever interface is listed first in the routes. eth0 works, for example, but I get nothing from the socket bound to eth1.

Running wireshark on either interface shows data coming in successfully - that is, I can see data being sent from the Internet to either eth0's or eth1's IP in Wireshark (so NAT is not a problem with either), but my program just blocks on recvfrom without getting any data.

I have tried using bind on the sockets to make them listen on their respective interface's IP, and also tried not using bind to have them listen on 0.0.0.0 (each on a different port), but I still have the same problem.

How can I make sure both sockets get the data they're supposed to?

Edit: sample code:

int createDeviceBoundUDPSocket(uint32_t sip, uint16_t sport, const char* bind_dev) {
    printf("bind_dev = %s", bind_dev);
   int s = socket(AF_INET, SOCK_DGRAM, 0);   
   int result;
   struct sockaddr_in my_ip_addr;

   if (s < 0) {
      perror("socket");
      return s;
   }

   memset(&my_ip_addr, 0, sizeof(my_ip_addr));

   my_ip_addr.sin_family = AF_INET;
   my_ip_addr.sin_addr.s_addr = htonl(sip);
   my_ip_addr.sin_port = htons(sport);

   // commenting this section out doesn't seem to make a difference
   // listening on 0.0.0.0 or the interface's IP both have the same problem
   result = bind(s, (struct sockaddr*)(&my_ip_addr), sizeof(my_ip_addr));
   if (result < 0) {
      perror("Error in bind");
      return result;
   }

   if (bind_dev) {
      // Bind to specific device.
      if (setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE,
                     bind_dev, strlen(bind_dev) + 1)) {
         perror("Error binding to device");
         return -1;
      }
   }

   return s;
}

Answer 1

The solution, as hinted by the Linux Advanced Routing and Traffic Control guide, is to turn off reverse path filtering. Not sure which of the three interfaces needed to have it turned off, but doing

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

made it work.