RAW socket send: packet loss

Question

During the RAW-socket based packet send testing, I found very irritating symptoms. With a default RAW socket setting (especially for SO_SNDBUF size), the raw socket send 100,000 packets without problem but it took about 8 seconds to send all the packets, and the packets are correctly received by the receiver process. It means about 10,000 pps (packets per second) is achieved by the default setting. (I think it's too small figure contrary to my expectation.)

Anyway, to increase the pps value, I increased the packet send buffer size by adjusting the /proc/sys/net/core/{wmem_max, wmem_default}. After increasing the two system parameters, I have identified the irritating symptom. The 100,000 packets are sent promptly, but only the 3,000 packets are received by the receiver process (located at a remote node).

At the sending Linux box (Centos 5.2), I did netstat -a -s and ifconfig. Netstat showed that 100,000 requests sent out, but the ifconfig shows that only 3,000 packets are TXed.

I want to know the reason why this happens, and I also want to know how can I solve this problem (of course I don't know whether it is really a problem).

Could anybody give me some advice, examples, or references to this problem?

Best regards, bjlee

Answer 1

You didn't say what size your packets were or any characteristics of your network, NIC, hardware, or anything about the remote machine receiving the data.

I suspect that instead of playing with /proc/sys stuff, you should be using ethtool to adjust the number of ring buffers, but not necessarily the size of those buffers.

Also, this page is a good resource.

Answer 2

I have just been working with essentially the same problem. I accidentally stumbled across an entirely counter-intuitive answer that still doesn't make sense to me, but it seems to work.

I was trying larger and larger SO_SNDBUF buffer sizes, and losing packets like mad. By accidentally overrunning my system defined maximum, it set the SO_SNDBUF size to a very small number instead, but oddly enough, I no longer had the packet loss issue. So I intentionally set SO_SNDBUF to 1, which again resulted in a very small number (not sure, but I think it actually set it to something like 1k), and amazingly enough, still no packet loss.

If anyone can explain this, I would be most interested in hearing it. In case it matters, my version of Linux is RHEL 5.11 (yes, I know, I'm a bit behind the times).