libpcap not receiving in real time, seems to be buffering packets

Question

So I'm working with a device where I need to send and receive raw ethernet frames. It's a wireless radio and it uses ethernet to send status messages to its host. The protocol it uses is actually IPX, but I figured it would be easier to send raw ethernet frames using libpcap than to dig through decades old code implementing IPX (which got replaced by TCP/IP, so it's quite old).

My program sends a request packet (this packet is exactly the same every time, it's stateless) and the device returns a response packet with the data I need. I'm using pcap_inject to send the frame and pcap_loop in another thread to do the receiving. I originally had it in one thread, but tried 2 threads to see if it fixed the issue I'm having.

The issue is that libpcap doesn't seem to be receiving the packets in real time. It seems to buffer about 5 of them and then process them all at once. I want to be able to read them as fast as they come. Is there some way to disable this buffering on libpcap, or increase the refresh rate?

Some example output (I just printed out the time that a packet was received). Notice how there is about a second of time between each group

Time: 1365792602.805750 
Time: 1365792602.805791
Time: 1365792602.805806
Time: 1365792602.805816
Time: 1365792602.805825
Time: 1365792602.805834

Time: 1365792603.806886
Time: 1365792603.806925
Time: 1365792603.806936
Time: 1365792603.806944
Time: 1365792603.806952

Time: 1365792604.808007
Time: 1365792604.808044
Time: 1365792604.808055
Time: 1365792604.808063
Time: 1365792604.808071

Time: 1365792605.809158
Time: 1365792605.809194
Time: 1365792605.809204
Time: 1365792605.809214
Time: 1365792605.809223

Here's the inject code:

char errbuf[PCAP_ERRBUF_SIZE];
char *dev="en0";
if(dev==NULL){
    fprintf(stderr,"Pcap error: %s\n",errbuf);
    return 2;
}

printf("Device: %s\n",dev);

pcap_t *handle;
handle=pcap_open_live(dev, BUFSIZ, 1, 1000, errbuf);
if(handle==NULL){
    fprintf(stderr, "Device open error: %s\n",errbuf);
    return 2;
}

//Construct the packet that will get sent to the radio
struct ether_header header;
header.ether_type=htons(0x0170);
int i;
for(i=0;i<6;i++){
    header.ether_dhost[i]=radio_ether_address[i];
    header.ether_shost[i]=my_ether_address[i];
}

unsigned char frame[sizeof(struct ether_header)+sizeof(radio_request_packet)];
memcpy(frame, &header, sizeof(struct ether_header));
memcpy(frame+sizeof(struct ether_header), radio_request_packet, sizeof(radio_request_packet));
if(pcap_inject(handle, frame, sizeof(frame))==-1){
    pcap_perror(handle, errbuf);
    fprintf(stderr, "Couldn't send frame: %s\n",errbuf);
    return 2;
}

bpf_u_int32 mask;
bpf_u_int32 net;
if(pcap_lookupnet(dev,&net,&mask,errbuf)==-1){
    pcap_perror(handle, errbuf);
    fprintf(stderr,"Net mask error: %s\n",errbuf);
    return 2;
}

char *filter="ether src 00:30:30:01:b1:35";
struct bpf_program fp;
if(pcap_compile(handle, &fp, filter, 0, net)==-1){
    pcap_perror(handle, errbuf);
    fprintf(stderr,"Filter error: %s\n",errbuf);
    return 2;
}

if(pcap_setfilter(handle, &fp)==-1){
    pcap_perror(handle, errbuf);
    fprintf(stderr, "Install filter error: %s\n",errbuf);
    return 2;
}

printf("Starting capture\n");
pthread_t recvThread;
pthread_create(&recvThread, NULL, (void *(*)(void *))thread_helper, handle);

while(1){
    if(pcap_inject(handle, frame, sizeof(frame))==-1){
        pcap_perror(handle, errbuf);
        fprintf(stderr, "Couldn't inject frame: %s\n",errbuf);
        return 2;
    }
    usleep(200000);
}

pcap_close(handle); 
return 0;

And the receiving code:

void got_packet(u_char *args,const struct pcap_pkthdr * header,const u_char * packet){
    struct timeval tv;
    gettimeofday(&tv, NULL);
    double seconds=(double)tv.tv_sec + ((double)tv.tv_usec)/1000000.0;
    printf("Time: %.6f\n",seconds);
}


void *thread_helper(pcap_t *handle){
    pcap_loop(handle, -1, got_packet, NULL);
    return NULL;
}

Answer 1

Is there some way to disable this buffering on libpcap

There's currently no libpcap API to do that.

However, depending on what OS you're running, there may be ways to do it for that particular OS, i.e. you can do it, but in a non-portable fashion.

For systems that use BPF, including *BSD and...

...OS X, which, given the "en0", I suspect you're using, the way to do it is to do something such as:

Creating a set_immediate_mode.h header file containing:

extern int set_immediate_mode(int fd);

Creating a set_immediate_mode.c source file containing:

#include <sys/types.h>
#include <sys/time.h>
#include <sys/ioctl.h>
#include <net/bpf.h>

#include "set_immediate_mode.h"

int
set_immediate_mode(int fd)
{
    int on = 1;

    return ioctl(fd, BIOCIMMEDIATE, &on);
}

Adding #include <string.h> and #include <errno.h> to your program if it's not already including those files, adding #include "set_immediate_mode.h" to your program, and adding, after the pcap_open_live() call succeeds, the following code:

int fd;
fd = pcap_fileno(handle);
if (fd == -1) {
    fprintf(stderr, "Can't get file descriptor for pcap_t (this should not happen)\n");
    return 2;
}
if (set_immediate_mode(fd) == -1) {
    fprintf(stderr, "BIOCIMMEDIATE failed: %s\n", strerror(errno));
    return 2;
}

That will completely disable the buffering that BPF normally does (that's the buffering you're seeing with libpcap; see the BPF(4) man page), so that packets are delivered as soon as they arrive. That changes the way buffering is done in ways that might cause BPF's internal buffers to fill up faster than they would if the normal buffering is done, so that might cause packets to be lost when they wouldn't otherwise be lost, but using pcap_set_buffer_size(), as suggested by Kiran Bandla, could help that if it happens (which it might not, especially given that you're using a filter to keep "uninteresting" packets from being put into BPF's buffer in the first place).

On Linux, this is currently not necessary - what buffering is done doesn't have a timeout for the delivery of packets. On Solaris, it would be done similarly on Solaris 11 (as libpcap uses BPF), but would be done differently on earlier versions of Solaris (as they didn't have BPF and libpcap uses DLPI). On Windows with WinPcap, pcap_open() has a flag for that.

A future version of libpcap will probably have an API for this; I can't promise when that will happen.

Answer 2

You can set the capture buffer size by using pcap_set_buffer_size. Make sure you do this before you activate your capture handle.

Lowering the buffer size is not always a good idea. Watchout for your CPU and also dropped packets at high capture rate.