linux kernel->userspace multicast streaming datagrams

Question

I'm planning to write a linux driver for some memory-mapped hardware (it's in an FPGA, so I can adjust this memory-mapped interface at both ends if needed).

This FPGA logic generates a sequence of datagrams, which I must process and then transmit over an Ethernet link. There's no reason for either the processing or the networking code to be in the kernel, so I'm asking about the "best" mechanism for moving blocks of data from hardware to user-space. The biggest complication is that the user-space processing should be distributed among multiple processes.

The data rate is not very high (up to 1Mbps) and the mmio interface is fed by a rather deep FIFO (2KB currently, could be increased as high as 8KB) so I think a high priority user-mode process could keep up.

What I'd really like is a pointer to an existing driver with an existing multicast userspace interface (and not complicated with much else). But an outline of what must be done would be a reasonable substitute.

I've collected the following ideas so far:

• AF_NETLINK: supports multicast, takes care of buffering for me. But the API is unstable, I have to define a new socket id which may conflict with other in-tree drivers, and the user-mode interface is quite specialized, I can't use standard tools like socat to test the data stream.

• Pass a datagram-mode socket or FIFO file descriptor in from userspace, and write to it (how?). There's a unix-domain datagram socket multicast patch that I could apply.

• Expose a character-mode device to a single high-priority user-mode application, which acts as a unix-domain datagram socket server and copies the datagrams to each connected node. Are datagram boundaries preserved for character-mode devices (i.e. if my driver read function returns fewer bytes than the fread buffer size, will fread return that block of data as a unit, or can it fragment and reassemble blocks? Does it help if I use read (2) instead of fread (3)? Is there anything like EMSGSIZE that a driver's read function can use to indicate that a datagram was truncated, or is that available only for sockets?)

• Expose a character-mode device which can be opened by multiple user-mode applications simultaneously, and buffer data to each in-kernel.

I'm leaning toward the character-mode device with a unix domain server which reroutes incoming packets. This saves me from having to implement buffering logic inside the kernel driver. The question then becomes how to wake the user process from a select call or blocking read when an interrupt occurs. It seems that my poll function can read the control register and return POLLIN|POLLRDNORM if data is already available, and unmask the interrupt if not. And then the interrupt handler would use wake_up to flag the wait_queue as ready. read would always mask the interrupt.

Answer 1

I think char driver is a better option, because you will find a lot more documentation for it, and the kernel part is simpler. The API is well known, and you will find more people with experience on char driver.

Start small, ie with a working interupt based blocking read :

• if no data is available, sleep on queue
• else return available data to userspace.

with an interrupt routine eventually waking the queue if data is available.

Once it works, implement poll.

Answer 2

I think netlink is your best option. BTW, you can treat a netlink socket as a regular socket and use POLL, EPOLL, select on it.

Also, what do you mean by 'the API is unstable'? Netlink is used a lot, and it has a pretty decent API.

You just need to use generic netlink to send (and possibly receive) messages. Your task becomes even easier if there is one-way communication, i.e., from kernel -> userspace.

EDIT: If you have access, refer to pg. 810 onwards (Chap. 12) of the book Professional Linux Kernel Architecture by Wrox. As far as I know, it has a (relatively) good description of how you could go about using netlink for communication with userland.

The only deficiency of netlink is the scant documentation. Otherwise, it is ok, in my opinion.

Answer 3

With the character device option, the read(2) from userspace will end up calling into your driver's read() function (specified in the struct file_operations you registered your device with). So it's up to your implementation whether you maintain datagram boundaries, and what errors you want to return in various failure cases.