Searching for C/C++ network library with multiplexing

Question

I'm writing an application that has a number (hundreds) of concurrent network operations running in parallel between two instances. Since the average lifetime of a connection is quite short (at most seconds), I thought that the overhead of using many TCP connections and doing a handshake every time (especially for a TLS handshake) would be too large.

I started to look at a couple of protocols and libraries implementing multiplexing (mostly AMQP implementations like Apache Qupid, RabbitMQ, as mentioned in answers to this question). However all of them seem to run over TCP, which introduces some overhead and doesn't make a lot of sense (this post explains the problem quite well and comes to the conclusion that TCP multiplexing is stupid). Also all of them feel quite fat, I'd prefer something small and light (ZeroMQ unfortunately doesn't implement multiplexing afaik). That got me thinking whether using UDP would be an option. Of course one has to implement stuff like recovery and ACKs properly, but with the knowledge about the multiple streams over the connection that should be much more efficient than simply using TCP.

Do you think that my reasoning above is correct, or did I miss something important? Are there any good C/C++ libraries out there that implement multiplexing via UDP?

Answer 1

Do the simplest thing that could possibly work, and make it more complex only as necessary:

1. use a single TCP connection and multiplex logical sessions over it

   • if these logical entities are just asynchronous request/response pairs, for example, you may be able to dispense with explicit logical sessions entirely

2. if you have multiple concurrent components in each instance which really need their own queues with pushback to throttle over-eager senders:

   • first consider just capping the number of outstanding requests/active sessions on the sending side, instead of requiring a specific ack
   • only if you need to dynamically vary queue length (eg. because you're really trying to limit working memory, which varies by session) use an explicit logical ack for this

3. only if you hit some case where your logical sessions really interact badly with TCP, then consider implementing your own reliable-flow-controlled-datagram protocol