one two-directed tcp socket OR two one-directed? (linux, high volume, low latency)

Question

I need to send (interchange) a high volume of data periodically with the lowest possible latency between 2 machines. The network is rather fast (e.g. 1Gbit or even 2G+). Os is linux. Is it be faster with using 1 tcp socket (for send and recv) or with using 2 uni-directed tcp sockets?

The test for this task is very like NetPIPE network benchmark - measure latency and bandwidth for sizes from 2^1 up to 2^13 bytes, each size sent and received 3 times at least (in teal task the number of sends is greater. both processes will be sending and receiving, like ping-pong maybe).

The benefit of 2 uni-directed connections come from linux:

http://lxr.linux.no/linux+v2.6.18/net/ipv4/tcp_input.c#L3847

3847/*
3848 *      TCP receive function for the ESTABLISHED state. 
3849 *
3850 *      It is split into a fast path and a slow path. The fast path is 
3851 *      disabled when:
...
3859 *      - Data is sent in both directions. Fast path only supports pure senders
3860 *        or pure receivers (this means either the sequence number or the ack
3861 *        value must stay constant)
...
3863 *
3864 *      When these conditions are not satisfied it drops into a standard 
3865 *      receive procedure patterned after RFC793 to handle all cases.
3866 *      The first three cases are guaranteed by proper pred_flags setting,
3867 *      the rest is checked inline. Fast processing is turned on in 
3868 *      tcp_data_queue when everything is OK.

All other conditions for disabling fast path is false. And only not-unidirected socket stops kernel from fastpath in receive

Answer 1

There are too many variables for a single answer to always hold here. Unless you have a very very fast network link - probably > 1 GBit/sec on modern hardware - the fastpath/slowpath stuff you linked to probably doesn't matter.

Just in case, you can choose to write your program to work either way. Just store a readsocket and a writesocket, and at connect() time, you can either assign them to be the same socket, or two different sockets. Then you can just try it both ways and see which is faster.

It's highly likely you won't notice any difference between the two.

Answer 2

I know this doesn't directly answer your question, but I would suggest taking a look at something like ZeroMQ. There is an introductory article about it on lwn: 0MQ: A new approach to messaging.

I haven't gotten to try it out yet, but I've read up on it a bit and it looks like it might be what you're looking for. Why reinvent the wheel?