What is the maximum length of the cable can be for infiniband(RDMA)?

Question

What is the maximum length of the cable can be for infiniband(RDMA)?

For example. Here have said, that it can be up to 10 M for a four-channel copper cable. And using the Connects Cables using fiber, which fully electrically compatible with standard InfiniBand 4x and Ethernet 10GBaseCX4, up to 100 M:

http://electronicdesign.com/communications/system-extends-infiniband-cable-reach-100-m

Answer 1

The article that you linked to is from 2007. Six years is infinity in this area. So the answer depends on many parameters.

• Which HCA vendor?
• Which cable vendor?
• Copper or optical?
• Passive or active?
• What speed are you willing to achieve in your IB setup?

For instance, you can get a 300m active cable that supports speeds up to QDR (40 Gb/s). Or you can get a 0.5m passive copper cable that supports up speeds up to FDR10.

In general, the longer the cable you need and the faster the speed you want, you'll have to jump to "fancier" technology.

You can get a passive copper cable that supports FDR10 with length up to 5m. Connect same pair of FDR10-capable HCAs with 7m cable of the same family - and HCAs will auto-negotiate max speed at QDR.

More on cables: here

Answer 2

You can refer the following table for the appropriate cable selection.

In a topology like fat-tree, if you like to connect a leaf switch with a spine switch, even the length is less than 10m, go for optical cables since the bandwidth will be high and will be fatter when traversing upwards in the topology.The attenuation will be higher in the copper cable in high frequencies compared to optical cables.

Answer 3

If latency is a concern in the system, which is frequently the case when InfiniBand is a design choice, then you should consider the effect that longer cables will have. The speed of light through fiber is around 5 nanoseconds per meter. Typical FDR latency is around 700ns, so a 100 meter cable is going to nearly double that. It may not be a huge concern as long as all the cables are the same length, but it depends on your use case.