The simple answer is "you don't really care about that".
TCP works by estabilishing a two-way "virtual" persistent connection. To achieve this illusion, it uses a separate port for communication with a given client. You still connect to port 80 (for example), but a thousand clients use a thousand different ports for the actual communication with the server.
ExclusiveAddressUse
allows you to bind more sockets to the same port - but each has to have its own IP address.
Networking is hard. Us as high an abstraction as you can, be it TcpListener, HttpListener or even OWIN.