This isn't really an Indy question, more a general networking one.
Different clients at the same address must use distinct ports. And here I mean the port at the client end of the connection. So you distinguish between different clients at the same address by dint of them using different ports.