If a server listens on a port no matter which protocol is in use, there sits some application logic on top, expecting certain data to arrive on this port.
As per your example if the server listens on two ports (A + B) it certainly expects a different kind of data on each of those two ports.
If the clinet now decides to switch ports, say send the data meant to go to port A to port B the server's application logic surely will get confused as it does not understand what it receives on port B and (if it's well designed software in the server) it will drop the connection on port B.
Such kind of scenario is completley independed of which kind of protocol is used.
It's the same as if you'd try to plug in the power cable of your pc into the network-socket but into the power-socket. Although both sockets are build into the PC you cannot simply use any socket for any kind of connection/transfer.