It is not as easy as that to set up the connection. There is more information that needs to be provided than ip and port. There are 'offer', 'answer' and 'candidates' that are sent between the peers.
These are readable text and the could very well be typed in by the user but since they are quite large, it is more easy to provide them via the server in the middle.
You could mail them and type them in at the remote peer...