You receive the response from the intermediate server.
It sounds like your intermediate server acts as a (reverse) proxy server. When you make a request to your intermediate server, it in turn makes a request to the server on your network (just as you described it). However, the "real" server has usually no idea that the request was initiated by you. In particular, it does not know your IP address or the port you sent your request from. The only thing it does know is the IP address and port of the intermediate (the proxy) server. The proxy server on the other hand still knows the IP address and port from which you sent your request.
So you send your request to the proxy, the proxy sends it to the "real" server, the real server sends its response back to the proxy, which sends it back to you. You and the "real" server do not communicate directly with each other.