How do you use Suds to connect to a SOAP API on a private network from a public web server?

Question

I have a website that needs to connect to a SOAP API that's behind a firewall on a private network. I've tried a couple things with various levels of success.

1. SSH Tunnel

I tried setting up a SSH tunnel on the website, using the following (IPs are random examples).

export WEB_HOST=203.0.113.10
export LINUX_HOST_WITH_PUBLIC_IP_ON_PRIVATE_NETWORK=203.0.113.11
export SOAP_API_HOST=198.51.100.10

# from $LINUX_HOST_WITH_PUBLIC_IP_ON_PRIVATE_NETWORK I run the following
ssh -f $LINUX_HOST_WITH_PUBLIC_IP_ON_PRIVATE_NETWORK -L 4000:$SOAP_API_HOST:80 -N

# test with curl
$ curl -I http://localhost:4000
HTTP/1.1 200 OK
...

However when I try using Suds with the API it doesn't seem to work.

$ cat temp.py

from suds.client import Client

url = 'http://localhost:4000/scripts/WebObjects.exe/WebServices.woa/ws/Law?wsdl'
client = Client(url)
print(client.service.doThing())

$ python temp.py
Traceback (most recent call last):
  File "temp.py", line 6, in <module>
    print(client.service.doThing())
  File "/Users/foouser/.virtualenvs/project/lib/python2.7/site-packages/suds/client.py", line 542, in __call__
    return client.invoke(args, kwargs)
  File "/Users/foouser/.virtualenvs/project/lib/python2.7/site-packages/suds/client.py", line 602, in invoke
    result = self.send(soapenv)
  File "/Users/foouser/.virtualenvs/project/lib/python2.7/site-packages/suds/client.py", line 637, in send
    reply = transport.send(request)
  File "/Users/foouser/.virtualenvs/project/lib/python2.7/site-packages/suds/transport/https.py", line 64, in send
    return  HttpTransport.send(self, request)
  File "/Users/foouser/.virtualenvs/project/lib/python2.7/site-packages/suds/transport/http.py", line 77, in send
    fp = self.u2open(u2request)
  File "/Users/foouser/.virtualenvs/project/lib/python2.7/site-packages/suds/transport/http.py", line 118, in u2open
    return url.open(u2request, timeout=tm)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 61] Connection refused>

2. Wrapping API calls with a python contextmanager

I've also tried wrapping API calls with a python contextmanager based on code from Fabric.

$ cat temp2.py                                                                                                     
from contextlib import contextmanager
import socket
import paramiko
import logging


@contextmanager
def remote_tunnel(remote_port, local_port=None, local_host="localhost", remote_bind_address="127.0.0.1", transport=None):
    if local_port is None:
        local_port = remote_port

    sockets = []
    channels = []
    threads = []

    def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):
        channels.append(channel)
        sock = socket.socket()
        sockets.append(sock)

        try:
            sock.connect((local_host, local_port))
        except Exception, e:
            print "[%s] rtunnel: cannot connect to %s:%d (from local)" % (env.host_string, local_host, local_port)
            chan.close()
            return

        print "[%s] rtunnel: opened reverse tunnel: %r -> %r -> %r"\
              % (env.host_string, channel.origin_addr,
                 channel.getpeername(), (local_host, local_port))

        th = ThreadHandler('fwd', _forwarder, channel, sock)
        threads.append(th)

    transport.request_port_forward(remote_bind_address, remote_port, handler=accept)

    try:
        yield
    finally:
        for sock, chan, th in zip(sockets, channels, threads):
            sock.close()
            chan.close()
            th.thread.join()
            th.raise_if_needed()
        transport.cancel_port_forward(remote_bind_address, remote_port)


def main():

    WEB_HOST = '203.0.113.10'
    LINUX_HOST_WITH_PUBLIC_IP_ON_PRIVATE_NETWORK = '203.0.113.11'
    SOAP_API_HOST = '198.51.100.10'
    LOCAL_PORT = 4000
    REMOTE_PORT = 80
    SSH_USER = 'foouser'    

    # Connect to SSH host
    client = paramiko.SSHClient()
    client.load_system_host_keys()
    client.set_missing_host_key_policy(paramiko.WarningPolicy())
    ssh_host = (LINUX_HOST_WITH_PUBLIC_IP_ON_PRIVATE_NETWORK, 22, SSH_USER)

    logging.debug('Connecting to ssh host {}:{:d} ...'.format(ssh_host[0], ssh_host[1]))
    try:
        client.connect(ssh_host[0], ssh_host[1], username=ssh_host[2], key_filename=None, look_for_keys=True, password=None)
    except Exception as e:
        logging.error('Failed to connect to {}:{:d}: {:r}' % (ssh_host[0], ssh_host[1], e))

    with remote_tunnel(remote_port=REMOTE_PORT, local_port=LOCAL_PORT, local_host='localhost', remote_bind_address=SOAP_API_HOST, transport=client.get_transport()):
        print(requests.get('http://localhost:4000/'))

if __name__ == '__main__':
    main()


$ python temp2.py
Traceback (most recent call last):
  File "temp2.py", line 80, in <module>
    main()
  File "temp2.py", line 76, in main
    with remote_tunnel(remote_port=REMOTE_PORT, local_port=LOCAL_PORT, local_host='localhost', remote_bind_address=SOAP_API_HOST, transport=client.get_transport()):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "temp2.py", line 35, in remote_tunnel
    transport.request_port_forward(remote_bind_address, remote_port, handler=accept)
  File "/Users/foouser/.virtualenvs/project/lib/python2.7/site-packages/paramiko/transport.py", line 810, in request_port_forward
    raise SSHException('TCP forwarding request denied')
paramiko.SSHException: TCP forwarding request denied

Update

Based on @scott-talbert's answer, I was able to get the first approach to work by using the following after setting up the SSH tunnel.

from suds.client import Client
import os

url = 'http://{}/scripts/WebObjects.exe/WebServices.woa/ws/Law?wsdl'.format(os.getenv('SOAP_API_HOST'))
client = Client(url)
client.set_options(proxy={'http': '127.0.0.1:4000'})
print(client.service.doThing())

It would still be nice to figure out how to get my second approach to work so you don't have to setup and manage the SSH tunnel.

Answer 1

I suspect the reason #1 is failing is that the WSDL contains URLs that your client cannot reach directly and that is why you're getting the "connection refused" message. It looks like Suds can be configured tell its urllib2 instance to use a proxy server - see https://fedorahosted.org/suds/wiki/Documentation. That might work for your situation. You might have to run ssh in "SOCKS" mode by using the -D option.