Sending client-side authorisation certificate to HTTPS site - Python

Question

I've had a long, long day of trying to get a Client Authorisation certificate working with either mechanize or requests and have finally resorted to IEC for automation which has been blessedly simple - up till now. I just have to click to say "yes, use this certificate" and then the automation fills in forms, finds the links and tries to download the files.

I've got to the stage of bringing up the Open Save Cancel dialog for the first file but can't see how to interact with that. Googling around I'm not sure it's possible without using mouse clicks which seems like overkill and I can't get SendKeys.py (link) to work as I'm apparently misssing vcvarsall.bat.

Everything seems to be a dead end today!

So there are really two questions here.

1) Is there a way of passing cert.pem and key.pem files to either requests or mechanize? (my preferred route)

My current mechanize code here is:

br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
br.open(root_url + '/home.html')
# Hangs on the following line due to not passing Client Certificate
r = br.open(root_url + '/secure/secureTerms.html')

Traceback when it eventually times out is:

Traceback (most recent call last):
  File "C:\Users\Jamie\Dropbox\workspace\XXXXX\XXXXX\myfile.py", line 29, in <module>
    r = br.open('https://www.example.com/secure/authorisedUsers.html')
  File "build\bdist.win32\egg\mechanize\_mechanize.py", line 203, in open
  File "build\bdist.win32\egg\mechanize\_mechanize.py", line 230, in _mech_open
  File "build\bdist.win32\egg\mechanize\_opener.py", line 193, in open
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 344, in _open
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 332, in _call_chain
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 1170, in https_open
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 1116, in do_open
  File "C:\Python27\Lib\httplib.py", line 1025, in getresponse
    response.begin()
  File "C:\Python27\Lib\httplib.py", line 401, in begin
    version, status, reason = self._read_status()
  File "C:\Python27\Lib\httplib.py", line 365, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

And my requests code is:

    agent = requests.session()
    agent.get(root_url + '/home.html')
    data = {'redirectPage':'reportSearchAddressByPostcode',
            'accept':'Accept Terms'}
    agent.post(URL, data=data, headers=hdr)
    # Again, this hangs on the following line
    response = agent.get(root_url + '/secure/secureTerms.html', headers=hdr)

And here's the Traceback I get from requests:

Traceback (most recent call last):
  File "C:\Users\Jamie\Dropbox\workspace\XXXXX\XXXXX\myfile.py", line 27, in <module>
    headers=hdr, cert=('cert.pem', 'key.pem'))
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 363, in get
    return self.request('GET', url, **kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 347, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 460, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python27\lib\site-packages\requests\adapters.py", line 327, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.example.com', port=443): Max retries exceeded with url: /secure/authorisedUsers.html (Caused by <class 'httplib.BadStatusLine'>: '')

2) Or is there a way to bypass or click through the Open Save Cancel? (acceptable kludge)

EDIT

I've tried this in cURL and it works fine with the following parameters:

curl -k -v --key key.pem --cert cert.pem https://www.example.com/secure/authorisedUsers.html

The server responds with:

* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*        subject: OU=Go to https://www.name.com/repository/index.html; OU=Name
 SSL123 certificate; OU=Domain Validated; CN=www.example.com
*        start date: 2012-12-03 00:00:00 GMT
*        expire date: 2013-12-03 23:59:59 GMT
*        issuer: C=US; O=Name, Inc.; OU=Domain Validated SSL; CN=Name DV SSL
 CA
*        SSL certificate verify result: unable to get local issuer certificate (
20), continuing anyway.
> GET /secure/authorisedUsers.html HTTP/1.1
> User-Agent: curl/7.30.0
> Host: www.example.com
> Accept: */*
>
* SSLv3, TLS handshake, Hello request (0):
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS handshake, CERT verify (15):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
< HTTP/1.1 302 Moved Temporarily
< Date: Fri, 21 Jun 2013 00:52:44 GMT
< Set-Cookie: site-text-size=medium; Domain=www.example.com; Expires=Sat,
21-Jun-2014 00:52:44 GMT; Path=/
< Set-Cookie: user=SITE_MEMBER_V2; Domain=www.example.com; Path=/
< Location: https://www.example.com/secure/addressSearch.html
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8
<
<html><body><p>Redirecting to <a href="https://www.example.com/secure/addr
essSearch.html">https://www.example.com/secure/addressSearch.html</a></p><
/body></html>* Closing connection 0
* SSLv3, TLS alert, Client hello (1):

This is all what I was expecting to see, so is there anything in here that would suggest why it's failing in Python?

Answer 1

Here is how you specify the client certificates to use for that HTTPS session:

Mechanize

br = mechanize.Browser()
br.add_client_certificate(myurl, mykey, cert)
br.set_handle_robots( False )
#httplib.HTTPSConnection.connect = connect
resp = br.open(myurl)

requests

resp = requests.get('<https_url>', cert=(mycert, mykey), verify=False)

Update: I edited the code for mechanize. We need to use add_client_certificate API with the Browser() instance to specify the client certificate.