How to authenticate with requests module using a trust store?

Question

I am currently writing a python language plugin for a compiler I have written that automates http calls for a RESTful API. I have managed to get the login/authentication working using the socket and ssl modules, but this low-level approach seems to create potential problems with parsing the response in order to obtain the authentication token and secret. The requests module seems popular/efficient, however, I cannot seem to get it to function properly for my particular authentication needs. I am using a trust store in the form of a .pem file (containing just a public key) that I converted from my .jks file used to authenticate for the Java plugin. The server expects the username and password to be submitted in the request body in json format. Here is the code I have been trying to use:

#Server and login data
...
host = 'localhost'
port = 8443
pem_file = "C:\\Users\\aharasta\\pycert.pem"

#Digest password with MD5 algorithm
m = hashlib.md5()
m.update(password)
encrypted_password = m.hexdigest()

url = <url>
data = {'userid': user_name, 'password': encrypted_password}
json_data = json.dumps(data)
headers = {'Content-type': 'application/json', 'Accept': 'text/plain', 'Content \                     
          Length': len(json_data)} 

r = requests.post(url, headers = headers, data = json_data, cert = pem_file)
print(r)

Upon execution, this code will raise an SSL error stating "certificate verify failed". If I add the parameter verify = False or verify = pem_file, I will receive a 404 response from the server. I should also note that when I launch the server in debug mode and execute the request (with one of the verify parameters), it never makes it to the server's authentication methods, or any methods for that matter. Any insight or help on this matter would be greatly appreciated!

Answer 1

First there are a couple problems with what you posted:

• You specify host and port but don't give an example url, so we can guess that you're using a local deployment for testing and can actually view the requests to the server. I'm not sure what methods you're talking about, but if you're using something akin to Flask for server development, you might not want to send the Authentication as JSON encoded data. There are Authentication headers for a reason, and requests has Authentication handlers for a reason. ;-)

• You shouldn't specify the Content-Length header yourself. requests will do this for you. Beyond that, you're specifying it incorrectly (according to what you posted), so the 404 may be from receiving a header your server doesn't recognize.

Now, there should be no reason to specify verify=False, and you can specify either cert=pem_file or verify=pem_file. Either or both should be fine but you should never use verify=False.

Finally, the SSLError that is raised is telling you that the pem file you're providing is not matching what the server is specifying it. With that in mind, you might want to check your local server's settings. Requests doesn't handle certificate verification itself, but urllib3 provides that. We just set it up based on the parameters you provide. And I doubt this is the fault of urllib3 since it raises an SSLError which arises from the standard library's ssl module.

Edit

The explanation is in the documentation specifying a *.pem file with cert is invalid. You have to use verify='/path/to/file.pem' to do this correctly.

Edit #2

To inspect a request that was already sent you can do this:

r = requests.post(...)
r.request
# PreparedRequest('POST', url, ...)
r.request.body
r.request.headers
# etc.

To modify a request before sending you can do the following:

from requests import Request, Session

s = Session()
r = Request('POST', url, datajson_data, headers=headers)
p = r.prepare()
p.body = 'New body'
p.headers = #etc.
s.send(p, verify=pem_file)