Python-3 Request parameter encoding error

Question

I have been working on a django project based upon Python 3. I am trying to incorporate captcha. I choosed django-recaptcha but unfortunately the package is not available for python3. So tried to tailor it for python3. I did some 2to3 stuff and made some changes as per need. Everything seems to work fine except the url encoding for Request.

Following snippet yields POST data should be bytes or an iterable of bytes. It cannot be of type str. exception.

def encode_if_necessary(s):
    if isinstance(s, str):
        return s.encode('utf-8')
    return s

params = urllib.parse.urlencode({
        'privatekey': encode_if_necessary(private_key),
        'remoteip':  encode_if_necessary(remoteip),
        'challenge':  encode_if_necessary(recaptcha_challenge_field),
        'response':  encode_if_necessary(recaptcha_response_field),
        })

if use_ssl:
    verify_url = "https://%s/recaptcha/api/verify" % VERIFY_SERVER
else:
    verify_url = "http://%s/recaptcha/api/verify" % VERIFY_SERVER

request = urllib.request.Request(
    url= verify_url,
    data=params,
    headers={
        "Content-type": "application/x-www-form-urlencoded",
        "User-agent": "reCAPTCHA Python"
        }
    )

httpresp = urllib.request.urlopen(request)

So I tried to encode the url and other stuff in request-

request = urllib.request.Request(
    url= encode_if_necessary(verify_url),
    data=params,
    headers={
        "Content-type": encode_if_necessary("application/x-www-form-urlencoded"),
        "User-agent": encode_if_necessary("reCAPTCHA Python")
        }
    )

But this yields urlopen error unknown url type: b'http exception.

Does anyone know how to fix it? Any help is appreciated :).

Answer 1

Alright I will answer this myself :P.

Taking hints from an example at python's official documentation, I excluded data from Request and separately passed request and data to urlopen(). Below is the updated snippet -

params = urllib.parse.urlencode({
        'privatekey': encode_if_necessary(private_key),
        'remoteip':  encode_if_necessary(remoteip),
        'challenge':  encode_if_necessary(recaptcha_challenge_field),
        'response':  encode_if_necessary(recaptcha_response_field),
        })

if use_ssl:
    verify_url = "https://%s/recaptcha/api/verify" % VERIFY_SERVER
else:
    verify_url = "http://%s/recaptcha/api/verify" % VERIFY_SERVER
# do not add data to Request instead pass it separately to urlopen()
data = params.encode('utf-8')
request = urllib.request.Request(verify_url)
request.add_header("Content-type","application/x-www-form-urlencoded")
request.add_header("User-agent", "reCAPTCHA Python")

httpresp = urllib.request.urlopen(request, data)

Despite of solving the problem I still do not know why the code generated by 2to3.py did not work. According to the documentation it should have worked.

Answer 2

You guessed correctly, you need to encode the data, but not the way you did it.

As @Sheena wrote in this SO answer, you need 2 steps to encode you data:

data = urllib.parse.urlencode(values)
binary_data = data.encode('utf-8')
req = urllib.request.Request(url, binary_data)

And do not encore the url.