Python interface to PayPal - urllib.urlencode non-ASCII characters failing

Question

I am trying to implement PayPal IPN functionality. The basic protocol is as such:

1. The client is redirected from my site to PayPal's site to complete payment. He logs into his account, authorizes payment.
2. PayPal calls a page on my server passing in details as POST. Details include a person's name, address, and payment info etc.
3. I need to call a URL on PayPal's site internally from my processing page passing back all the params that were passed in abovem and an additional one called 'cmd' with a value of '_notify-validate'.

When I try to urllib.urlencode the params which PayPal has sent to me, I get a:

While calling send_response_to_paypal. Traceback (most recent call last):
  File "<snip>/account/paypal/views.py", line 108, in process_paypal_ipn
    verify_result = send_response_to_paypal(params)
  File "<snip>/account/paypal/views.py", line 41, in send_response_to_paypal
    params = urllib.urlencode(params)
  File "/usr/local/lib/python2.6/urllib.py", line 1261, in urlencode
    v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 9: ordinal not in range(128)

I understand that urlencode does ASCII encoding, and in certain cases, a user's contact info can contain non-ASCII characters. This is understandable. My question is, how do I encode non-ASCII characters for POSTing to a URL using urllib2.urlopen(req) (or other method)

Details:

I read the params in PayPal's original request as follows (the GET is for testing):

def read_ipn_params(request):
    if request.POST:  
        params= request.POST.copy()  
        if "ipn_auth" in request.GET:
            params["ipn_auth"]=request.GET["ipn_auth"]
        return params
    else:  
        return request.GET.copy()  

The code I use for sending back the request to PayPal from the processing page is:

def send_response_to_paypal(params):
    params['cmd']='_notify-validate'  
    params = urllib.urlencode(params)  
    req = urllib2.Request(PAYPAL_API_WEBSITE, params)  
    req.add_header("Content-type", "application/x-www-form-urlencoded") 
    response = urllib2.urlopen(req)  
    status = response.read()  
    if not status == "VERIFIED":  
        logging.warn("PayPal cannot verify IPN responses: " + status)
        return False

    return True

Obviously, the problem only arises if someone's name or address or other field used for the PayPal payment does not fall into the ASCII range.

Answer 1

Try converting the params dictionary to utf-8 first... urlencode seems to like that better than unicode:

params = urllib.urlencode(dict([k, v.encode('utf-8')] for k, v in params.items()))

Of course, this assumes your input is unicode. If your input is something other than unicode, you'll want to decode it to unicode first, then encode it:

params['foo'] = my_raw_input.decode('iso-8859-1')
params = urllib.urlencode(dict([k, v.encode('utf-8')] for k, v in params.items()))

Answer 2

Instead of encoding to utf-8, one should encode to what ever the paypal is using for the post. It is available under key 'charset' in the form paypal sends.

So the following code worked for me:

data = dict([(k, v.encode(data['charset'])) for k, v in data.items()])

Answer 3

I know it's a bit late to chime in here, but the best solution I found was to not even parse what they were giving back. In django (don't know what you're using) I was able to get the raw request they sent, which I passed back verbatim. Then it was just a matter of putting the cmd key onto that.

This way it never matters what encoding they send you, you're just sending it right back.