Unicode, decoding and encoding problems caused by Microsoft Translator API

Question

Error message occurs such as:

Internal Server Error: /Translator/
Traceback (most recent call last):
  File "D:\Python27\lib\site-packages\django\core\handlers\base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "D:\Project\Reservation\Translator\views.py", line 72, in getParams
    content = request.POST['content'].decode('utf-8').encode('utf-8')
  File "D:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)
[05/Oct/2013 23:32:51] "POST /Translator/ HTTP/1.1" 500 65244
Internal Server Error: /Translator/
Traceback (most recent call last):
  File "D:\Python27\lib\site-packages\django\core\handlers\base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "D:\Project\Reservation\Translator\views.py", line 72, in getParams
    content = request.POST['content'].decode('utf-8').encode('utf-8')
  File "D:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)

My code follows bellow:

def get_access_token():
    post_data = urllib.urlencode({'client_id':client_id,'client_secret':client_secret, 'scope':ACCESS_TOKEN_SCOPE, 'grant_type':ACCESS_TOKEN_GRANT_TYPE })
    token_data = json.loads(requests.post(ACCESS_TOKEN_URL,data=post_data).content)
    access_token = token_data["access_token"]

    return access_token

def detect(access_token,detect_text):

    headers = {'Authorization': 'bearer'+ ' ' + access_token}

    detect_url_all = DETECT_URL + "?" + urllib.urlencode({'text':detect_text})
    detect_language = requests.get(detect_url_all,headers=headers).content[3:]

    return detect_language


def Translator(text,orignal,access_token):

    headers = {'Authorization': 'bearer'+ ' ' + access_token}

    translation_ars = {
            'text': text,
            'to': 'zh',
            'from': orignal
        }

    transate_url_all = TRANSLATE_URL + "?" + urllib.urlencode(translation_ars)
    result = requests.get(transate_url_all,headers=headers).content
    return result

def getParams(request):
    if request.method == 'POST':
        form = Junk(request.POST)
        if form.is_valid():
            content = request.POST['content'].decode('utf-8').encode('utf-8')
            country = detect(get_access_token(),content)
            result = Translator(content,country,get_access_token())
            return render_to_response('Translator/translate.html',{'result':result})
    else:
        form = Junk()
    return render_to_response('Translator/index.html',{'form': form})

First, I want to detect the language of a text first. My program can't get to know what the text's coding is, so I can't make a decoding or encoding.

Answer 1

I think the problem is very simple, you're using the wrong method. Use result.encode (instead of result.decode) method to encode it to utf-8 encoding.

Sorry for my late answer.

Answer 2

The error message indicates request.POST['content'] was already Unicode. .decode('utf-8') expects the object to be a byte string, so an implicit .encode('ascii') is used to turn Unicode string into a byte string. since it is already a Unicode string and you seem to want a UTF-8 byte string, all you need is:

content = request.POST['content'].encode('utf-8')

request must have been passed to getParams as a Unicode string, but that part of your code isn't present.

If you have further problems, this example might help: https://gist.github.com/dpapathanasiou/2790853