urllib.parse.quote won't take utf8

Question

I am trying to use urllib.parse.quote as intended but cant get it to work. I even tried the example given in the documentation

Example: quote('/El Niño/') yields '/El%20Ni%C3%B1o/'.

If I try this following happens.

quote('/El Niño/')
  File "<stdin>", line 0

    ^
SyntaxError: 'utf-8' codec can't decode byte 0xf1 in position 13: invalid continuation byte

Anyone got a hint what is wrong? I am using Python 3.2.3

PS: Link to the docs http://docs.python.org/3.2/library/urllib.parse.html

Answer 1

\xf1 is a latin-1 encoded ñ

>>> print(b'\xf1'.decode("latin-1"))
ñ

..not a utf-8 encoded character, like Python 3 assumes by default:

>>> print(b'\xf1'.decode("utf-8"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 0: unexpected end of data

Meaning, there is an encoding issue either with the .py file you have written, or the terminal in which you are running the Python shell - it is supplying latin-1 encoded data to Python, not utf-8

Answer 2

try adding the following line at the begining of your source code:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os, sys
…

per default, python assumes that your source code is written in ASCII, and thus it can't read unicode strings from your source file. Read PEP-0263 on the topic.

Though, if you switch to python3, you don't need to place that coding: utf-8 comment after the shebang line, because utf-8 is the default.

Edit: Just noticed that you are actually trying to do python3, which should be utf-8-safe. Though looking at the error, it looks to me that you're actually executing python2 code whereas you think you are executing python3.

Is the shebang line correctly set?

Are you calling the script with the right interpreter?

here's the right shebang line:

#/usr/bin/env python3

or

#/usr/bin/python3

and not just /usr/bin/python or /usr/bin/env python.

can you give your full failing script, and the way you're calling it in your question?