configuring repoze.bfg to use utf-8 encoding

Question

I'm using repoze.bfg v1.3 and chameleon v2 (zpt templates). I got troubles with encoding while rendering template:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 9: ordinal not in range(128)

How can i configure repoze.bfg to use utf-8 encoding with chameleon?
I added the following to Configurator:

 config.add_settings(encoding="UTF-8")
 config.add_settings(default_encoding="UTF-8")

And hasn't helped.

Answer 1

The problem is in translationstring library which is used by chameleon. While chameleon can be configured to use different encoding, it pass the data which it get directly to translationstring.
Translationstring in the constructor, where it tries to make unicode from the data it get. The error comes when the data is a non-ascii byte sequence (str in python 2.x).

The solution is to always pass unicode to translation string or update library itself using following diff:

65c69
<        self = unicode.__new__(self, msgid)
--- patch
>       try:
>           self = unicode.__new__(self, msgid, "utf8")  # FIXED~
>       except Exception as e:
>           self = unicode.__new__(self,msgid)

Answer 2

0xc5 is Å using latin-1, python cannot convert it in utf-8 without knowing the source encoding

if this string comes from a cgi form, make sure the server sets the correct encoding

hint:

lynx -dump -mime_header http://url_of_the_page_with_the_form_to_compile|less

and look for something like

Content-Type: text/html; charset=UTF-8

if the charset is not utf-8 your configuration is wrong, maybe apache overrides your setting?