Error in django unittest while loading a fixture

Question

I am making unittests for a django app. I need some data in the database for my tests so I am using a json fixture.

I have two fixtures:

1. for users and it works ok.
2. for some webpages

The fixture 2 cause the following error:

Problem installing fixture 'C:\Users\luc\Dev\Hg\mnl-adminpub\website\fixtures\website-unittest.json': Traceback (most recent call last):
  File "C:\Python26\lib\site-packages\django\core\management\commands\loaddata.py", line 169, in handle
    obj.save(using=using)
  File "C:\Python26\lib\site-packages\django\core\serializers\base.py", line 165, in save
    models.Model.save_base(self.object, using=using, raw=True)
  File "C:\Python26\lib\site-packages\django\db\models\base.py", line 528, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)
  File "C:\Python26\lib\site-packages\django\db\models\manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)
  File "C:\Python26\lib\site-packages\django\db\models\query.py", line 1479, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "C:\Python26\lib\site-packages\django\db\models\sql\compiler.py", line 783, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)
  File "C:\Python26\lib\site-packages\django\db\models\sql\compiler.py", line 727, in execute_sql
    cursor.execute(sql, params)
  File "C:\Python26\lib\site-packages\django\db\backends\mysql\base.py", line 86, in execute
    return self.cursor.execute(query, args)
  File "C:\Python26\lib\site-packages\MySQLdb\cursors.py", line 173, in execute
    self.errorhandler(self, exc, value)
  File "C:\Python26\lib\site-packages\MySQLdb\connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (1366, "Incorrect string value: '\\xE2\\x80\\xA8<br...' for column 'html' at row 1")

I've created the fixture with dumpdata. I've tested it with loaddata and it works fine. I am using mysql.

Any idea of the possible cause of the problem?

Thanks for your help

Answer 1

You should use TEST_CHARSET, but inside DATABASE config. Like that:

DATABASES = {
      'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'test_sbet',                      
        'USER': 'test_sbet',                      
        'TEST_CHARSET': 'UTF8',
    }
}

Answer 2

If this only happens during testing, I would suspect some of the TEST_ settings, like TEST_CHARSET. Maybe your (regular, not test) db has some non default charset? If so, you need to specify which charset the test database should be set up with.

BTW, If you are using an older (pre 1.2) version, then this should be done using TEST_DATABASE_CHARSET.

Answer 3

Had the same issue. Using utfmb4 in MySql database and adding a CHARSET key to the TEST dictionary helped (Django 1.11):

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'OPTIONS': {
            ...
            'charset': 'utf8mb4',
        },
        'TEST': {'CHARSET': 'utf8mb4',},
    }
}