django-pyodbc: MSSQL unicode issue

Question

I'm having problems logging in users on Django using django-pyodbc, freetds and unixodbc. I can run syncdb and connect with pyodbc without any problems.

The full error that I get is:

 ('42000', '[42000] [FreeTDS][SQL Server]Unicode data in a Unicode-only collation or ntext data cannot be sent to clients using DB-Library (such as ISQL) or ODBC version 3.7 or earlier. (4004) (SQLExecDirectW)

freetds conf file:

[PS123]
host = <ip number>
port = 2254
tds version = 8.0
instance = T_INS01
client charset = UTF-8

# somebody suggested this had to be global so I put it here as well
[global]
client charset = UTF-8
tds version = 8.0

odbcinst.ini

[ODBC]
Trace = Yes
TraceFile = /tmp/odbc.log

[FreeTDS]
Description = TDS driver (Sybase/MS SQL)
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
Setup =  /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
UsageCount = 1
tds_version = 8.0 # several people suggested using underscore

odbc.ini

[PS123]
Driver = FreeTDS
Description = A wonderful description goes here
tds_version = 8.0 # also tried without underscore here
Database = database
Servername = PS123
Trace = Yes
TraceFile = /tmp/freetdssql.log

If I log in the user through django-debug-toolbar's debugsqlshell, I can see these are the queries that happens before the exception:

In [10]: login(request, user)
SELECT TOP 1 (1) AS [a] FROM [django_session] WHERE [django_session].[session_key] = magyul563p13z6e33t6rexesaxx1kszx [5.03ms]
INSERT INTO [django_session] ([session_key], [session_data], [expire_date])
VALUES (magyul563p13z6e33t6rexesaxx1kszx,
        ZjgwN2E1NGZhNTE4YTI2ZWQxMDM3M2ZlZThiNWVlY2NlYTlmZWQ1YzqAAn1xAS4=, 2014-02-14 10:20:31+00:00) [3.23ms]
SELECT [django_session].[session_key], [django_session].[session_data], [django_session].[expire_date] FROM [django_session] WHERE [django_session].[session_key] = magyul563p13z6e33t6rexesaxx1kszx [1.56ms]

But when I run those queries using only pyodbc like below:

import pyodbc
cnx = pyodbc.connect("DSN=PS123;UID=username;PWD=password")

cursor = cnx.cursor()
cursor.execute("SELECT TOP 1 (1) AS [a] FROM [django_session] WHERE [django_session].[session_key] = 'azozj51b9a5y9lnbq2b4hydhrryrplpz'")
cursor.execute("INSERT INTO [django_session] ([session_key], [session_data], [expire_date]) VALUES ('azozj51b9a5y9lnbq2b4hyd', 'ZwN2E1NGZhNTE4YTI2ZWQxMDM3M2ZlZThiNWVlY2NlYTlmZWQ1YzqAAn1x', '2014-02-15')")
cursor.execute("SELECT [django_session].[session_key], [django_session].[session_data], [django_session].[expire_date] FROM [django_session] WHERE [django_session].[session_key] = 'azozj51b9a5y9lnbq2b4hyd'")

It fetches and inserts without any problem. The only issue I had was that I had to remove the time and timezone in the date string. I am assuming this is unrelated to my problem.

These are my Django database settings:

DATABASES = {
    'default': {
        'ENGINE': 'sql_server.pyodbc', # also tried django_pyodbc
        'NAME': 'database',          
        'USER': 'username',
        'PASSWORD': 'password', 
        'HOST': 'PS123',
        'PORT': '2254', 
        'OPTIONS': {
           'host_is_server': True
        },
    }
}

Full traceback:

Traceback:
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/contrib/admin/sites.py" in wrapper
  219.                 return self.admin_view(view, cacheable)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/utils/decorators.py" in _wrapped_view
  91.                     response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/cache.py" in _wrapped_view_func
  89.         response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/contrib/admin/sites.py" in inner
  196.             if not self.has_permission(request):
File "/usr/local/lib/python2.7/dist-packages/django/contrib/admin/sites.py" in has_permission
  149.         return request.user.is_active and request.user.is_staff
File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py" in inner
  204.             self._setup()
File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py" in _setup
  270.         self._wrapped = self._setupfunc()
File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/middleware.py" in <lambda>
  18.         request.user = SimpleLazyObject(lambda: get_user(request))
File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/middleware.py" in get_user
  10.         request._cached_user = auth.get_user(request)
File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/__init__.py" in get_user
  136.         user_id = request.session[SESSION_KEY]
File "/usr/local/lib/python2.7/dist-packages/django/contrib/sessions/backends/base.py" in __getitem__
  44.         return self._session[key]
File "/usr/local/lib/python2.7/dist-packages/django/contrib/sessions/backends/base.py" in _get_session
  167.                 self._session_cache = self.load()
File "/usr/local/lib/python2.7/dist-packages/django/contrib/sessions/backends/db.py" in load
  18.                 expire_date__gt=timezone.now()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py" in get
  143.         return self.get_query_set().get(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py" in get
  398.         num = len(clone)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py" in __len__
  106.                 self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py" in iterator
  317.         for row in compiler.results_iter():
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py" in results_iter
  775.         for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py" in execute_sql
  846.         cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py" in execute
  41.             return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/sql_server/pyodbc/base.py" in execute
  396.             raise utils.DatabaseError(*e.args)

Exception Type: DatabaseError at /admin/
Exception Value: ('42000', '[42000] [FreeTDS][SQL Server]Unicode data in a Unicode-only collation or ntext data cannot be sent to clients using DB-Library (such as ISQL) or ODBC version 3.7 or earlier. (4004) (SQLExecDirectW)')

Answer 1

I would also recommend including the following in options, to ensure Unicode results:

'OPTIONS': {
    'host_is_server': True,
    'autocommit': True,
    'unicode_results': True,
    'extra_params': 'tds_version=8.0'
},

The autocommit option is also necessary for Django 1.6. When you connect in this manner, it bypasses any DSNs you may have set up.

Edit: Django’s documentation for v1.6 reports “Since Django 1.6, autocommit is turned on by default”, so I guess you don't need to set this

Answer 2

Turns out I had to specify the TDS version in the Django database settings as well:

'OPTIONS' : { 'host_is_server' : True, 'extra_params' : 'TDS_VERSION=8.0', }

I hope that helps someone.