setup.py packages and unicode_literals

Question

I've created a package in Py2.7 and I'm trying to make it compatible with Py3. The problem is that if I include unicode_literals in

__init__.py

imports the build returns this error

error in daysgrounded setup command: package_data must be a dictionary mapping
package names to lists of wildcard patterns

I've read the PEP, but I can't understand what it has to do with a dict like

__pkgdata__

can anyone help?

__init__.py
#!/usr/bin/env python
# -*- coding: latin-1 -*-

"""Manage child(s) grounded days."""

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)
# ToDo: correct why the above unicode_literals import prevents setup.py from working

import sys
from os import path
sys.path.insert(1, path.dirname(__file__))

__all__ = ['__title__', '__version__',
           '__desc__', '__license__', '__url__',
           '__author__', '__email__',
           '__copyright__',
           '__keywords__', '__classifiers__',
           #'__packages__',
           '__entrypoints__', '__pkgdata__']

__title__ = 'daysgrounded'
__version__ = '0.0.9'

__desc__ = __doc__.strip()
__license__ = 'GNU General Public License v2 or later (GPLv2+)'
__url__ = 'https://github.com/jcrmatos/DaysGrounded'

__author__ = 'Joao Matos'
__email__ = 'jcrmatos@gmail.com'

__copyright__ = 'Copyright 2014 Joao Matos'

__keywords__ = 'days grounded'
__classifiers__ = [# Use below to prevent any unwanted publishing
                   #'Private :: Do Not Upload'
                   'Development Status :: 4 - Beta',
                   'Environment :: Console',
                   'Environment :: Win32 (MS Windows)',
                   'Intended Audience :: End Users/Desktop',
                   'Intended Audience :: Developers',
                   'Natural Language :: English',
                   'Natural Language :: Portuguese',
                   'License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)',
                   'Operating System :: OS Independent',
                   'Programming Language :: Python',
                   'Programming Language :: Python :: 2.7',
                   'Programming Language :: Python :: 3.4',
                   'Topic :: Other/Nonlisted Topic']

#__packages__ = ['daysgrounded']

__entrypoints__ = {
    'console_scripts': ['daysgrounded = daysgrounded.__main__:main'],
    #'gui_scripts': ['app_gui = daysgrounded.daysgrounded:start']
    }

__pkgdata__ = {'daysgrounded': ['*.txt']}
#__pkgdata__= {'': ['*.txt'], 'daysgrounded': ['*.txt']}


setup.py
#!/usr/bin/env python
# -*- coding: latin-1 -*-

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

from setuptools import setup, find_packages
#import py2exe

#from daysgrounded import *
from daysgrounded import (__title__, __version__,
                          __desc__, __license__, __url__,
                          __author__, __email__,
                          __keywords__, __classifiers__,
                          #__packages__,
                          __entrypoints__, __pkgdata__)

setup(
    name=__title__,
    version=__version__,

    description=__desc__,
    long_description=open('README.txt').read(),
    #long_description=(read('README.txt') + '\n\n' +
    #                  read('CHANGES.txt') + '\n\n' +
    #                  read('AUTHORS.txt')),
    license=__license__,
    url=__url__,

    author=__author__,
    author_email=__email__,

    keywords=__keywords__,
    classifiers=__classifiers__,

    packages=find_packages(exclude=['tests*']),
    #packages=__packages__,

    entry_points=__entrypoints__,
    install_requires=open('requirements.txt').read(),
    #install_requires=open('requirements.txt').read().splitlines(),

    include_package_data=True,
    package_data=__pkgdata__,

    #console=['daysgrounded\\__main__.py']
)

Thanks,

JM

Answer 1

using unicode_literals is the same as using u'...' for each string literal in your input file, which means that in your __init__.py specifying

__pkgdata__ = {'daysgrounded': ['*.txt']}

is actually the same as

__pkgdata__ = {u'daysgrounded': [u'*.txt']}

for python2, setuptools doesn't expect unicode here but str, so it fails.

As it seems you don't use any unicode characters in your string literals in __init__.py anyway, just plain ascii, so you can simply remove the unicode_literals import. If you really use unicode literals at some place in the file that isn't shown in your post, use explicit unicode literals there.

Answer 2

This is a bug in setuptools. It is validating values with isinstance(k, str) which fails when strings are transformed into the 2.x unicode class by the unicode_literals import. It should be patched to use isinstance(k, basestring).

The easiest solution is to put the configuration settings directly into setup.py rather than storing them in __init__.py. If you need programmatic access to __version__ then put it in a separate package that is included by both setup.py and __init__.py.

From setuptools dist.py:

def check_package_data(dist, attr, value):
    """Verify that value is a dictionary of package names to glob lists"""
    if isinstance(value,dict):
        for k,v in value.items():
            if not isinstance(k,str): break
            try: iter(v)
            except TypeError:
                break
        else:
        return
    raise DistutilsSetupError(
        attr+" must be a dictionary mapping package names to lists of "
        "wildcard patterns"
   )

Answer 3

The usage of unicode_literals is to bring Python 2 compatibility to Python 3 code where str is now unicode-strings vs byte-strings in Python 2. It is great at prevent the mixing of byte-strings and unicode-strings, long time problem on Py2, however there are a few pitfalls like this issue.

Kevin has explained the bug and I'd would say for setup.py it is not strictly required and to fix it is a bit ugly especially if you have a large number of entries for package_data.

---

If you want to keep unicode_literals in setup.py you will only need to encode the dict key as a byte-string:

__pkgdata__ = {b'daysgrounded': ['*.txt']}

However under Python 3 it will fail with same message so need to cover both versions:

if sys.version_info.major == 2:
    __pkgdata__ = {b'daysgrounded': ['*.txt']}
else:
    __pkgdata__ = {'daysgrounded': ['*.txt']}

---

Alternatively use bytes_to_native_str from future module:

from future.utils import bytes_to_native_str

__pkgdata__ = {bytes_to_native_str(b'daysgrounded'): ['*.txt']}