Is there an easier way to package with Python?

Question

I tried to package a django app today. It's a big baby, and with the setup file, I have to manually write all packages and sub packages in the 'package' parameter. Then I have to find a way to copy fixtures, htmls / Css / image files, documentations, etc.

It's a terrible way to work. We are computer scientists, we automatize, doing this makes no sense.

And what when I change my app structure ? I have to rewrite the setup.py.

Is there a better way ? Some tool to automate that ? I can't believe a language than value developer time like Python makes packaging such a chore.

I want to be able to eventually install the app using a simple pip install. I know about build out, but it's not much simpler, and is not pip friendly.

Answer 1

At the very least if you use setuptools (an alternative to the stdlib's distutils) you get an awesome function called find_packages() which when ran from the package root returns a list of package names in dot-notation suitable for the packages parameter.

Here is an example:

# setup.py

from setuptools import find_packages, setup

setup(
    #...
    packages=find_packages(exclude='tests'),
    #...
)

p.s. Packaging sucks in every language and every system. It sucks no matter how you slice it.

Answer 2

I've been through this pain myself today. I used the following, yoinked straight from Django's setup.py, which walks the app's filesystem looking for packages and data files (assuming you never mix the two):

import os
from distutils.command.install import INSTALL_SCHEMES

def fullsplit(path, result=None):
    """
    Split a pathname into components (the opposite of os.path.join) in a
    platform-neutral way.
    """
    if result is None:
        result = []
    head, tail = os.path.split(path)
    if head == '':
        return [tail] + result
    if head == path:
        return result
    return fullsplit(head, [tail] + result)

# Tell distutils to put the data_files in platform-specific installation
# locations. See here for an explanation:
# http://groups.google.com/group/comp.lang.python/browse_thread/thread/35ec7b2fed36eaec/2105ee4d9e8042cb
for scheme in INSTALL_SCHEMES.values():
    scheme['data'] = scheme['purelib']

# Compile the list of packages available, because distutils doesn't have
# an easy way to do this.
packages, data_files = [], []
root_dir = os.path.dirname(__file__)
if root_dir != '':
    os.chdir(root_dir)
myapp_dir = 'myapp'

for dirpath, dirnames, filenames in os.walk(myapp_dir):
    # Ignore dirnames that start with '.'
    for i, dirname in enumerate(dirnames):
        if dirname.startswith('.'): del dirnames[i]
    if '__init__.py' in filenames:
        packages.append('.'.join(fullsplit(dirpath)))
    elif filenames:
        data_files.append([dirpath, [os.path.join(dirpath, f) for f in filenames]])

Answer 3

I think the tool you are looking for is Buildout. There lots of places where you can learn more about it, from SlideShare to Pycon videos.

Other similar or related tools which you might want to check out include virtualenv, Fabric, and PIP.

Answer 4

I've been doing a bit of research myself on Django deployment methods recently.

I found these two resources very useful:

• http://www.caktusgroup.com/blog/2010/04/22/basic-django-deployment-with-virtualenv-fabric-pip-and-rsync/ - A basic overview of how to use virtualenv, pip and fabric to do easy Django deployment from dev machine to production.
• https://github.com/jacobian/django-deployment-workshop/ - Jacob(ian) did a Django deployment workshop, with all the code posted on GitHub. All the necessary links are there as well.