A simple Python deployment problem - a whole world of pain

Question

We have several Python 2.6 applications running on Linux. Some of them are Pylons web applications, others are simply long-running processes that we run from the command line using nohup. We're also using virtualenv, both in development and in production. What is the best way to deploy these applications to a production server?

In development we simply get the source tree into any directory, set up a virtualenv and run - easy enough. We could do the same in production and perhaps that really is the most practical solution, but it just feels a bit wrong to run svn update in production. We've also tried fab, but it just never works first time. For every application something else goes wrong. It strikes me that the whole process is just too hard, given that what we're trying to achieve is fundamentally very simple. Here's what we want from a deployment process.

1. We should be able to run one simple command to deploy an updated version of an application. (If the initial deployment involves a bit of extra complexity that's fine.)
2. When we run this command it should copy certain files, either out of a Subversion repository or out of a local working copy, to a specified "environment" on the server, which probably means a different virtualenv. We have both staging and production version of the applications on the same server, so they need to somehow be kept separate. If it installs into site-packages, that's fine too, as long as it works.
3. We have some configuration files on the server that should be preserved (ie. not overwritten or deleted by the deployment process).
4. Some of these applications import modules from other applications, so they need to be able to reference each other as packages somehow. This is the part we've had the most trouble with! I don't care whether it works via relative imports, site-packages or whatever, as long as it works reliably in both development and production.
5. Ideally the deployment process should automatically install external packages that our applications depend on (eg. psycopg2).

That's really it! How hard can it be?

Answer 1

Development and deployment of Python code is made much easier by setuptools in combination with virtualenv and pip.

Core ideas

The trickiest part, I've found, is running a development environment that mirrors the deployed setup as closely as possible, while at the same time respecting Pythonic tools and idioms. But it turns out that this is very easy to achieve with pip and setuptools, which together allow you to "install" a development tree into a Python environment without moving the files around. (Actually setuptools does this all by itself, but pip acts as a front end handle dependencies better.)

Another key issue is preparing a clean environment with a known package set across both environments. Python's virtualenv is a god-send in this respect, allowing you to configure a completely customised Python environment with your own choice of packages, without requiring root access, or OS packages (rpm or dpkg), and without being constrained by whatever packages and versions thereof that happen to be installed on your distro.

Finally, one annoying bug-bear is the difficulty of creating command-line scripts that play nice with PYTHON_PATH. This is also dealt with quite elegantly by setuptools.

Setting up

(To keep things simple, this is fairly prescriptive. Feel free to diverge as appropriate.)

1. Prepare a working directory that your Python stuff can call home.
2. Grab the latest virtualenv from the bottom of this page and untar it (doesn't matter where).

3. From the working directory, setup a new Python virtual environment:

   $ python <untarred_directory>/virtualenv.py venv
   

4. You'll want to do most of your work from inside this virtual environment. Use this command to do so (. is a shortcut for source):

   $ . venv/bin/activate
   

5. Install pip:

   $ easy_install pip
   

6. Create directories for each installable package you want to create.

7. In each directory, you'll need a setup.py that defines the content and structure of the package. The setuptools documentation is a very good resource for getting started on this. It's worth taking the time absorb large chunks of it.

Development

Once your tree structure is ready, you are almost ready to begin coding. But right now, packages that depend on each other can't see each other as they will in the deployed environment. This problem is resolved with a neat little trick that setuptools offers, and which pip makes use of. For each package you are developing, run the following command (make sure you are in the virtual environment for your project, as per step 3, above):

$ pip install -e pkg1

This command will install pkg1 into your virtual environment, and it does so without copying any of your files. It simply adds a link to the site-packages directory pointing to the package's development root and creates an egg-info directory in that root. You can also do this without pip, as follows:

$ cd pkg1
$ python setup.py develop

And it will usually work, but if you have third-party dependencies (which should be listed in setup.py, as explained here in the setuptools documentation), pip is smarter about finding them.

One caveat to note is that neither setuptools nor pip have any smarts about finding dependencies amongst your own packages. If PkgB in directory B, depends on PkgA in directory A, then pip install -e B will fail, because pip has no way of knowing that PkgA can be found in directory A; it will instead try, and fail, to download PkgA from its online repository sources. The workaround is simply to install each package after its dependencies.

At this point, you can start python, load up one of your modules and start toying with it. You can edit code, and it will be immediately available the next time you import it.

Finally, if you want to create command-line tools with your packages. Don't write them by hand. You'll end up with a horrible mess of PYTHON_PATH hacks that never quite works properly. Just read up on automatic script creation in the setuptools documentation. This will spare you a lot of grief.

Deployment

When your packages are ready for action, you can use setup.py to create deployment packages. There are way too many options to go into here, but the following should get you started:

$ cd pkg1
$ python setup.py --help
$ python setup.py --help-commands

Loose ends

Due to the broad nature of the question, this answer is necessarily incomplete. I haven't dealt with long-running servers, web frameworks or the actual deployment process itself (in particular, use of pip install's --index-url to manage a private repository of third-party and internal packages and -e vcs+..., which will pull packages out of svn, git, hg or bzr). But I hope I've given you enough rope to tie it all together (just don't hang yourself with it :-).

Answer 2

This is really not hard. You need to play mostly with buildout and supervisord IMO.

While learning buildout may take a little time but it's worth it, given amount of pain it reduces in repeated setups.

About nohup: nohup approach does not suit serious deployments. I have very good experience with supervisord. It is excellent solution for running production python applications. It is very easy to setup.

Some specific answers below.

1. single command to deploy: Buildout is the answer. We are using it since a couple of years with not much of problems
2. Usually it's like you checkout the source. Then run buildout. Further it may not be a good idea to let the setup copy into site-packages. Better keep environments separate.
3. Configs would not be overwritten.
4. You may/should consider building egg(s) for the common packages. Like you may build the egg for a package (say commonlib) and upload it on your code repository. Then you can specify this as a dependency in your buildout.cfg
5. Buildout is capable of building most essential packages completely separate from central/top level install. However in my experience the python packages with c extension if installed as OS packages, it's much easier.

Answer 3

I've been working on implementing this for our work projects. It's a few different parts involved.

First, we customize virtualenv.py using their bootstrap abilities to add in your own custom post-creation functions and flags. These allow us to define common types of projects and also gives us a single command to create a new virtualenv, checkout a project from the git repository, and install any requirements into the virtualenv using pip and requirements.txt files.

so our commands look like: python venv.py --no-site-packages -g $git_proj -t $tag_num $venv_dir

http://pypi.python.org/pypi/virtualenv http://pip.openplans.org/

Now that gets us through the initial checking out of an existing project. As we work and update the project we use fabric commands within each project to build releases and then to deploy them:

http://docs.fabfile.org/0.9.0/

I've got a fab command: make_tag which checks for unused commits, opens files that need version strings updated, builds and uploads sphinx docs, and then commits the final tag to the repository.

The flip side is a fab deploy command which will, over ssh, do a git co of the tag specified, run a pip update on any new requirements, run any database migrations needed, and then resets the web server if this is a web application.

Here's an example of the tagging function: http://www.google.com/codesearch/p?hl=en#9tLIXCbI4vU/fabfile.py&q=fabfile.py%20git%20tag_new_version&sa=N&cd=1&ct=rc&l=143

There are a ton of good fabric files you can browse through using the google code search. I know I cheat-sheeted a few for my own use.

It's definitely complicated and has several parts in order to get things running smooth. Once you get it running though, the flexibility and speed for things is just awesome.

Answer 4

Have a look at Buildout for reproducible deployments.

Answer 5

Another vote for fabric (haven't tried Buildout yet). We've been using it successfully for a couple of months now.

If you're having trouble with fabric, another option is Capistrano. Works great (even for non-rails apps). Only stopped using it because it feels weird to use Ruby to deploy Python apps ;)

Answer 6

I would use rsync to synchronize outwards from your production "prime" server to the others, and from your "beta test" platform, to your production "prime" server.

rsync has the benefit of copying only those files which changed, and copying only parts of files that changed partially, and verifying the integrity and identical content at the end on all machines. An update that gets part way through and is interrupted can easily be continued later, making your deployment more robust.

Subversion or Mercurial would not be a bad idea in this case either. Mercurial has the advantage of allowing you to "pull" or "push" instead of just updating from one central source. You might find interesting cases where a decentralized model (mercurial) works better.

Answer 7

if you are a buildout person, then you should know about minitage.recipe.scripts ability to generate a file to set your python environment. Source to your web server and your buildout is completely portable.

Answer 8

It sounds like what you want is a build script. So write one, using shell, python, ant, or your favorite build tool. If you don't like writing in XML, pant allows you to write ant scripts in python. Several people have mentioned buildout but I don't have any experience with it.

First define your steps. It sounds like you want to:

1. SVN export from your production tag (you don't want to have a working copy in prod)
2. set virtualenv
3. easy_install or pip install required packages (or probably use pre-downloaded & tested versions)
4. copy production configuration files to your target (it's not a good idea to keep this information in your source repo -- though you could have them versioned separately)
5. restart your server & do any other setup task
6. run smoke tests and rollback on failure

If you're doing load balancing or depending on other services production you might want to figure out a way to roll out limited scope so all your customers aren't affected at once. If you have a production-like staging environment that might also satisfy that need.