Why does my Python subprocess error when managed by supervisord?

Question

I wrote a small API for screen captures - a flask app that passes capture requests to CasperJS using subprocess.Popen

In development on a Mac, and when I run the server in the shell on my production server which is Ubuntu 13.04 - everything works great.

When I manage the server with supervisord, however, the subprocess call returns an error that CasperJS cannot find PhantomJS (Casper runs on Phantom).

The error thrown is:

Fatal: [Errno 2] No such file or directory; did you install phantomjs?

The code is all open source.

Here is the subprocess call:

https://github.com/pwalsh/moment/blob/master/moment/models.py#L215

Here is the supervisor conf file for the server (I generate the actual file with Fabric, but it should be clear):

https://github.com/pwalsh/moment/blob/master/fabfile/templates.py#L56

There are only two users on the system - root, and my app's user. When I log on the machine as either of these users, I can run a dev server successfully, and I can run PhantomJS and CasperJS successfully.

Why does my subprocess error with supervisord?

Edit: Adding code + stacktrace

Supervisord conf for the gunicorn server:

; Generated via Fabric on 2013-08-18 23:05:50.928087
; gunicorn configuration for Moment
[program:moment-gunicorn]

command=/srv/environments/moment/bin/gunicorn moment:app --bind 127.0.0.1:9000 --workers 4 --timeout 30 --access-logfile /srv/logs/moment_gunicorn_access.log --error-logfile /srv/logs/moment_gunicorn_error.log

environment=PATH="/srv/environments/moment/bin"
directory=/srv/projects/moment
user=moment
autostart=true
autorestart=true

The code that sends data to the CasperJS/PhantomJS subprocess. It is a method of a class, the full code is here:

def capture(self):

    filename = '{key}.{format}'.format(key=self.get_key().lstrip(self.prefix),
                                      format=self.arguments['format'])

    image = os.path.join(conf.CAPTURES_ROOT, filename)

    params = [conf.CASPER, conf.CAPTURE_SCRIPT, self.arguments['url'],
              image, self.arguments['viewport'], self.arguments['target']]

    casper = subprocess.Popen(params, stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE)

    casper_output, casper_errors = casper.communicate()

    logging.info(casper_output)
    logging.info(casper_errors)
    logging.info(casper.returncode)

    # Here we are relying on convention:
    # If success, subprocess.returncode == 0
    # This could be fragile, need to investigate.
    if casper.returncode:

        raise Exception(casper_errors)

    else:

        return image

Traceback:

WARNING:root:Fatal: [Errno 2] No such file or directory; did you install phantomjs?
WARNING:root:
WARNING:root:1
ERROR:moment:Exception on /capture/ [GET]
Traceback (most recent call last):
  File "/srv/environments/moment/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/srv/environments/moment/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/srv/environments/moment/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/srv/environments/moment/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/srv/environments/moment/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/srv/projects/moment/moment/views.py", line 45, in get_capture
image = capture.capture()
File "/srv/projects/moment/moment/models.py", line 229, in capture
raise Exception(casper_errors)
Exception

Note:

• I am running in a virtualenv called "moment", and under a user called "moment".
• The error is in the casper_output variable - those first three warning are the warning I log when I start the subprocess
• I note that those warning are raised by root - I'd have expected them to be raised by "moment", the user that the supervisord process is supposed to run as

Answer 1

Although (for reasons you should investigate) the user has escalated from the original moment user to root, this does not mean that the process has the environment present when you login to a shell as root.

The chances are that your path is only that which is set in your supervisord.conf, and that's why phantomjs appears to be absent.

Environments are not looked up in some database of users; instead they are constructed by explicitly setting values (e.g. using a script) or by inheritance from the spawning process. In this case, you are inheriting from supervisor, and will get whatever environment supervisor had. If supervisor was run by something like cron, that environment will be empty.

Best practice in relation to supervisor is either to run it using a wrapper script which sets up the environment correctly, or just to explicitly set everything in the supervisord.conf. I generally recommend the latter, unless you have a common set of environment fixups in a file used by a lot of scripts (such as because you want it to run inside a virtualenv).