dotcloud supervisord twistd exiting early

Question

I'm trying to set up a twistd daemon on dotcloud:

My supervisord.conf file:

[program:apnsd]
command=/home/dotcloud/env/bin/twistd --logfile /var/log/supervisor/apnsd.log apnsd -c gp_config.py
directory=/home/dotcloud/current/apnsd

However, it looks like the command 'exits early', which then prompts Supervisor to try and restart, which then fails because the twistd dameon is running in the background.

From the supervisord log:

more supervisord.log
2012-05-19 03:07:52,723 CRIT Set uid to user 1000
2012-05-19 03:07:52,723 WARN Included extra file "/etc/supervisor/conf.d/uwsgi.c
onf" during parsing
2012-05-19 03:07:52,723 WARN Included extra file "/home/dotcloud/current/supervi
sord.conf" during parsing
2012-05-19 03:07:52,922 INFO RPC interface 'supervisor' initialized
2012-05-19 03:07:52,922 WARN cElementTree not installed, using slower XML parser
 for XML-RPC
2012-05-19 03:07:52,923 CRIT Server 'unix_http_server' running without any HTTP
authentication checking
2012-05-19 03:07:52,932 INFO daemonizing the supervisord process
2012-05-19 03:07:52,934 INFO supervisord started with pid 144
2012-05-19 03:07:53,941 INFO spawned: 'apnsd' with pid 147
2012-05-19 03:07:53,949 INFO spawned: 'uwsgi' with pid 149
2012-05-19 03:07:54,706 INFO exited: apnsd (exit status 0; not expected)
2012-05-19 03:07:55,712 INFO spawned: 'apnsd' with pid 175
2012-05-19 03:07:55,712 INFO success: uwsgi entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)
2012-05-19 03:07:56,261 INFO exited: apnsd (exit status 1; not expected)
2012-05-19 03:07:58,267 INFO spawned: 'apnsd' with pid 176
2012-05-19 03:07:58,783 INFO exited: apnsd (exit status 1; not expected)
2012-05-19 03:08:01,790 INFO spawned: 'apnsd' with pid 177
2012-05-19 03:08:02,840 INFO success: apnsd entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)

From the apnsd log:

dotcloud@hack-default-www-0:/var/log/supervisor$ more apnsd-stderr---supervisor
-m7GnKV.log
INFO:root:Reactor Type: <twisted.internet.pollreactor.PollReactor object at 0x10
a09d0>
DEBUG:root:Creating listener: apnsd.listeners.line.LineProtocolFactory
INFO:root:Listening on Line Protocol on :1055
DEBUG:root:Listener Created: <apnsd.listeners.line.LineProtocolFactory instance
at 0x12fc8c0>
DEBUG:root:Creating App Factory: apnsd.daemon.APNSFactory
INFO:root:Connecting to APNS Server, App: apns_dev:AAA.com.company.www
INFO:root:apns_dev:AAA.com.company.www -> Started connecting to APNS con
nector...
INFO:root:Registering Application: apns_dev:GoParcel...
DEBUG:root:Creating App Factory: apnsd.daemon.APNSFactory
INFO:root:Connecting to APNS Server, App: apns_dev:T365ED94A9.com.appitems.parce
ls
INFO:root:apns_dev:T365ED94A9.com.appitems.parcels -> Started connecting to APNS
 connector...
INFO:root:Registering Application: apns_dev:GoParcelVictor...
Another twistd server is running, PID 172

This could either be a previously started instance of your application or a
different application entirely. To start a new one, either run it in some other
directory, or use the --pidfile and --logfile parameters to avoid clashes.

Another twistd server is running, PID 172
--More--(42%)

Status of worker is failed:

./dotcloud run hack.worker supervisorctl status
USER PATH IS:  C:\Users\Taras/.dotcloud\dotcloud.conf
# supervisorctl status
apnsd                            FATAL      Exited too quickly (process log may
have details)

But the twistd process is there (ps -ef):

dotcloud   171     1  0 03:13 ?        00:00:00 /home/dotcloud/env/bin/python /home/dotcloud/env/bin/twistd --logfile /var/log/supervisor/apnsd.log apnsd -c gp_config.py

I am having a similar problem when trying to start the process through a wrapper script (and using exec so that a child process isn't created). What am I doing wrong?

Answer 1

Supervisor expects the controlled process to remain in the foreground, but twistd forks to the background by default. Supervisor therefore thinks that it has exited, and tries to start it again.

You should start twistd with the --nodaemon option: twistd will remain in the foreground, and Supervisor will be happy!