Rails / Passenger: The backend application did not send a valid HTTP response

Question

This morning my Rails application crashed because the mysql db couldn't write tmp files anymore because my disc space was full. I fixed this by pointing mysql to a tmpdir on a partition with more space, restarted mysql, restarted Apache, all of my other apps are running happily again. But not my Rails app.

I deploy it with Apache, Passenger and Capistrano. Now, after I fixed the mysql/space problem the application comes up again, but whenever I send a POST request to the server it comes back with a 500 Internal Server Error. I checked the Apache error_log and this is what it says:

[Tue Apr 03 12:19:41 2012] [error] [client 130.xx.x.xxx] Premature end of script headers: logout, referer: http://myrailsapp.com/customers
[ pid=8942 thr=47916460421888 file=ext/apache2/Hooks.cpp:817 time=2012-04-03 12:19:41.52 ]: The backend application (process 9139) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashe
d; please check whether there are crashing bugs in this application.
[ pid=9139 thr=231420160 file=utils.rb:176 time=2012-04-03 12:19:41.053 ]: *** Exception Errno::ENOENT in application (No such file or directory - /tmp/RackRewindableInput20120403-9139-60boph.lock) (process 9139, thread #<Thread:0x00
00001b966200>):
    from /usr/local/lib/ruby/1.9.1/tempfile.rb:342:in `rmdir'
    from /usr/local/lib/ruby/1.9.1/tempfile.rb:342:in `rmdir'
    from /usr/local/lib/ruby/1.9.1/tempfile.rb:334:in `ensure in locking'
    from /usr/local/lib/ruby/1.9.1/tempfile.rb:334:in `locking'
    from /usr/local/lib/ruby/1.9.1/tempfile.rb:144:in `block in initialize'
    from /usr/local/lib/ruby/1.9.1/tmpdir.rb:133:in `create'
    from /usr/local/lib/ruby/1.9.1/tempfile.rb:134:in `initialize'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/utils/rewindable_input.rb:86:in `new'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/utils/rewindable_input.rb:86:in `make_rewindable'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/utils/rewindable_input.rb:30:in `read'
    from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.3.6/lib/rack/request.rb:183:in `POST'
    from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.3.6/lib/rack/methodoverride.rb:15:in `call'
    from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.3.6/lib/rack/runtime.rb:17:in `call'
    from /usr/local/lib/ruby/gems/1.9.1/gems/activesupport-3.1.3/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
    from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.3.6/lib/rack/lock.rb:15:in `call'
    from /usr/local/lib/ruby/gems/1.9.1/gems/rack-cache-1.1/lib/rack/cache/context.rb:132:in `forward'
    from /usr/local/lib/ruby/gems/1.9.1/gems/rack-cache-1.1/lib/rack/cache/context.rb:139:in `pass'
    from /usr/local/lib/ruby/gems/1.9.1/gems/rack-cache-1.1/lib/rack/cache/context.rb:151:in `invalidate'
    from /usr/local/lib/ruby/gems/1.9.1/gems/rack-cache-1.1/lib/rack/cache/context.rb:70:in `call!'
    from /usr/local/lib/ruby/gems/1.9.1/gems/rack-cache-1.1/lib/rack/cache/context.rb:50:in `call'
    from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.1.3/lib/rails/engine.rb:456:in `call'
    from /usr/local/lib/ruby/gems/1.9.1/gems/railties-3.1.3/lib/rails/railtie/configurable.rb:30:in `method_missing'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_request_handler.rb:513:in `accept_and_process_next_request'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/rack/application_spawner.rb:171:in `block in handle_spawn_application'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/utils.rb:479:in `safe_fork'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/rack/application_spawner.rb:166:in `handle_spawn_application'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_server.rb:180:in `start'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/rack/application_spawner.rb:129:in `start'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in spawn_rack_application'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_application'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize'
    from <internal:prelude>:10:in `synchronize'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/helper-scripts/passenger-spawn-server:99:in `<main>'

    [root@ngrl logs]# 
    [root@ngrl logs]# tail /etc/httpd/logs/error_log
    from <internal:prelude>:10:in `synchronize'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
    from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.11/helper-scripts/passenger-spawn-server:99:in `<main>'

Here's what I've tried: I restarted Apache several times and I restarted the rails app by putting a restart.txt file under tmp. I've also redeployed it with cap deploy, but nothing seems to work. I don't have this problem with the development version deployed with Webrick, which has exactly the same code and a database in the same location as the production version. It must be a Passenger problem. GET requests work fine, but not POST or PUT.

Any ideas anyone?

Answer 1

I'm thinking you may have other space issues. Have you made sure your rails app still has space for the log and tmp directories?