104: Connection reset by peer: nginx + rainbows + over 1 mb uploads

Question

I am running ThreadPool rainbows + nginx (unix socket)

On large file uploads I am getting the following in nginx error log (nothing in the application log):

readv() failed (104: Connection reset by peer) while reading upstream

The browser receives response:

413 Request Entity Too Large

Why does this happen?

• "client_max_body_size 80M;" is set both on http and server level (just in case) in nginx
• nginx communicates with rainbows over a unix socket (upstream socket + location @ proxy_pass)
• I don't see anything in the other logs. I have checked:
  • rainbows log
  • foreman log
  • application log
  • dmesg and /var/log/messages
• This happens when uploading a file ~> 1 MB size

Answer 1

Turns out Rainbows had a configuration option called client_max_body_size that defaulted to 1 MB The option is documented here

If this options is on, Rainbows will 413 to large requests silently. You might not know it's breaking unless you run something in front of it.

Rainbows! do
  # let nginx handle max body size
  client_max_body_size nil 
end

Answer 2

The ECONNRESET (Connection reset by peer) error means that connection was uncleanly closed by a backend application. This usually happens if backend application dies, e.g. due to segmentation fault, or killed by the OOM killer. To find out exact reason you have to examine your backend logs (if any) and/or system logs.

Answer 3

Maybe you have client_max_body_size set into your nginx.conf that limits the size of the body to 1Mb, e.g.

client_max_body_size 1M;

In this case you'd need to remove it to allow uploading files of more than 1M.