FastCGI - Broken socket pipe OR Connection reset by peer (PHP5-FPM or php-fpm)

Question

I am experiencing a deterministic issue with php-fpm.

Every time that I send a LARGE GET request to Web server, with more than about 100KB, php-fpm aborts the socket connection, breaking the socket pipe. The web server, in turns, return a 500 Internal Server Error. The Apache error log shows a (32) Broken pipe:

[Fri May 09 02:42:23 2014] [error] [client 127.0.0.1] (32)Broken pipe: FastCGI: comm with server "/usr/lib/cgi-bin/php5-fcgi" aborted: write failed

I know that such large requests should be implemented using POST, instead of GET, but, since I am using Wikimedia software as a wikitext parser, I can't control the way it handles the requests, which are GET method. Furthermore, since the texts are large, reaching even 1MB, I have to send large query strings.

The issue is freaking me out!

I tried to change the php-fpm debug level to Error in /etc/php5/fpm/php-fpm.conf, but NOTHING related to the error is logged to /var/log/php5-fpm.log. Since it aborted the connection (broke the socket pipe), I was expecting, at least, an error log.

The way I found to debug the problem was to analyse the connection packages using tcpflow. I clearly see php-fpm aborting the connection in the middle of TCP transmission, exactly when it reaches about 100K.

Googling, I could see that other people experienced related issues, but there is no clear answer to the problem, and most of them experienced the broken pipe error in an intermittent way.

In my case it is deterministic and happens when the request surpass 100KB.

Looking at the php-fpm configuration file I don't see any limit on the request size. I am running the php-fpm as a socket (not host) service in Apache 2.

How can I solve or work around it?

Is there something hard coded in php-fpm that aborts the connection when the query_string goes beyond 100K? Is it a parameter?

What makes me confident that it is exclusively related to PHP5-FPM is that when I implement PHP5 using Apache modules, it handles the large requests.

My configuration:

marcelo@marcelo-VirtualBox:/usr/local/apache2/logs$ /usr/local/apache2/bin/apachectl -v
Server version: Apache/2.2.27 (Unix)
Server built:   May 10 2014 06:21:24
marcelo@marcelo-VirtualBox:/usr/local/apache2/logs$ 


marcelo@marcelo-VirtualBox:/usr/local/apache2/logs$ php -v
PHP 5.5.12 (cli) (built: May 10 2014 07:47:39) 
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
marcelo@marcelo-VirtualBox:/usr/local/apache2/logs$ 

*Apache2.conf:*

<IfModule mod_fastcgi.c>
    AddHandler php5-fcgi .php
    Action php5-fcgi /php5-fcgi
    Alias /php5-fcgi /usr/lib/cgi-bin/php5-fcgi
    #FastCgiExternalServer /usr/lib/cgi-bin/php5-fcgi -host 127.0.0.1:9000 -pass-header Authorization -idle-timeout 300
    FastCgiExternalServer /usr/lib/cgi-bin/php5-fcgi -socket /tmp/php5-fpm.sock -pass-header Authorization -idle-timeout 300
</IfModule>

EDIT 1

In order to add to the question I attach the strace when passing a query_string where text='IT IS A TEST.' (repeated 1000 times (13KB size)). Anyone can easily replicate this, using ANY php code. As I said, it does not depend on the script, but only to the fact that I am passing a large query_string.

IMPORTANT:

PID php-fpm: -p 8653 -p 8654 -p 8655 -p 8656

PID Apache: -p 4178 -p 4562 -p 4563 -p 4564 -p 4565 -p 4566 -p 4567 -p 4568 -p 4570 -p 8425 -p 8426 -p 8427

Below the AN EXCERPT of command: sudo strace -e trace=open,read,write,readv,writev,recv,recvfrom,send,sendto -s 999 -p 8653 -p 8654 -p 8655 -p 8656 -p 4178 -p 4562 -p 4563 -p 4564 -p 4565 -p 4566 -p 4567 -p 4568 -p 4570 -p 8425 -p 8426 -p 8427 2> ~/tmp/trace.txt

[pid  8427] write(10, "\1\1\0\1\0\10\0\0\0\1\0\0\0\0\0\0\1\4\0\1\0\35\0\0\20\vREDIRECT_HANDLERphp-fastcgi\1\4\0\1\0\24\0\0\17\3REDIRECT_STATUS200\1\4\0\1\0\24\0\0\t\tHTTP_HOSTlocalhost\1\4\0\1\0-\0\0\24\27HTTP_ACCEPT_ENCODINGgzip, deflate, compress\1\4\0\1\0\20\0\0\v\3HTTP_ACCEPT*/*\1\4\0\1\0L\0\0\17;HTTP_USER_AGENTpython-requests/2.2.1 CPython/2.7.3 Linux/3.11.0-20-generic\1\4\0\1\0B\0\0\4<PATH/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\1\4\0\1\0\22\0\0\20\0SERVER_SIGNATURE\1\4\0\1\0007\0\0\17&SERVER_SOFTWAREApache/2.2.27 (Unix) mod_fastcgi/2.4.6\1\4\0\1\0\26\0\0\v\tSERVER_NAMElocalhost\1\4\0\1\0\26\0\0\v\tSERVER_ADDR127.0.0.1\1\4\0\1\0\17\0\0\v\2SERVER_PORT80\1\4\0\1\0\26\0\0\v\tREMOTE_ADDR127.0.0.1\1\4\0\1\0(\0\0\r\31DOCUMENT_ROOT/usr/local/apache2/htdocs\1\4\0\1\0\35\0\0\f\17SERVER_ADMINyou@example.com\1\4\0\1\0007\0\0\17&SCRIPT_FILENAME/usr/local/apache2/cgi-bin/php-fastcgi\1\4\0\1\0\22\0\0\v\5REMOTE_PORT50305\1\4\0\1\374\26\0\0\25\200\1\373\374REDIRECT_QUERY_STRINGaction=parse&text=IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+"..., 8192) = 8192
[pid  8656] read(3, "\1\1\0\1\0\10\0\0", 8) = 8
[pid  8427] write(10, "ST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+"..., 8192 <unfinished ...>
[pid  8656] read(3,  <unfinished ...>
[pid  8427] <... write resumed> )       = 8192
[pid  8656] <... read resumed> "\0\1\0\0\0\0\0\0", 8) = 8
[pid  8656] read(3, "\1\4\0\1\0\35\0\0", 8) = 8
[pid  8427] write(10, ".IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TE"..., 8192 <unfinished ...>
[pid  8656] read(3,  <unfinished ...>
[pid  8427] <... write resumed> )       = 8192
[pid  8656] <... read resumed> "\20\vREDIRECT_HANDLERphp-fastcgi", 29) = 29
[pid  8427] write(10, "T+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST"..., 8192) = 8192

**...**
**(THUS, PID 8427 (`APACHE`) WRITE THE FASTCGI PROTOCOL AND PASS THE LARGE QUERY_STRING...)**
**...**
**(NOW PID 8656 (`PHP-FPM`) JUMPS IN...READING THE FASTCGI PROTOCOL STREAM...)**
**...**

[pid  8656] read(3, "\1\4\0\1\0\24\0\0", 8) = 8
[pid  8656] read(3, "\17\3REDIRECT_STATUS200", 20) = 20
[pid  8656] read(3, "\1\4\0\1\0\24\0\0", 8) = 8
[pid  8656] read(3, "\t\tHTTP_HOSTlocalhost", 20) = 20
[pid  8656] read(3, "\1\4\0\1\0-\0\0", 8) = 8
[pid  8656] read(3, "\24\27HTTP_ACCEPT_ENCODINGgzip, deflate, compress", 45) = 45
[pid  8656] read(3, "\1\4\0\1\0\20\0\0", 8) = 8
[pid  8656] read(3, "\v\3HTTP_ACCEPT*/*", 16) = 16
[pid  8656] read(3, "\1\4\0\1\0L\0\0", 8) = 8
[pid  8656] read(3, "\17;HTTP_USER_AGENTpython-requests/2.2.1 CPython/2.7.3 Linux/3.11.0-20-generic", 76) = 76
[pid  8656] read(3, "\1\4\0\1\0B\0\0", 8) = 8
[pid  8656] read(3, "\4<PATH/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", 66) = 66
[pid  8656] read(3, "\1\4\0\1\0\22\0\0", 8) = 8
[pid  8656] read(3, "\20\0SERVER_SIGNATURE", 18) = 18
[pid  8656] read(3, "\1\4\0\1\0007\0\0", 8) = 8
[pid  8656] read(3, "\17&SERVER_SOFTWAREApache/2.2.27 (Unix) mod_fastcgi/2.4.6", 55) = 55
[pid  8656] read(3, "\1\4\0\1\0\26\0\0", 8) = 8
[pid  8656] read(3, "\v\tSERVER_NAMElocalhost", 22) = 22
[pid  8656] read(3, "\1\4\0\1\0\26\0\0", 8) = 8
[pid  8656] read(3, "\v\tSERVER_ADDR127.0.0.1", 22) = 22
[pid  8656] read(3, "\1\4\0\1\0\17\0\0", 8) = 8
[pid  8656] read(3, "\v\2SERVER_PORT80", 15) = 15
[pid  8656] read(3, "\1\4\0\1\0\26\0\0", 8) = 8
[pid  8656] read(3, "\v\tREMOTE_ADDR127.0.0.1", 22) = 22
[pid  8656] read(3, "\1\4\0\1\0(\0\0", 8) = 8
[pid  8656] read(3, "\r\31DOCUMENT_ROOT/usr/local/apache2/htdocs", 40) = 40
[pid  8656] read(3, "\1\4\0\1\0\35\0\0", 8) = 8
[pid  8656] read(3, "\f\17SERVER_ADMINyou@example.com", 29) = 29
[pid  8656] read(3, "\1\4\0\1\0007\0\0", 8) = 8
[pid  8656] read(3, "\17&SCRIPT_FILENAME/usr/local/apache2/cgi-bin/php-fastcgi", 55) = 55
[pid  8656] read(3, "\1\4\0\1\0\22\0\0", 8) = 8
[pid  8656] read(3, "\v\5REMOTE_PORT50305", 18) = 18
[pid  8656] read(3, "\1\4\0\1\374\26\0\0", 8) = 8
[pid  8656] read(3, "\25\200\1\373\374REDIRECT_QUERY_STRINGaction=parse&text=IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+A+TEST.IT+IS+"..., 64534) = 64534
**[pid  8427] read(10, 0x1528f20, 8192)   = -1 ECONNRESET (Connection reset by peer)**

**As can be seen, the connection is reset in the middle of transmission!!!**

SOLVED

Please, see the answer below.

Answer 1

There is a limit on the request header size of fastcgi protocol specification.

The maximum size is 0xffff or ~64KB.

Since GET method pass the query string within the header, no matter what web server you install, if it implements the fastcgi protocol, you will end with an error (usually converted to 500).

A workaround is to change your request method to POST, given that the string goes within the message body, instead of the header.

It is not a solution to the problem, since the limits are "hardcoded" on the protocol API, but, at least, a workaround.