Can't Seem to hit MySQL max_connections Although New Connections Fail Under Load

Question

We are benchmarking one of our sites with ab (apachebench) on a page that creates a MySQL connection, queries and then closes the connection (through PHP).

We have hard coded our my.cnf with a max_connections limit of 500.

When we run the stress test, MySQL never seems to hit the connection limit we specify although we DO HAVE "unable to connect to the database" type errors returned from our script. It's as if MySQL can't open any more connections than around ~237 at one time.

Here is our current my.cnf:

[mysqld]
max_connections = 500
port            = 3306
socket          = /var/mysql/mysql.sock
skip-external-locking
key_buffer_size = 384M
max_allowed_packet = 1M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 128M
thread_concurrency = 8
max_heap_table_size = 512M
tmp_table_size = 512M
table_cache = 2048

User Limits:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 16384
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1250
virtual memory          (kbytes, -v) unlimited

The problem seems to be either with PHP or MySQL (or of course misconfiguration by us).

Any help, suggestions and tips are greatly appreciated - thanks in advance.

Answer 1

Okay, so I have found the culprit for this problem.

When running the stress load, it was not gradual - all of the connections were opened at once flooding the server.

MySQL has a default back_log of 50 - which is the number of outstanding connections MySQL can have before processing them. We were hitting this limit when flooding the server and further connections over the 50 threshold were refused.

To fix this, we edited my.cnf with the following line, increasing the listen backlog to a reasonable 128 which works for our setup:

back_log = 128

Please be aware that the maximum value you can enter here is dependant on your OS and the limits set within your OS. MySQL can have a maximum listen backlog of 65535 however your OS may have a figure less than that.

With the new back_log of 128 our problem is fixed and we are no longer getting Connection refused error message under load.