WARN max_connections from pt-variable-advisor - what to configure for DBaaS?

Question

Our VM configuration (hosted on Vmware)

# cat /proc/cpuinfo |grep "cpu cores" | awk -F: '{ num+=$2 } END{ print "cpu cores", num }'
cpu cores 32

# free -h
             total       used       free     shared    buffers     cached
Mem:           62G        23G        39G       500K       349M        10G
-/+ buffers/cache:        12G        50G
Swap:          50G         0B        50G

I got from pt-variable-advisor a warning about max_connections:

pt-variable-advisor h=localhost,u=root,p=Quule0juqu7aifohvo2Ahratit --socket /var/vcap/sys/run/mysql/mysqld.sock
(...)
# WARN max_connections: If the server ever really has more than a thousand threads running, then the system is likely to spend more time scheduling threads than really doing useful work.
(...)

Why? Any details?

Configuration in my.cnf

max_connections                 = 15360

settings from prd db cluster (MariaDB 10.1.x and Galera)

MariaDB [(none)]> SHOW STATUS WHERE `variable_name` = 'Threads_connected';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Threads_connected | 718   |
+-------------------+-------+
1 row in set (0.01 sec)

MariaDB [(none)]> SHOW STATUS WHERE `variable_name` = 'Max_used_connections';
+----------------------+-------+
| Variable_name        | Value |
+----------------------+-------+
| Max_used_connections | 924   |
+----------------------+-------+
1 row in set (0.02 sec)

I found a MySQL manual and an (very) old similar question.

The default value is 151 to improve performance

The number of connections permitted is controlled by the max_connections system variable. The default value is 151 to improve performance when MySQL is used with the Apache Web server. (Previously, the default was 100.) If you need to support more connections, you should set a larger value for this variable.

and

The maximum number of connections MySQL supports depends on the quality of the thread library on a given platform, the amount of RAM available, how much RAM is used for each connection, the workload from each connection, and the desired response time. Linux or Solaris should be able to support at least 500 to 1000 simultaneous connections routinely and as many as 10,000 connections

We currently have 460 users and every user is allowed to do 100 max_connections. This would be the max value. Is the 100 max_connections per user and database too high? With modern connection pooling we can set this to 20? How we should configure that not risking overloading our server with context switching? Is it possible that one web app use one connection (every statement over same connection)?

Answer 1

There is a difference between Threads_running and Threads_connected. The former implies that many queries are simultaneously doing something. The other are sleeping, waiting for the user (or app) to present another SQL statement to the server.

max_connections is a limit on Threads_connected. (Max_used_connections is a "high-water-mark".)

A typical busy system will have only 1 in 7 'connected' threads actually running. And Threads_connected is typically much less than max_connections. (Like 20x)

But, in bad times, too many connections each doing some slow query, leads to MySQL stumbling over itself. Throughput (queries/second) hits some maximum and may even go down; latency goes through the roof, and a panicky DBA will reboot the machine because he thinks it has crashed, not just gotten terribly tangled up.

The solution is to find the slow queries and figure out how to speed them up. Or re-architect the app to be more efficient. Or change the schema and indexes. It is rarely to get bigger hardware or change the tunables.

In a well-managed MySQL server, I rarely see more than 2-3 cores in use. Yet, thousands of queries/second can be performed even with very few cores active.

In DBaaS, you are at the mercy of the customers. Any one of them can write a sloppy app that hogs connections, cores, I/O, network bandwidth, anything.

Consider limiting each user to 10 connections unless he explains what he needs to do and/or pays you more money. With the extra money, you can isolate him in a separate VM. I once estimated, in a shop with Apache and MySQL, that 10 threads would swamp Apache, to MaxChild needed to be much lower than the default. (We had rather heavy pages.) If you have control over web servers, limit how many threads they have. That is stop the problem before it gets to MySQL.

By putting every user in his own VM, you consume extra resources, but you can prevent the excesses of one from harming the others.

I'll bet many of your clients use WordPress? Here's one performance patch to the schema: http://mysql.rjweb.org/doc.php/index_cookbook_mysql#speeding_up_wp_postmeta

Answer 2

If you want to avoid context switches, use MariaDB thread pool, ignore whatever pt-advisor says, and do not limit your connections. of course if you have many concurrent long and CPU intensive requests, it might not help too much, and in this case you'd need to figure out how to fix the long queries.