20 mysqld processes using only 1 port?

Question

These are from Aliyun Cloud DB Servers，20 mysqld processes using one 3306 port

• Has anyone seen this before ?
• Can anybody explain how to do that ?
• Are there any advantages to this ?

Answer 1

All those processes you are seeing in the OS are all kernel threads emanating from mysqld.

Look carefully at the first picture

• Linux Process ID 26101 is mysqld_safe
• Linux Process ID 26307 is thread mysqld_safe launches mysqld
• Linux Process ID 26308 is actually mysqld
• 19 Linux Process IDs 26309 - 26334 are threads coming from mysqld (26308)

My guess is that they are simply DB Connections but other internal threads are possible

If you want to corroborate this, you can run the following:

• In mysql, SELECT COUNT(1) FROM information_schema.processlist;
• In the OS, run netstat | grep -c 3306 or netstat | grep mysql | grep -c ESTABLISHED
• Results may be off in favor of the OS

Older versions of mysqld tend to do this in the OS. As an example, look at the following:

• Nov 28, 2013 : Why do top and ps show different PIDs for the same processes?
• Apr 28, 2012 : Is multiple mysqld processes for 1 mysql server is normal?
• May 29, 2007 : mysql creating lots of processes (not threads, linux processes)

I have always suspected that when mysqld has this behavior, it was probably built via source compiling but without proper compiling options. mysqld would manifest this as multiple connections in the eyes of the OS.

• Baron Schwartz seemed to think so from 2007
• Scott Tanner seemed to think so from 2007
• I was just mentioning this last year : Too many mysql instances. How to kill?
• I made a passing remark about this 4 years ago : innodb_buffer_pool_size not changed

CONCLUSION

You really have nothing to worry about. mysqld will still function properly. It is all about the environment the mysqld binaries were created in versus where those binary are running.

Answer 2

What we see is impossible, so the logical assumption is that it's an illusion.

Multiple MySQL server instances can't listen on the same port. They also can't share the same pid file, as these do, nor can the data files for InnoDB be shared.

The most sensible explanation for this is that the tools are showing you threads, not processes -- for whatever reason. Threads in Linux (if that's what this is) share the same process ID, but they also do have their own thread ID, a number that comes from the same numbering space as PIDs.

Most compelling, I suspect, is there are 20+ entries, each consuming 10.9% of available memory. While it is highly unlikely for multiple identical instances if MySQL server to hold the same amount of memory (particularly the busy ones), it is impossible that your memory usage on this server is in excess of 200%.

mysql> SHOW FULL PROCESSLIST; when executed by a user with the SUPER or PROCESS privilege should show you the active client threads, and will likely contain entries representing to two that seem particularly busy, here. Of course, there will be fewer in the output than are shown here, since not all threads are client threads.

Prove to yourself that there is really only one instance running, by repeatedly connecting and disconnecting and issuing SELECT CONNECTION_ID(), UUID_SHORT(); and/or by doing this on more than one connection at the same time. Both values will increment pretty much monitically (connection_id increments with each connection but remains the same while connected, uuid_short increments with each query) and will never be smaller. This would be virtually impossible if multiple MySQL server daemon instances were involved.

Answer 3

They're not all using the same port.

The main mysqld process will have forked its children off (you can see the child<>parent relationship in the 2nd and 3rd columns of the ps output - pid 26308 looks to be the master "thread". Only one of the processes will actually be listening on the TCP port. You can verify this using lsof. I'll add that a forked process will inherit the commandline from its parent, unless argv[] has been explicitly altered before the fork() call, hence the confusion with --port=3306 - it means nothing.

Answer 4

Depending on the Operating System, you might see 1 process for mysqld, or you might see 1 per connection. Each initially connected on port 3306.

If you have connection pooling (in the client) and/or a high value of thread_cache_size and a high value of wait_timeout, there may be a bunch of idle connections that "don't go away".

As Rolando says, "nothing to worry about".

On the other hand, 39.6% CPU is very high. Do SHOW FULL PROCESSLIST; to see if you can catch it. Then consider improving that busy query. (This may or may not be something to "worry about".)