MySQL can take more than an hour to start

Question

I have a mysql (Percona) 5.7 instance with over 1Million tables. When I start the database, it can take more than an hour to start. Errorlog doesn't show anything, but when I trace mysqld_safe, I found out that MySQL is getting a stat on every file in the DB.

Any idea why this may happen? Also, please no suggestion to fix my schema, this is a blackbox. Also keep in mind that I've heard it all regarding how bad the design is....

Thanks

Answer 1

This turned out to be 2 issues (other than millions of tables)!

1. When MySQL start, and a crash recovery is needed, as of 5.7.17, it needs to traverse you datadir to build it's dictionary. This will be lifted in future releases(8.0), as MySQL will have it's own catalog, and will not rely on datadir content anymore. Thanks to 'Rick James' for this hint. Doc states that this isn't done anymore. It's true and false. It doesn't read the 1st page of ibd files, but It does a file stat. Filed Bug
2. Once it finished (1), it starts a new process, "Executing 'SELECT * FROM INFORMATION_SCHEMA.TABLES;' to get a list of tables using the deprecated partition engine.". That of course open all the files again. Use disable-partition-engine-check if you think you don't need it. Doc

All this can be observed using sysdig. very powerful handy dtrace like tool.

sysdig proc.name=mysqld | grep "open fd="

Ok, now, it's time to reduce the number of files. Again "Rick James", gave me some good solution....

Comments? Opinions?

Answer 2

The solution is to drastically decrease the number of files. I see 3 approaches, depending on your version:

Plan A (the only option for 5.6 and earlier):

Set innodb_file_per_table=OFF, restart your client, then do ALTER TABLE ... ENGINE=InnoDB all but the 1000 largest tables. This will shovel all the non-huge tables into ibdata1.

Plan B (if you have at least 5.7, and you have 10-1000 databases):

If you have multiple databases, create a General Tablespace per database. Put non-huge tables into those tablespaces.

Leave the huge tables in their own tablespaces.

To facilitate either Plan A or B, write a SELECT against information_schema that creates the desired ALTER statements, then execute them.

Plan C: Move to 8.0, which might actually work reasonably fast with a million tables due to its Data Dictionary. (Or, you might need a combo of Plan C together with A or B.)

Answer 3

It sounds like your database administrator has set the innodb_stats_persistent= value to OFF. Change this value to ON, and do a restart. The restart will take a long time as normal, but the following reboots should be much faster.

If this device is really a "blackbox" to you, I doubt you can change this yourself. I'd contact your DBA.

See the documentation for more details.