Insert-heavy InnoDB table won't use all my CPU

Question

I have a packet log database, which is almost never queried. It just needs to be fast on inserts. I'm using InnoDB because I'd like to maintain ACID compliance, since even losing a single packet could be damaging to our customers. In a performance tuning scenario, I send 1,000,000 packets to the server over multiple DB connections. But no matter what settings I use in my.cnf, I can't get the mysqld process to use more than 900% CPU on a system with 12 cores. (Nothing else is running on the box.)

I've set the following

• innodb_file_per_table = 1
• innodb_write_io_threads = 64
• innodb_read_io_threads = 64
• innodb_thread_concurrency = 0

If I use MyISAM, I can get all the packets written in about 6 seconds. But InnoDB takes about 25. Can I make MySQL use the rest of the system resources and insert faster?

Edit: Here's the schema for the table:

+-------+----------------------+------+-----+---------+-------+
| Field | Type                 | Null | Key | Default | Extra |
+-------+----------------------+------+-----+---------+-------+
| t     | bigint(20) unsigned  | YES  |     | NULL    |       |
| a     | char(1)              | YES  |     | NULL    |       |
| sa    | int(10) unsigned     | YES  |     | NULL    |       |
| sb    | int(10) unsigned     | YES  |     | NULL    |       |
| sc    | int(10) unsigned     | YES  |     | NULL    |       |
| sd    | int(10) unsigned     | YES  |     | NULL    |       |
| sp    | smallint(5) unsigned | YES  |     | NULL    |       |
| da    | int(10) unsigned     | YES  |     | NULL    |       |
| db    | int(10) unsigned     | YES  |     | NULL    |       |
| dc    | int(10) unsigned     | YES  |     | NULL    |       |
| dd    | int(10) unsigned     | YES  |     | NULL    |       |
| dp    | smallint(5) unsigned | YES  |     | NULL    |       |
+-------+----------------------+------+-----+---------+-------+

edit2: I've batched more inserts together so that a single query is near the maximum length (about 16,000,000 chars). The database now spikes to 1100% for two seconds, then goes down to 100% for the rest of the time. Total time is now 21 seconds, or about 16% faster than when I started.

Answer 1

You have to crank up innodb_io_capacity as well.

The default is 200. Raise it to 5000 for starters. I would go to 20000.

You may also want to make sure ib_logfile0 and ib_logfile1 are sufficiently large. The default value for innodb_log_file_size is 5M. I would raise that to 1G for starters.

A larger InnoDB Buffer Pool would also help, perhaps 4G.

To recap, use these additional settings:

[mysqld]
innodb_io_capacity=5000
innodb_buffer_pool_size=4G
innodb_log_file_size=1G

After adding these settings to my.cnf, to resize ib_logfile0/ib_logfile1 do the following

service mysql stop
rm -f /var/log/mysql/ib_logfile[01]
service mysql start

The files ib_logfile0 and ib_logfile1 are recreated. Don't worry, I have done this many times.

You may have to do something out of the ordinary for InnoDB

Try the following:

• Full Table Lock on the InnoDB table
• Perform the Bulk Load
• Release the Lock

Answer 2

There are a number of factors that impact the ability to maximize use of multiple cores.

• Some mutexes will impact multiple CPUs leaving some waiting before they can proceed.
• You need as many active threads as you have CPUs. If your workload results in 9 parallel thread, you can't fill 12 cores.
• I/O capacity must be sufficient to provide enough work for all CPUs. If you are queueing on disk I/O or waiting for network messages, then you won't be able to fill the CPUs.

Tools like SAR will enable you to determine if there are any bottlenecks which are reducing your capacity. Just be warned, eliminating one bottleneck, will just move the bottleneck.