MySQL partitioning considerations

Question

I have a table that will eventually have about 1.4 billion records in it, 11 columns, 3 primary keys.

I've populated about the first 1/3 of the table and the query times are already quite slow. I've reformatted several columns to reduce space requirements, one of the primary keys is an index-like integer that links to a meta data table where each index expands into 3 additional options (effectively normalizing what were 6 primary keys down to 3 primary keys).

My next thought was to create a lot of partitions. The second primary key is a date column, and this is by far the most common dimension for slicing the data. I'm hoping that partitions will be effective and also improve insert times too.

However, how do I know what the right number of partitions is? My initial guess was 8 and this is too slow. I was about to try 100 when I thought I should ask and Googling didn't give me any helpful answers.

The table's so large, and the computational process that generates it is so expensive, that it's infeasible to try very many different partition numbers. So any kind of standard advice for this is extremely useful.

Answer 1

Here is what you need to consider:

• Maximum number of partitions in MySQL 5.5+ is 1024.
• You can change number of partitions without regenerating numbers, although it requires rebuilding indexes.
• Partitioning speeds upqueries only if the query hits less partitions then you have available cores

If your query hits all or many partitions, then partitioing in this way will not help them much. There is also a chance that your setup is efficient, but hardware is not capable to produce result faster.