mysql 8 - rds - 1022 Can't write; duplicate key in table

Question

Documenting this error and solution here in case it could help anyone. We have a MYSQL 8 database on Amazon RDS. The below group by used to work just fine on Mysql 5.7 but it started giving
1022 Can't write; duplicate key in table '/rdsdbdata/tmp/#sqlf80_1656bc_0'
error once we upgraded to Mysql 8.

GROUP BY DATE_FORMAT(CONCAT(YEAR(performance_stats_date),'-',  
MONTH(performance_stats_date),'-',DAY(performance_stats_date)),
 '%Y-%m-%d')  

Answer 1

There are two solutions we found:
1. Either reduce the number of columns in the query (in our case we have about 800 columns in this particular query, reducing it to less than about 100 solved the problem)
2. or simplify the group by statement to something like GROUP BY performance_stats_date

Answer 2

That's the wrong way to use DATE_FORMAT. It is given a DATE or DATETIME and a string providing the 'formatting'. Example:

mysql> SELECT DATE_FORMAT(NOW(), "%Y-%m-%d"), DATE(NOW());
+--------------------------------+-------------+
| DATE_FORMAT(NOW(), "%Y-%m-%d") | DATE(NOW()) |
+--------------------------------+-------------+
| 2019-01-30                     | 2019-01-30  |
+--------------------------------+-------------+

In your case, it would be something like

  GROUP BY DATE_FORMAT(performance_stats_date, "%Y-%m-%d")

But this is even simpler:

  GROUP BY DATE(performance_stats_date)

For further discussion, please provide

SHOW VARIABLES LIKE 'char%'
SHOW CREATE TABLE` for each table.
the entire query

Answer 3

The most likely you already have a constraint with the name "x" (the one you are trying to create).

Constraints must be unique for the entire database, not just for the specific table you are creating/altering.

To find out where the constraints are currently in use you can use the following query:

SELECT `TABLE_SCHEMA`, `TABLE_NAME`
FROM `information_schema`.`KEY_COLUMN_USAGE`
WHERE `CONSTRAINT_NAME` = 'x';

as a solution you can try to use another name for this constraint.

Answer 4

Error caused when query creates an internal temporary table while executing a query and internal_tmp_mem_storage_engine=TempTable (default) setting.

most likely you will see this error for queries with the group by clause and when tables use by query has a large number of columns.

This is a bug in MySQL-8 and the following is a workaround for now.

Workaround:

SET session internal_tmp_mem_storage_engine=MEMORY;

Run Query.