(Mysql) Using Memory engine for 4.3 million row table?

Question

My new company stores an intermediate data for our loading process in a table using the memory engine. However a quite simple update (set a.id=b.id) is taking something like six hours to run.

I've checked how big the tables are and in this import they have a about 4.3 million rows of data in them. I'm certain the problem is the memory engine and the lack of index on id (there is a PK but its across 5 other attributes) based on HASH.

I'm 95% certain I need to change the table engine and index type but I want to make sure before I push for the change.

EDIT: Here is the statement that is is so slow:

select  @Extraction_id := id from extraction;

update items as a
set a.extraction_id=@Extraction_id;

Edit 2: create statement of table

CREATE TABLE `items` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `extraction_id` char(12) NOT NULL,
  `a` char(6) NOT NULL,
  `b` char(10) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
  `c` date NOT NULL,
  `d` varchar(100) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
  `e` varchar(255) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL,
  `f` decimal(17,6) DEFAULT NULL,
  `g` decimal(17,6) DEFAULT NULL,
  `h` varchar(150) DEFAULT NULL,
  `i` varchar(150) DEFAULT NULL,
  PRIMARY KEY     (`id`,`extraction_id`,`a`,`b`,`c,`d`),
  KEY `fk_extraction` (`extraction_id`),
) ENGINE=MEMORY AUTO_INCREMENT=2343054016 DEFAULT CHARSET=latin1;

Answer 1

ENGINE=MEMORY is not designed for big tables. InnoDB is well designed for virtually all use cases; switch to it.

It sounds like you need more than an ENGINE and index type change. Please provide SHOW CREATE TABLE for the relevant tables, and the statement that is so slow.

set a.id=b.id?? -- By convention, DBAs call the PRIMARY KEY id. It is very strange to be setting the PK of one table to the PK of another.

Chunking

One approach to UPDATEing 4M rows will less impact is to "chunk" up the problem. I discuss details here.