Why MYSQL "NOT IN" operation on 100 million records is not working?

Question

Am trying to get the all rows from Tabl1 which are not available in Table2 with help of NOT IN MSQL query. But am getting timeout exception and query is not getting executed. Below is the mysql query which I am using.

    SELECT * FROM identity WHERE
  unique_id NOT IN (SELECT Message_Queue.index FROM Message_Queue);

Could any please tell the reason or any other way for replacement of NOT IN operation?

Answer 1

When you have so many records in the in() clause then you should use a join instead

SELECT t1.* 
FROM table1 t1
left join table2 t2 on t2.myId = t1.myId 
where t2.myId is null

Answer 2

Because in MySQL NOT IN is less performant, try using EXISTS

SELECT  * 
FROM    identity a
WHERE   NOT EXISTS
        (
            SELECT  null
            FROM    Message_Queue b
            WHERE   b.index = a.unique_id 
        );

you should also put an index on those columns.