optimise mysql query with LIKE operator for 10k records

Question

Note:

So as I figured, the real problem is because of the IN clause I'm using for tagids. Changing the portion of query for text search didn't help much. Any idea how to improve the query?

The query takes too long while running on the server. Here Partha S is a search item entered by user. The table contacts contains personal information , tags contains category name and id; and contacts2tags table contains contactid and tagid with values similar to id in contacts and tags respectively.

    SELECT *
    FROM
    (
    SELECT *,
     IF
    (
     first_name LIKE 'Partha S'
    OR last_name LIKE 'Partha S'
    OR phone_number LIKE 'Partha S'
    OR mobile_number LIKE 'Partha S'
    OR email_address LIKE 'Partha S'
    OR address LIKE 'Partha S'
    OR organization LIKE 'Partha S'
    OR other LIKE 'Partha S'
    OR sector LIKE 'Partha S'
    OR designation LIKE 'Partha S'
    OR concat ( first_name,  ' ',  last_name ) LIKE 'Partha S'
    OR concat ( last_name,  ' ',  first_name ) LIKE 'Partha S',
     1,
     0 )
     as exact,
     IF
    (
    (
     first_name LIKE '%Partha%'
    OR last_name LIKE '%Partha%'
    OR phone_number LIKE '%Partha%'
    OR mobile_number LIKE '%Partha%'
    OR email_address LIKE '%Partha%'
    OR address LIKE '%Partha%'
    OR organization LIKE '%Partha%'
    OR other LIKE '%Partha%'
    OR sector LIKE '%Partha%'
    OR designation LIKE '%Partha%' )
    AND
    (
     first_name LIKE '%S%'
    OR last_name LIKE '%S%'
    OR phone_number LIKE '%S%'
    OR mobile_number LIKE '%S%'
    OR email_address LIKE '%S%'
    OR address LIKE '%S%'
    OR organization LIKE '%S%'
    OR other LIKE '%S%'
    OR sector LIKE '%S%'
    OR designation LIKE '%S%' )
    ,
     1,
     0 )
     as normal
    FROM contacts
    WHERE id in
    (
    SELECT DISTINCT contacts.id
    from contacts INNER
    JOIN contacts2tags ON contacts.id = contacts2tags.contactid
    WHERE ( tagid in ( 178 ) ) )
     )
     d
    WHERE exact = 1
    OR normal = 1
    ORDER BY exact desc,
     last_name asc LIMIT 0,
     20

UPDATE: As per the suggestions, I removed the LIKE operator for exact search, and used MATCH(..) AGAINST(..) instead of LIKE in the latter case. While the first change did improve the performance a little, but using MATCH() AGAINST() didn't change the execution time surprisingly. Here's the updated query. PS I tried using both MATCH(all cols) AGAINST(search item) and MATCH(single cols) AGAINST (search item) combined with OR. Please suggest. thanks

     SELECT *
    FROM
    (
    SELECT *,
     IF
    (
         first_name ='Partha S'
       OR last_name ='Partha S'
       OR phone_number ='Partha S'
       OR mobile_number ='Partha S'
       OR email_address = 'Partha S'
       OR address ='Partha S'
       OR organization ='Partha S'
       OR other ='Partha S'
       OR sector ='Partha S'
       OR designation ='Partha S'
       OR concat ( first_name,  ' ',  last_name ) ='Partha S'
       OR concat ( last_name,  ' ',  first_name ) ='Partha S',
       1,
       0 )
      as exact,
       IF
      ( match(first_name,last_name,phone_number,mobile_number,email_address,  address,organization,other,sector,designation) against( 'Partha')                 
    OR  match(first_name,last_name,phone_number,mobile_number,email_address,address,organization,other,sector,designation) against( 'S')


    ,
    1,
    0 )
     as normal
    FROM contacts
    WHERE id in
    (
    SELECT DISTINCT contacts.id
    from contacts INNER
    JOIN contacts2tags ON contacts.id = contacts2tags.contactid
    WHERE ( tagid in ( 178 ) ) )
     )
     d
    WHERE exact = 1
    OR normal = 1
    ORDER BY exact desc,
     last_name asc LIMIT 0,
      20

Answer 1

One optimization is that in the exact case, you don't need to use LIKE (you should only use it with the wildcard - %).

Another thing that you can do to make the things faster is adding an INDEX to the fileds you're going to be searching in.

Also, only if you're using MyISSAM as your storage engine (for that table) you can use full text search like this

SELECT * FROM normal WHERE MATCH(title, body) AGAINST ('Queried_string')

first_name LIKE '%S%'
OR last_name LIKE '%S%'
OR phone_number LIKE '%S%'
OR mobile_number LIKE '%S%'
OR email_address LIKE '%S%'
OR address LIKE '%S%'
OR organization LIKE '%S%'
OR other LIKE '%S%'
OR sector LIKE '%S%'
OR designation LIKE '%S%' )

seems to be bringing very little value to the whole process.

Hope this helps.

Answer 2

It's not just all the LIKEs, but also the ORs. Even for conditions using LIKE, indexes will be used though. So to speed this query up, you can make one very big index per table that combines all the fields you are searching in.

But if you really want to build a search enginge, you might want to consider using Sphinx or ElasticSearch instead of MySQL monster queries like this.

Answer 3

You might like to have a look at match() and against() functionality of MySQL.

Here is some sample code from their documents

mysql> CREATE TABLE articles (
    ->   id INT UNSIGNED AUTO_INCREMENT NOT NULL PRIMARY KEY,
    ->   title VARCHAR(200),
    ->   body TEXT,
    ->   FULLTEXT (title,body)
    -> ) ENGINE=MyISAM;
Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO articles (title,body) VALUES
    -> ('MySQL Tutorial','DBMS stands for DataBase ...'),
    -> ('How To Use MySQL Well','After you went through a ...'),
    -> ('Optimizing MySQL','In this tutorial we will show ...'),
    -> ('1001 MySQL Tricks','1. Never run mysqld as root. 2. ...'),
    -> ('MySQL vs. YourSQL','In the following database comparison ...'),
    -> ('MySQL Security','When configured properly, MySQL ...');
Query OK, 6 rows affected (0.00 sec)
Records: 6  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM articles
    -> WHERE MATCH (title,body) AGAINST ('database');
+----+-------------------+------------------------------------------+
| id | title             | body                                     |
+----+-------------------+------------------------------------------+
|  5 | MySQL vs. YourSQL | In the following database comparison ... |
|  1 | MySQL Tutorial    | DBMS stands for DataBase ...             |
+----+-------------------+------------------------------------------+
2 rows in set (0.00 sec)

READ MORE HERE - http://dev.mysql.com/doc/refman/4.1/en/fulltext-natural-language.html