How can I optimize this query

Question

I am fighting this since yesterday morning but i still cannot get it working fast enough.

The items table has the artist and the name fields as varchar. All other fields are integers of various types indexed. I have tested the query with or without the indexes on the two varchar fields and no change.

The items_categories has three columns: id, item_id and cat_id. SQL_CALC_FOUND_ROWS has no affect whatsoever.

I have 65k records in both tables, each item having one category thus one relation in the items_category table.

The culprit query.

SELECT SQL_CALC_FOUND_ROWS `items`.`id`
  FROM `items`
    LEFT JOIN `items_categories` ON `items`.`id` = `items_categories`.`item_id`
  WHERE `items`.`id` > 0 AND `items`.`hidden` = 0 AND `items`.`deleted` = 0 AND `items_categories`.`cat_id` IN(1)
  GROUP BY `items`.`id` 
  ORDER BY `items`.`artist` ASC, `items`.`name` ASC
  LIMIT 0, 100;

This is what explain shows me.

"id"    "select_type"   "table" "type"      "possible_keys"                          "key"  "key_len"   "ref"   "rows"  "Extra"
"1"     "SIMPLE"       "items_categories"   "ref"                                    "item_id,cat_id"   "cat_id"    "4" "const" "12152" "Using where; Using temporary; Using filesort"
"1"    "SIMPLE"        "items"  "eq_ref"    "PRIMARY,hidden,deleted,hidden_deleted" "PRIMARY"   "4" "staviny_db.items_categories.item_id"   "1" "Using where"

The whole thing takes about 0.2 - 0.3 seconds. I know it is due to the order clause is responsible for half that time.

Is there anything I could do to at least cut my time in half without major changes?

Answer 1

There are a couple of problems with your query.

• You use a LEFT OUTER JOIN, yet in the WHERE clause you say AND "items_categories"."cat_id" IN(1) which makes the OUTER JOIN effectively an INNER JOIN because each row in "items" that has no corresponding record in "items_categories" will return NULL for "items_categories"."cat_id" and thus get eliminated by the WHERE clause
• There quite a bit of blogs that indicate that SQL_CALC_FOUND_ROWS can have a negative effect on performance; are you sure you really need it ?
• I'm confused you can do GROUP BY "items"."id" and then follow it with ORDER BY "items"."artist" ASC, "items"."name" ASC. I'm used to MSSQL and this simply will not compile!?!
• I'm assuming that the the GROUP BY is there because the JOIN would cause 'doubles' as multiple records in "items"categories" might fit the requirements. If so, you should rather use a WHERE EXISTS construction instead of doing the actual JOIN and then GROUPing everything again to get rid of them. (same story for DISTINCT which is just a fancy GROUP BY anyway)

Hence, I would suggest this:

SELECT "items"."id"
  FROM "items"  
 WHERE "items"."id" > 0 
   AND "items"."hidden" = 0 
   AND "items"."deleted" = 0 
   AND EXISTS ( SELECT *
                  FROM "items_categories" 
                 WHERE "items_categories"."item_id" = "items"."id"
                   AND "items_categories"."cat_id" IN (1) )
 ORDER BY "items"."artist" ASC, "items"."name" ASC
 LIMIT 0, 100;

(which --more or less-- would also compile in MSSQL =)

Answer 2

SELECT SQL_CALC_FOUND_ROWS `items`.`id`
FROM `items`
LEFT JOIN `items_categories` ON `items`.`id` = `items_categories`.`item_id`
WHERE `items_categories`.`cat_id` = 1 AND `items`.`hidden` = 0 AND `items`.`deleted` = 0 
GROUP BY `items`.`id` 
ORDER BY `items`.`artist`, `items`.`name`
LIMIT 0, 100;

Answer 3

Using group by already orders the results, so you are orderign them twice. Maybe you could use only an GROUP BY clause:

     SELECT SQL_CALC_FOUND_ROWS `items`.`id`
      FROM `items`
 LEFT JOIN `items_categories` ON `items`.`id` = `items_categories`.`item_id`
     WHERE `items`.`id` > 0 
       AND `items`.`hidden` = 0 
       AND `items`.`deleted` = 0 
       AND `items_categories`.`cat_id` IN(1)
  GROUP BY `items`.`artist` ASC, `items`.`name` ASC, `items`.`id` 
     LIMIT 0, 100;

I suppose that you have already the right indices created, don't you?