Optimizing the mysql query - Avoid creation of temporary table?

https://stackoverflow.com/questions/22746817

24-06-2023
|

Question

This is the query that I am using on tables : products, reviews, replies, review_images.

Query :

SELECT products.id, reviews.*,
GROUP_CONCAT(DISTINCT CONCAT_WS('~',replies.reply, replies.time)) AS Replies,
GROUP_CONCAT(DISTINCT CONCAT_WS('~',review_images.image_title, review_images.image_location)) AS ReviewImages
FROM products
LEFT JOIN reviews on products.id = reviews.product_id
LEFT JOIN replies on reviews.id = replies.review_id
LEFT JOIN review_images on reviews.id = review_images.review_id
WHERE products.id = 1
GROUP BY products.id, reviews.id;

Schema :

Products :

id  |  name  |  product_details....

Reviews :

id  |  product_id  |  username  |  review  |  time  | ...

Replies :

id  |  review_id   |  username  |  reply  |  time  | ...

Review Images :

id  |  review_id  |  image_title  |  image_location  | ...

Indexes:

Products :

PRIMARY KEY - id

Reviews :

PRIMARY KEY - id

FOREIGN KEY - product_id (id IN products table)

FOREIGN KEY - username (username IN users table)

Replies :

PRIMARY KEY - id

FOREIGN KEY - review_id (id IN reviews table)

FOREIGN KEY - username (username IN users table)

Review Images :

PRIMARY KEY - id

FOREIGN KEY - review_id (id IN reviews table)

Explain Query :

1 | SIMPLE | reviews | ALL | product_id | 4 | Using where; Using join buffer (Block Nested Loop)

1 | SIMPLE | replies | ref | review_id | 1 | Null

1 | SIMPLE | review_images | ALL | review_id | 5 | Using where; Using join buffer (Block Nested Loop)

I don't know what is wrong here, that it needs to use filesort and create a temporary table?

Here are few Profiling results :

Opening Tables 140 µs

Init 139 µs

System Lock 34 µs

Optimizing 21 µs

Statistics 106 µs

Preparing 146 µs

Creating Tmp Table 13.6 ms

Sorting Result 27 µs

Executing 11 µs

Sending Data 11.6 ms

Creating Sort Index 1.4 ms

End 89 µs

Removing Tmp Table 8.9 ms

End 34 µs

Query End 25 µs

Closing Tables 66 µs

Freeing Items 41 µs

Removing Tmp Table 1.4 ms

Freeing Items 46 µs

Removing Tmp Table 1.2 ms

Freeing Items 203 µs

Cleaning Up 55 µs

As from the Explain and Profiling results, it is clear that temporary table is created to produce the results. How can I optimize this query to get similar results and better performance and avoid the creation of temporary table?

Help would be appreciated. Thanks in advance.

EDIT

Create Tables

CREATE TABLE `products` (
 `id` int(11) NOT NULL AUTO_INCREMENT,
 `name` varchar(100) NOT NULL,
 `description` varchar(100) NOT NULL,
 `items` int(11) NOT NULL,
 `price` int(11) NOT NULL,
 PRIMARY KEY (`id`)
) ENGINE=InnoDB

CREATE TABLE `reviews` (
 `id` int(11) NOT NULL AUTO_INCREMENT,
 `username` varchar(30) NOT NULL,
 `product_id` int(11) NOT NULL,
 `review` text NOT NULL,
 `time` datetime NOT NULL,
 `ratings` int(11) NOT NULL,
 PRIMARY KEY (`id`),
 KEY `product_id` (`product_id`),
 KEY `username` (`username`)
) ENGINE=InnoDB

CREATE TABLE `replies` (
 `id` int(11) NOT NULL AUTO_INCREMENT,
 `review_id` int(11) NOT NULL,
 `username` varchar(30) NOT NULL,
 `reply` text NOT NULL,
 `time` datetime NOT NULL,
 PRIMARY KEY (`id`),
 KEY `review_id` (`review_id`)
) ENGINE=InnoDB

CREATE TABLE `review_images` (
 `id` int(11) NOT NULL AUTO_INCREMENT,
 `review_id` int(11) NOT NULL,
 `image_title` text NOT NULL,
 `image_location` text NOT NULL,
 PRIMARY KEY (`id`),
 KEY `review_id` (`review_id`)
) ENGINE=InnoDB

EDIT:

I simplified the query above and now it does not create temporary tables. The only reason as mentioned by @Bill Karwin was that I was using GROUP BY on second table in the joins.

Simplified query :

SELECT reviews. * ,
GROUP_CONCAT( DISTINCT CONCAT_WS( '~', replies.reply, replies.time ) ) AS Replies,
GROUP_CONCAT( DISTINCT CONCAT_WS( '~', review_images.image_title, review_images.image_location ) ) AS ReviewImages
FROM reviews
LEFT JOIN replies ON reviews.id = replies.review_id
LEFT JOIN review_images ON reviews.id = review_images.review_id
WHERE reviews.product_id = 1
GROUP BY reviews.id

Now the PROBLEM that I'm facing is :

Because I'm using GROUP_CONCAT, there is a limit to the data it can hold which is in the variable GROUP_CONCAT_MAX_LEN, so as I'm concatenating the replies given by the users, it could go very very long and can possibly exceed the memory defined. I know I can change the value of GROUP_CONCAT_MAX_LEN for current session, but still there is a limitation to it that at some point in time, the query may fail or unable to fetch complete results.

How can I modify my query so as not to use GROUP_CONCAT and still get results expected.

POSSIBLE SOLUTION :

Simply using LEFT JOINS, which creates duplicate rows for every new result in the last column and which makes it hard to traverse in php? Any suggestions?

I see this question is not getting enough response from SO members. But I've been looking for the solution and searching about concepts since last to last week. Still no luck. Hope some of you PROs can help me out. Thanks in advance.

Solution

You can't avoid creating a temporary table when your GROUP BY clause references columns from two different tables.

The only way to avoid the temporary table in this query is to store a denormalized version of the data in one table, and index the two columns you're grouping by.

Another way you can simplify and get results in a format that's easier to work with in PHP is to do multiple queries, without GROUP BY.

First get the reviews. Example is in PHP & PDO, but the principle applies to any language.

$review_stmt = $pdo->query("
    SELECT reviews.*,
    FROM reviews
    WHERE reviews.product_id = 1");

Arrange them in an associative array keyed by the review_id.

$reviews = array();
while ($row => $review_stmt->fetch(PDO::FETCH_ASSOC)) {
    $reviews[$row['d']] = $row;
}

Then get the replies and append them to an array using the key 'replies'. Use INNER JOIN instead of LEFT JOIN, because it's okay if there are no replies.

$reply_stmt = $pdo->query("
    SELECT replies.*
    FROM reviews
    INNER JOIN replies ON reviews.id = replies.review_id
    WHERE reviews.product_id = 1");
while ($row = $reply_stmt->fetch(PDO::FETCH_ASSOC)) {
    $reviews[$row['review_id']]['replies'][] = $row; 
}

And do the same for review_images.

$reply_stmt = $pdo->query("
    SELECT review_images.*
    FROM reviews
    INNER JOIN review_images ON reviews.id = review_images.review_id
    WHERE reviews.product_id = 1");
while ($row = $reply_stmt->fetch(PDO::FETCH_ASSOC)) {
    $reviews[$row['review_id']]['review_images'][] = $row; 
}

The end result is an array of reviews, which contains elements which are nested arrays for related replies and images respectively.

The efficiency of running simpler queries can make up for the extra work of running three queries. Plus you don't have to write code to explode() the group-concatted strings.

Licensed under: CC-BY-SA with attribution

Not affiliated with StackOverflow