optimize expensive php query/subquery

Question

I have a table with 3 columns: submission_id, column_id, data. Each submission_id has 25 columns and data values.

I have a page that displays all submissions where column_id = 16 and data = ('' or 0). To get this, I use a subquery to get all the distinct submission_id's that I need, and then I get all columns in the main query. My query is:

SELECT sid,cid,data FROM webform_submitted_data WHERE sid in(select distinct sid from webform_submitted_data where cid=16 and data in (' ',0)) ORDER BY sid ASC

The table is getting large, and the query now takes 30-40 seconds when run from PHP, (though only 1.0e-6 seconds from MYSQL) It is not PHP overhead, I checked using the mysqld-slow.log file, I get the following: <-- # Query_time: 32.975552 Lock_time: 0.000138 Rows_sent: 108 Rows_examined: 177396 -->

I also tried running an explain in PHP ![explain]:(http://i.imgur.com/692eyHf.png)

One more thing, this page updates the current submission and puts an ID value in column_id 16, which takes it off of the page when it reloads. Reloads without an update take less than a second, but when we need to update 100 records, it rebuilds the cache every time.

Any thoughts would be greatly appreciated.

Answer 1

It's the DEPENDENT SUBQUERY reported by the EXPLAIN that is the most costly. That means that it executes the subquery once for each distinct sid in the outer query. How much overhead this costs is relative to the number of distinct sid values over your 180000+ rows.

You can try this query to ensure the subquery is executed only once. It may need to store the results in a temp table, but that temp table will only have ~7 rows, according to your EXPLAIN.

SELECT d1.sid, d1.cid, d1.data 
FROM webform_submitted_data AS d1 
INNER JOIN (
  SELECT DISTINCT sid FROM webform_submitted_data 
  WHERE cid=16 AND data IN (' ',0)) AS d2 USING (sid)
ORDER BY d1.sid ASC