Problems using MySql View - extremly slow, but why?

Question

I have a big problem using a mysql view. I am generating a view in which I am joining several tables to receive the informations i really need.

I have the following view:

CREATE OR REPLACE VIEW view_test AS

SELECT nve_nr.nve_nvenr AS collonr,
CONCAT (adr_emp.adr_name1, "\n", adr_emp.adr_name2, "\n", adr_emp.adr_strasse, "\n", adr_emp.adr_land , " -     ", adr_emp.adr_plz, " ", adr_emp.adr_ort) AS emp_adr,

UNIX_TIMESTAMP((SELECT MAX(t_1.colst_timestamp) FROM tbl_collo_status t_1 WHERE t_1.colst_nve_id = 
nve_nr.nve_id AND t_1.colst_status_code IN (10, 11, 76))) AS status_x,

UNIX_TIMESTAMP((SELECT MAX(t_2.colst_timestamp) FROM tbl_collo_status t_2 WHERE t_2.colst_nve_id = 
nve_nr.nve_id AND t_2.colst_status_code IN (20,25,91))) AS status_y,

UNIX_TIMESTAMP((SELECT MAX(t_3.colst_timestamp) FROM tbl_collo_status t_3 WHERE t_3.colst_nve_id = 
nve_nr.nve_id AND t_3.colst_status_code IN (30,99,104))) AS status_z,

TIME_TO_SEC(TIMEDIFF((SELECT MAX(t_2.colst_timestamp) FROM tbl_collo_status t_2 WHERE t_2.colst_nve_id = 
nve_nr.nve_id AND t_2.colst_status_code IN (20,25,91)), (SELECT MAX(t_1.colst_timestamp) FROM tbl_collo_status t_1 
WHERE t_1.colst_nve_id = nve_nr.nve_id AND t_1.colst_status_code IN (10, 11, 76)))) AS timediff_xy,

TIME_TO_SEC(TIMEDIFF((SELECT MAX(t_3.colst_timestamp) FROM tbl_collo_status t_3 WHERE t_3.colst_nve_id = 
nve_nr.nve_id AND t_3.colst_status_code IN (30,99,104)), (SELECT MAX(t_2.colst_timestamp) FROM tbl_collo_status t_2 
WHERE t_2.colst_nve_id = nve_nr.nve_id AND t_2.colst_status_code IN (20,25,91)))) AS timediff_yz,

TIME_TO_SEC(TIMEDIFF((SELECT MAX(t_3.colst_timestamp) FROM tbl_collo_status t_3 WHERE t_3.colst_nve_id = 
nve_nr.nve_id AND t_3.colst_status_code IN (30,99,104)), (SELECT MAX(t_1.colst_timestamp) FROM tbl_collo_status t_1 
WHERE t_1.colst_nve_id = nve_nr.nve_id AND t_1.colst_status_code IN (10, 11, 76)))) AS timediff_xz,

CAST(SUBSTR(nve_nr.nve_nvenr, 1, 3) AS UNSIGNED INTEGER) AS filter_collonr,

(SELECT t_5.colst_status_code FROM tbl_collo_status t_5 WHERE t_5.colst_nve_id=nve_nr.nve_id ORDER BY 
t_5.colst_timestamp DESC LIMIT 1) AS filter_last_status,

sdg.sdg_kunde AS filter_kunden_id,

sdg.sdg_id AS filter_sdg_id,

nve_nr.nve_id AS filter_nve_id,

adr_emp.adr_id AS filter_adr_emp_id

FROM ((tbl_nve_nr nve_nr LEFT JOIN tbl_sendungen sdg ON nve_nr.nve_sdg_id = sdg.sdg_id)

LEFT JOIN tbl_adressen adr_emp ON adr_emp.adr_id = sdg.sdg_adr2_id);

Now I have more than 250.000 datasets in that view an if I start a select on that View I will take more than 20 minutes to receive the result. The Explain Select will return the following:

//![Explain Select from View][1]

I cannot post images without having 10 reputations, so I will try it like this:

Can anyone tell me what I can do to speed up the view? On performing the query without using a View will take about 2 minutes with an Limit of 25 datasets that shouzld be selected from that 250.000 avaiable datasets.

//EDIT: So I've solved the problem on another way :) The solution described below does not maek the SQL-Query works faster enough, so I solved on the following way:

Dropping the views, because they are not needed any more, Altering tables nve_nr adding two fields for last status and timestamp of that. Creating a Trigger on collo_status that replaces that values, if timestamp is newer that existing. -> So one subquery less - Speeds up from 2 minutes to nearly 60 seconds.

Rebuilding the indexes and run the query directly in php -> More code in php, but much fatser - Now speed up to nearly 13 seconds :)

Updating the database with new fields includes an procedure to find the currently newest status.

So the new query is just like this:

SELECT SQL_CALC_FOUND_ROWS 
sdg.sdg_lieferschein AS referenz, 
nve_nr.nve_nvenr AS collonr, 
nve_nr.nve_last_timestamp AS filter_status_time, 
nve_nr.nve_last_status AS filter_last_status, 
sdg.sdg_kunde AS filter_kunden_id, 
sdg.sdg_id AS filter_sdg_id, 
nve_nr.nve_id AS filter_nve_id, 
adr_emp.adr_id AS filter_adr_emp_id, 
adr_emp.adr_name1 AS emp_adr_name1, 
adr_emp.adr_name2 AS emp_adr_name2, 
adr_emp.adr_land AS emp_adr_land, 
adr_emp.adr_plz AS emp_adr_plz, 
adr_emp.adr_ort AS emp_adr_ort,
UNIX_TIMESTAMP(t_1.colst_timestamp) AS status_x, 
t_1.colst_status_code AS status_code_x, 
UNIX_TIMESTAMP(t_2.colst_timestamp) AS status_y, 
t_2.colst_status_code AS status_code_y, 
UNIX_TIMESTAMP(t_3.colst_timestamp) AS status_z, 
t_3.colst_status_code AS status_code_z, 
TIME_TO_SEC(TIMEDIFF(t_2.colst_timestamp, t_1.colst_timestamp)) AS timediff_xy, 
TIME_TO_SEC(TIMEDIFF(t_3.colst_timestamp, t_2.colst_timestamp)) AS timediff_yz, 
TIME_TO_SEC(TIMEDIFF(t_3.colst_timestamp, t_1.colst_timestamp)) AS timediff_xz, 
CAST(SUBSTR(nve_nr.nve_nvenr, 1, 3) AS UNSIGNED INTEGER) AS filter_collonr 
FROM tbl_nve_nr nve_nr 
LEFT JOIN tbl_sendungen sdg 
ON nve_nr.nve_sdg_id = sdg.sdg_id 
LEFT JOIN tbl_adressen adr_emp 
ON (adr_emp.adr_id = sdg.sdg_adr2_id) 
LEFT JOIN tbl_collo_status t_1 
ON (t_1.colst_nve_id = nve_nr.nve_id AND t_1.colst_aktiv = 1 AND t_1.colst_status_code IN (10, 11, 76)) 
LEFT JOIN tbl_collo_status t_2 
ON (t_2.colst_nve_id = nve_nr.nve_id AND t_2.colst_aktiv = 1 AND t_2.colst_status_code IN (20,25,91)) 
LEFT JOIN tbl_collo_status t_3 
ON (t_3.colst_nve_id = nve_nr.nve_id AND t_3.colst_aktiv = 1 AND t_3.colst_status_code IN (30,99,104)) 
GROUP BY collonr 
ORDER BY collonr DESC 
LIMIT 900, 75

//[1]: http://i.stack.imgur.com/TPw6s.png

Answer 1

Main problem is probably indexes, but also well worth removing the co related sub query. Something like this:-

CREATE OR REPLACE VIEW view_test AS

SELECT nve_nr.nve_nvenr AS collonr,
CONCAT (adr_emp.adr_name1, "\n", adr_emp.adr_name2, "\n", adr_emp.adr_strasse, "\n", adr_emp.adr_land , " -     ", adr_emp.adr_plz, " ", adr_emp.adr_ort) AS emp_adr,

UNIX_TIMESTAMP((SELECT MAX(t_1.colst_timestamp) FROM tbl_collo_status t_1 WHERE t_1.colst_nve_id = 
nve_nr.nve_id AND t_1.colst_status_code IN (10, 11, 76))) AS status_x,

UNIX_TIMESTAMP((SELECT MAX(t_2.colst_timestamp) FROM tbl_collo_status t_2 WHERE t_2.colst_nve_id = 
nve_nr.nve_id AND t_2.colst_status_code IN (20,25,91))) AS status_y,

UNIX_TIMESTAMP((SELECT MAX(t_3.colst_timestamp) FROM tbl_collo_status t_3 WHERE t_3.colst_nve_id = 
nve_nr.nve_id AND t_3.colst_status_code IN (30,99,104))) AS status_z,

TIME_TO_SEC(TIMEDIFF((SELECT MAX(t_2.colst_timestamp) FROM tbl_collo_status t_2 WHERE t_2.colst_nve_id = 
nve_nr.nve_id AND t_2.colst_status_code IN (20,25,91)), (SELECT MAX(t_1.colst_timestamp) FROM tbl_collo_status t_1 
WHERE t_1.colst_nve_id = nve_nr.nve_id AND t_1.colst_status_code IN (10, 11, 76)))) AS timediff_xy,

TIME_TO_SEC(TIMEDIFF((SELECT MAX(t_3.colst_timestamp) FROM tbl_collo_status t_3 WHERE t_3.colst_nve_id = 
nve_nr.nve_id AND t_3.colst_status_code IN (30,99,104)), (SELECT MAX(t_2.colst_timestamp) FROM tbl_collo_status t_2 
WHERE t_2.colst_nve_id = nve_nr.nve_id AND t_2.colst_status_code IN (20,25,91)))) AS timediff_yz,

TIME_TO_SEC(TIMEDIFF((SELECT MAX(t_3.colst_timestamp) FROM tbl_collo_status t_3 WHERE t_3.colst_nve_id = 
nve_nr.nve_id AND t_3.colst_status_code IN (30,99,104)), (SELECT MAX(t_1.colst_timestamp) FROM tbl_collo_status t_1 
WHERE t_1.colst_nve_id = nve_nr.nve_id AND t_1.colst_status_code IN (10, 11, 76)))) AS timediff_xz,

CAST(SUBSTR(nve_nr.nve_nvenr, 1, 3) AS UNSIGNED INTEGER) AS filter_collonr,

MAX(t_5.colst_status_code) AS filter_last_status,

sdg.sdg_kunde AS filter_kunden_id,

sdg.sdg_id AS filter_sdg_id,

nve_nr.nve_id AS filter_nve_id,

adr_emp.adr_id AS filter_adr_emp_id

FROM tbl_nve_nr nve_nr 

LEFT JOIN tbl_sendungen sdg ON nve_nr.nve_sdg_id = sdg.sdg_id

LEFT JOIN tbl_adressen adr_emp ON adr_emp.adr_id = sdg.sdg_adr2_id

INNER JOIN tbl_collo_status t_5  ON t_5.colst_nve_id = nve_nr.nve_id

GROUP BY collonr, emp_adr, status_x, status_y, status_z, timediff_xy, timediff_yz, timediff_xz, filter_collonr, filter_kunden_id, filter_sdg_id, filter_nve_id, filter_adr_emp_id

Problem is that your query has a LOT of co related sub queries and these generally perform badly.

What may be the best solution is to replace all the co related sub queries with joins against other views (with those views containing the selects)

EDIT - splitting off the other subselects :-

Create the views:-

CREATE ALGORITHM = UNDEFINED VIEW vw_MaxTimeStamp_10_11_76 AS SELECT t_1.colst_nve_id, MAX(t_1.colst_timestamp) AS MaxTimestamp FROM tbl_collo_status t_1 WHERE t_1.colst_status_code IN (10, 11, 76) GROUP BY t_1.colst_nve_id

CREATE ALGORITHM = UNDEFINED VIEW vw_MaxTimeStamp_20_25_91 AS SELECT t_1.colst_nve_id, MAX(t_1.colst_timestamp) AS MaxTimestamp FROM tbl_collo_status t_1 WHERE t_1.colst_status_code IN (20,25,91) GROUP BY t_1.colst_nve_id

CREATE ALGORITHM = UNDEFINED VIEW vw_MaxTimeStamp_30_99_104 AS SELECT t_1.colst_nve_id, MAX(t_1.colst_timestamp) AS MaxTimestamp FROM tbl_collo_status t_1 WHERE t_1.colst_status_code IN (30,99,104) GROUP BY t_1.colst_nve_id

CREATE ALGORITHM = UNDEFINED VIEW vw_MaxStatusCode AS SELECT colst_nve_id, MAX(t_5.colst_status_code) AS MaxStatusCode FROM tbl_collo_status t_5 GROUP BY colst_nve_id

Then your main view can be simplified to something like this (afraid not tested so there may be a few typos)

CREATE OR REPLACE VIEW view_test AS

SELECT nve_nr.nve_nvenr AS collonr,
CONCAT (adr_emp.adr_name1, "\n", adr_emp.adr_name2, "\n", adr_emp.adr_strasse, "\n", adr_emp.adr_land , " -     ", adr_emp.adr_plz, " ", adr_emp.adr_ort) AS emp_adr,
UNIX_TIMESTAMP(vw_MaxTimeStamp_10_11_76.MaxTimestamp) AS status_x,
UNIX_TIMESTAMP(vw_MaxTimeStamp_20_25_91.MaxTimestamp) AS status_y,
UNIX_TIMESTAMP(vw_MaxTimeStamp_30_99_104.MaxTimestamp) AS status_z,
TIME_TO_SEC(TIMEDIFF(vw_MaxTimeStamp_20_25_91.MaxTimestamp, vw_MaxTimeStamp_10_11_76.MaxTimestamp)) AS timediff_xy,
TIME_TO_SEC(TIMEDIFF(vw_MaxTimeStamp_30_99_104.MaxTimestamp, vw_MaxTimeStamp_20_25_91.MaxTimestamp)) AS timediff_yz,
TIME_TO_SEC(TIMEDIFF(vw_MaxTimeStamp_30_99_104.MaxTimestamp, vw_MaxTimeStamp_10_11_76.MaxTimestamp)) AS timediff_xz,
CAST(SUBSTR(nve_nr.nve_nvenr, 1, 3) AS UNSIGNED INTEGER) AS filter_collonr,
vw_MaxStatusCode.MaxStatusCode AS filter_last_status
sdg.sdg_kunde AS filter_kunden_id,
sdg.sdg_id AS filter_sdg_id,
nve_nr.nve_id AS filter_nve_id,
adr_emp.adr_id AS filter_adr_emp_id
FROM tbl_nve_nr nve_nr 
LEFT JOIN tbl_sendungen sdg ON nve_nr.nve_sdg_id = sdg.sdg_id
LEFT JOIN tbl_adressen adr_emp ON adr_emp.adr_id = sdg.sdg_adr2_id
INNER JOIN tbl_collo_status t_5  ON t_5.colst_nve_id = nve_nr.nve_id
LEFT OUTER JOIN vw_MaxTimeStamp_10_11_76 ON vw_MaxTimeStamp_10_11_76.colst_nve_id = nve_nr.nve_id
LEFT OUTER JOIN vw_MaxTimeStamp_20_25_91 ON vw_MaxTimeStamp_20_25_91.colst_nve_id = nve_nr.nve_id
LEFT OUTER JOIN vw_MaxTimeStamp_30_99_104 ON vw_MaxTimeStamp_30_99_104.colst_nve_id = nve_nr.nve_id
LEFT OUTRER JOIN vw_MaxStatusCode ON vw_MaxStatusCode.colst_nve_id = nve_nr.nve_id