query optimization to find random sample

Question

I have an sql query to select randomly 1200 top retweeted tweets at least 50 times retweeted and the tweetDate should be 4 days older from 40 million records. The query I pasted below works but It takes 40 minutes, so is there any faster version of that query?

SELECT 
    originalTweetId, Count(*) as total, tweetContent, tweetDate
FROM
    twitter_gokhan2.tweetentities
WHERE
    originalTweetId IS NOT NULL
        AND originalTweetId <> - 1
        AND isRetweet = true
    AND (tweetDate < DATE_ADD(CURDATE(), INTERVAL - 4 DAY))
GROUP BY originalTweetId
HAVING total > 50
ORDER BY RAND()
limit 0 , 1200;


---------------------------------------------------------------

Table creation sql is like:

    CREATE TABLE `tweetentities` (
      `id` int(11) NOT NULL AUTO_INCREMENT,
      `tweetId` bigint(20) NOT NULL,
      `tweetContent` varchar(360) DEFAULT NULL,
      `tweetDate` datetime DEFAULT NULL,
      `userId` bigint(20) DEFAULT NULL,
      `userName` varchar(100) DEFAULT NULL,
      `retweetCount` int(11) DEFAULT '0',
      `keyword` varchar(500) DEFAULT NULL,
      `isRetweet` bit(1) DEFAULT b'0',
      `isCompleted` bit(1) DEFAULT b'0',
      `applicationId` int(11) DEFAULT NULL,
      `latitudeData` double DEFAULT NULL,
      `longitude` double DEFAULT NULL,
      `originalTweetId` bigint(20) DEFAULT NULL,
     PRIMARY KEY (`id`),
  KEY `index` (`originalTweetId`),
  KEY `index3` (`applicationId`),
  KEY `index2` (`tweetId`),
  KEY `index4` (`userId`),
  KEY `index5` (`userName`),
  KEY `index6` (`isRetweet`),
  KEY `index7` (`tweetDate`),
  KEY `index8` (`originalTweetId`),
  KEY `index9` (`isCompleted`),
  KEY `index10` (`tweetContent`(191))
) ENGINE=InnoDB AUTO_INCREMENT=41501628 DEFAULT CHARSET=utf8mb4$$

Answer 1

You are, of course, summarizing a huge number of records, then randomizing them. This kind of thing is hard to make fast. Going back to the beginning of time makes it worse. Searching on a null condition just trashes it.

If you want this to perform reasonably, you must get rid of the IS NOT NULL selection. Otherwise, it will perform badly.

But let us try to find a reasonable solution. First, let's get the originalTweetId values we need.

 SELECT MIN(id) originalId,
        MIN(tweetDate) tweetDate,
        originalTweetId, 
        Count(*) as total
   FROM twitter_gokhan2.tweetentities
  WHERE originalTweetId <> -1 
  /*AND originalTweetId IS NOT NULL We have to leave this out for perf reasons */
    AND isRetweet = true
    AND tweetDate < CURDATE() - INTERVAL 4 DAY
    AND tweetDate > CURDATE() - INTERVAL 30 DAY  /*let's add this, if we can*/
  GROUP BY originalTweetId
 HAVING total >= 50

This summary query gives us the lowest id number and date in your database for each subject tweet.

To get this to run fast, we need a compound index on (originalTweetId, isRetweet, tweetDate, id). The query will do a range scan of this index on tweetDate, which is about as fast as you can hope for. Debug this query, both for correctness and performance, then move on.

Now do the randomization. Let's do this with the minimum amount of data we can, to avoid sorting some enormous amount of stuff.

 SELECT originalTweetId, tweetDate, total, RAND() AS randomOrder
   FROM (
    SELECT MIN(id) originalId,
           MIN(tweetDate) tweetDate
           originalTweetId, 
           Count(*) as total
      FROM twitter_gokhan2.tweetentities
     WHERE originalTweetId <> -1 
     /*AND originalTweetId IS NOT NULL We have to leave this out for perf reasons */
       AND isRetweet = true
       AND tweetDate < CURDATE() - INTERVAL 4 DAY
       AND tweetDate > CURDATE() - INTERVAL 30 DAY  /*let's add this, if we can*/
      GROUP BY originalTweetId
     HAVING total >= 50
   ) AS retweets
  ORDER BY randomOrder
  LIMIT 1200

Great. Now we have a list of 1200 tweet ids and dates in random order. Now let's go get the content.

 SELECT a.originalTweetId, a.total, b.tweetContent, a.tweetDate
   FROM (
          /* that whole query above */
        ) AS a
   JOIN twitter_gokhan2.tweetentities AS b ON (a.id = b.id)
  ORDER BY a.randomOrder

See how this goes? Use a compound index to do your summary, and do it on the minimum amount of data. Then do the randomizing, then go fetch the extra data you need.

Answer 2

You're selecting a huge number of records by selecting every record older than 4 days old....

Since the query takes a huge amount of time, why not simply prepare the results using an independant script which runs repeatedly in the background....

You might be able to make the assumption that if its a retweet, the originalTweetId cannot be null/-1

Answer 3

Just to clarify... did you really mean to query everything OLDER than 4 days???

 AND (tweetDate < DATE_ADD(CURDATE(), INTERVAL - 4 DAY))

OR... Did you mean you only wanted to aggregate RECENT TWEETS WITHIN the last 4 days. To me, tweets that happened 2 years ago would be worthless to current events... If thats the case, you might be better to just change to

 AND (tweetDate >= DATE_ADD(CURDATE(), INTERVAL - 4 DAY))

Answer 4

See if this isn't a bit faster than 40 minutes:

Test first without the commented lines, then re-add them to compare performance impact. (especially ORDER BY RAND() is known to be horrible)

SELECT
    originalTweetId,
    total,
--    tweetContent,  -- may slow things somewhat
    tweetDate
FROM (
  SELECT
    originalTweetId,
    COUNT(*) AS total,
--    tweetContent,  -- may slow things somewhat
    MIN(tweetDate) AS tweetDate,
    MAX(isRetweet) AS isRetweet
  FROM twitter_gokhan2.tweetentities
  GROUP BY originalTweetId
) AS t
WHERE originalTweetId > 0
  AND isRetweet
  AND tweetDate < DATE_ADD(CURDATE(), INTERVAL - 4 DAY)
  AND total > 50
-- ORDER BY RAND()  -- very likely to slow performance,
                    -- test with and without...
LIMIT 0, 1200;

PS - originalTweetId should be indexed hopefully