DATE_SUB and INTERVAL calculation may be wrong

Question

This may be too localized, but hoping there may be some information out there that I simply haven't been able to find.

Background: We have a system with two separate Java daemons; one that creates data and inserts into a database, and another that grabs the latest sets of that data (no more than 1 hour old), and pushes to the client.

Problem: The data being grabbed from this table is occasionally greater than 1 hour old. The time difference can be anywhere between 2 hours to several months old.

Current Query: Obtaining the 1-hour data is a two-step process. First, we "reserve" a set of records by setting a random negative number:

UPDATE tracking_records as target
  JOIN 
    (SELECT tracking_records.id, `set`, unit_id, tracking_records.record_time 
     FROM tracking_records 
     WHERE `set` IS NULL and record_time > DATE_SUB(NOW(), INTERVAL 1 HOUR) 
     ORDER BY record_time DESC LIMIT 48) source 
  ON source.id = target.id
SET target.`set` = -1371504452;

Note: The -1371504452 is an example value; it is generated randomly in Java. The LIMIT 48 remains the same for each query.

And then we simply select the columns that contain that random set value.

This is the structure of the tracking_records table:

+-------------+---------------------+
| Field       | Type                |
+-------------+---------------------+
| id          | bigint(20) unsigned |
| unit_id     | int(11)             |
| record_time | datetime            |
| latitude    | int(11)             |
| longitude   | int(11)             |
| created_at  | datetime            |
| updated_at  | datetime            |
| set         | int(11)             |
+-------------+---------------------+

As you can see, the query should only match records where the set columns is null AND where the record time is at most 1 hour ago.

As mentioned, I detect time differences of a few hours, up to several months (the oldest I have detected so far is 2013-09-23 11:01:08). This does not make sense to me, given the time restrictions of the WHERE clause.

We are using mysql version 5.5.29-0ubuntu0.12.04.2

Question: I am quite confused, and I am wondering if there is a bug, or some other issue that would cause the time calculation to fail so drastically, and randomly. Either that, or there is a problem with the query itself that I am simply not seeing.

Has anyone observed a problem with incorrect time comparisons, the DATE_SUB() function, or the INTERVAL calculation, in this version of MySQL that might explain the anomalous times I'm seeing?

Answer 1

We're going to assume that id is the PRIMARY KEY of the table, or that it's a UNIQUE KEY.

The most likely suspect is that the "random" number is not unique. Likely, Java is using a psuedo-random number generator, which actually generates a repeatable sequence of numbers from the same seed.

I'd recommend a number generator that generates unique values, rather than random values, since the same random value is bound to reappear at some point.

If that ever happens, the set of 48 rows gets updated with the "random" value, but a subsequent query to retrieve those rows, based on that "random" value is also going to pick up rows that were previously updated to the same "random" value.

---

With that said, with the SQL statement you show, there might be a small window for a "collision" if two of these statements run at the exact same time. (That inline view gets materialized into a temporary MyISAM table, then the outer query runs. I'm not sure if the inline view query obtains an exclusive or intent-exclusive lock on the rows in tracking records. But the symptom of a collision there would be a subsequent query not finding 48 rows with the specified set value, because another query over-wrote the value with it's own value.

---

It's very unlikely you've discovered a bug in MySQL DATETIME handling. That code has been exercised for years, and it's unlikely a bug has been introduced there. (I've used it for better than a decade (versions 3.23, 4.x, 5.1, 5.5) and I've never encountered a bug with DATETIME. (I have been introduced, rudely, to documented behavior. But never encountered a real bug.)

Note that the DATE_SUB function is not necessary; you could get the same result with:

 NOW() - INTERVAL 1 HOUR

Note that NOW() gets evaluated at the start of the statement or block; we prefer to use it over other functions primarily because it's replication safe. The value of evaluated at the time of original execution is preserved in the binary log, so that the same value can be applied on the replica database(s). Other functions that return the current time get re-evaluated each time the function is called, and the value is not preserved for replication.