Elasticsearch query by date returning incorrect results

Question

So, I have a series of documents that are stored as pure Json in a NoSQL database (Couchbase). These documents include a "Created" field which contains a DateTime value. I'm performing the following query through Elasticsearch and it is returning results that I would not expect it to return.

Query:

"query":
{ 
"bool": 
{
    "must": [
    {
        "range": 
        {
            "couchbaseDocument.doc.Created": 
            { 
                "from":"2014-03-27T00:00:00.0000000",
                "to": "2014-03-27T23:59:00.0000000"
            }
        }
    },
    {
        "term":
        {
            "couchbaseDocument.meta.expiration": "0"
        }
    }],
    "must_not": [ ],
    "should": [ ]
}, 
from: 0, 
size:25000  
}

I would expect this query to only return results with a Created date some time during 3/27. However, I'm getting some results from 3/26. For instance, it is returning one document with the following created field:

"Created": "2014-03-26T21:40:26.2856631-04:00"

Why is this document being returned in the result set?

Answer 1

Your query is searching for date/times in the GMT timezone (aka UTC; aka Zulu time), which is to say that there is an implicit +00:00 at the end of them.

"from" : "2014-03-27T00:00:00.0000000+00:00",
"to"   : "2014-03-27T23:59:00.0000000+00:00"

GMT is a pretty standard timezone for storing date/times, but it appears that you have documents that are stored in a different timezone (they appear to be either a late EDT or AST, which is the Atlantic Time Zone), which is -04:00 or 4 hours behind GMT. Therefore, when it it is midnight on March 27, 2014 in GMT, it is still 8 PM on March 26, 2014 in AST (8 PM is the 20th hour of the day).

In essence, when you see -04:00, then you need to add 04:00 to the time, which is 4 hours and 0 minutes. Once done, then you can drop the timezone marker because:

2014-03-26T21:40:26.2856631-04:00

is the same as

2014-03-27T01:40:26.2856631+00:00

which is the same as

2014-03-27T01:40:26.2856631

because 21 + 04 = 25, and hour 25 of a day is hour 01 of the next day (25 - 24 = 01, thus giving 1 day and 1 hour; it is noteworthy that 24 - 24 = 00, which is why 00 represents midnight).

To bring it all back to the question: the time returned should be in your results because with respect to GMT, it is within bounds of your search.

For reference, EST is -05:00 and PST is -08:00.