How do I compare dates from Twitter data stored in MongoDB via PyMongo?

Question

Are the dates stored in the 'created_at' fields marshaled to Python datetime objects via PyMongo, or do I have to manually replace the text strings with Python Date objects? i.e.

How do I convert a property in MongoDB from text to date type?

It seems highly unnatural that I would have to replace the date strings with Python date objects, which is why I'm asking the question.

I would like to write queries that display the tweets from the past three days. Please let me know if there is a slick way of doing this. Thanks!

Answer 1

you can parse Twitter's created_at timestamps to Python datetimes like so:

import datetime, pymongo
created_at = 'Mon Jun 8 10:51:32 +0000 2009' # Get this string from the Twitter API
dt = datetime.strptime(created_at, '%a %b %d %H:%M:%S +0000 %Y')

and insert them into your Mongo collection like this:

connection = pymongo.Connection('mymongohostname.com')
connection.my_database.my_collection.insert({
    'created_at': dt,
    # ... other info about the tweet ....
}, safe=True)

And finally, to get tweets within the last three days, newest first:

three_days_ago = datetime.datetime.utcnow() - datetime.timedelta(days=3)
tweets = list(connection.my_database.my_collection.find({
    'created_at': { '$gte': three_days_ago }
}).sort([('created_at', pymongo.DESCENDING)]))

Answer 2

1) Test it out and see :-)

2) If you are using pymongo, and you set a field to be equal to a python datetime object, then it will store it in the proper bson format and return datetime objects back to you when you query it. The point of pymongo really is to be working in native python objects. You don't really set dates in the BSON format with $date. It does that under the hood. That being said, I'm not sure how pymongo would know to marshal a string date for you, without the ability to say its a date.

3) If you values are already in mongodb as raw strings, then you would need to convert them yourself on the client side using string formatting: http://docs.python.org/library/datetime.html#strftime-strptime-behavior

Edit: If you want to run a function on mongodb that converts your string dates to date objects:

How do I convert a property in MongoDB from text to date type?