Archiving a mongodb collection

Question

I'm using java Spring and spring data for mongodb.

I have a collection that needs to contain only documents from the last 3 months but all the documents should be saved in some way (maybe expoet to a file?). I'm looking for solution but all i can find talks about full DB backup.

What is the best way to keep the collection updated to only the last 3 months? (weekly cron?) How to save the collection archive? I think mongodump is an overkill.

Answer 1

Both mongoexport and mongodump support a -q option to specify a query to limit the documents that will be deleted. The choice for either is more of a function of what format you'd like the data to be stored in.

Let's assume that you have a collection with a timestamp field. You could run either one of these (filling in the required names and times in the angle brackets):

mongoexport -d <yourdatabase> -c <yourcollection> -q "{ timestamp: { \$gt: <yourtimestamp>}}" -o <yourcollection_export_yourtimestamp>.json
mongodump -d <yourdatabase> -c <yourcollection> -q "{ timestamp: { \$gt: <yourtimestamp>}}" 

And then delete the old data.

Alternatively you could take periodic snapshots via cron with either method on a collection with a ttl index so that you don't have to prune it yourself - mongodb will automatically delete older data:

db.collectioname.ensureIndex( { "createdAt": 1 }, { expireAfterSeconds: 7862400 } )

This will keep deleting any document older than 91 days based on a createdAt field in the document

http://docs.mongodb.org/manual/tutorial/expire-data/

Answer 2

With mongoexport you can backup a single collection instead of the whole database. I would recommand a Cron-Job (like you sad) to export the data and ceep the database limited to the Documents of the last 3 months my removing oder documents.

mongoexport -d databasename -c collectionname -o savefilename.json