Indexing 20M of records with python and mongoDB

Question

I would like to mention about my little project, and if I'm on track. I need work with all articles from Medline (http://www.nlm.nih.gov/bsd/licensee/2011_stats/baseline_doc.html). For those are not familiar with Medline database, I add a little information:

• There are approx. 20.000.000 of records (83.4 GB disk space), each has many fields and subfields.
• You can download this DB (with license) in XML format.
• These 20M of records are distributed in 653 files.
• Every file has one MedlineCitationSet, and this a set of records (MedlineCitation's).

I want to process these records and get information such as title, abstract... Then I have thought to index these files (or records) with python and mongodb. And I have one option:

I have created a medline parser and for each record is created a JSON entry for mongoDB, and after indexing by pubmedID. Then I can create a function likes get_abstract('pubmedID'):string.

My questions are:

• Is it a good idea? (XML parsing --> JSON --> insert and indexing!)
• Can I use GridFS and get chunks equivalents to records for each file? How?
• Do you know other way?

Answer 1

Is it a good idea? (XML parsing --> JSON --> insert and indexing!)

Does it work? JSON tends to be smaller than XML, this is probably a good change.

Can I use GridFS and get chunks equivalents to records for each file? How?

GridFS is generally for storing static files like images or video. MongoDB can support very large documents (16MB == several novels). The text for a medline article is probably going to be much smaller than this.

If you are just planning to store text, then you should not need any GridFS. If you want to store images, then GridFS is useful for doing this.

Please read up on GridFS here. Note that GridFS is just a specification. Everything is still stored inside of regular MongoDB collections.

---

PS: it looks like pubmedID is a unique key. You can save space by overriding the _id with pubmedID when you create the document.

i.e.: collection.insert({"_id": xml_obj.pubmedID, "text" : xml_obj.article_text})