Insert multiple records and/or update specific fields and return only new records inserted (MongoDB)

Question

Hi I have a collection as follows

var articles = [
                {
                  "title": "Article title1",
                  "content": "Article ... content......... 1. ",
                  "url": "http://matt.wordpress.com/article/X",
                  "last_fetched_time": new Date();
                },
                {
                  "title": "Article title2",
                  "content": "Article ... content......... 2. ",
                  "url": "http://matt.blogger.com/article/Y",
                  "last_fetched_time": new Date();
                }
            ];
db.collection('articles').insert(articles, {safe:true}, function(err, result) {}); //articles collection created

I want to fetch blog feeds from multiple endpoints in parallel periodically and add new articles into the collection and update the last fetched date-time field of the existing articles in the collection. If I'm not asking too much I also want the upsert's callback returns only the new articles inserted.

//fetch articles periodically
fetchArticles = function(req, res) {
async.parallel([
    //fetch word press endpoint
        //get "title", "content", "url"
        //set last_fetched_time with new Date();

    //fetch blogger endpoint
        //get "title", "content", "url"
        //set last_fetched_time with new Date();
], 
function(err, results) {
    //merge results[0] and results[1] in a batch =[]
    //if the article url is not already in the collection, insert article into the articles collection
    //if the article url is found in the collection, update article because last_fetched_time changed 
    //finally return only new inserted articles, not updated ones
    db.collection('articles').update(batch, {safe:true, upsert : true}, function(err, result) { 

        //result = only new articles inserted
    });
});

}

url field should be unique and I did

db.articles.ensureIndex({"url":1}, {unique: true, sparse:true, dropDups: true});

The problem is this code doesn't insert new articles

Answer 1

You seem to have your functions mixed around even though I clearly see what you are trying to do.

Your batch you are passing in contains an array of documents that you want to insert/update. The problem is that this functionality is only available to the insert method.

Since you are using the update method, the option to pass in an array of documents for batch processing is not available. Updates with upsert set as you have done are intended to be issued with the primary arguments of a selector and a single 'document'. The idea being that where the selector matches an existing document, that document is updated with the details in document. In the event the match is not found, then a new document is then inserted.

Additionally as you have not used, there is the multi option that can be applied. It's purpose is that when the selector matches more than one document, then the changes are applied to all of the matching documents. Unspecified the behavior is considered to be false and only the first matching document found will be updated.

It would seem that though you wish to have this accompanied by a batch processing functionality, it does not presently exist. There is a JIRA you can follow / support for this.

https://jira.mongodb.org/browse/SERVER-2172

Refer to the links for the functions in the documentation, which explains all the available arguments and options. Also see the shell documentation for a detailed explaination of the options:

http://docs.mongodb.org/manual/reference/method/db.collection.update/