Scraping with Meteor.js

Question

Can I scrape with meteor.js? Just discovered cheerio which works excellent combined with request. Can I use these with meteor, or is there something similar?

Do you have an working example?

Answer 1

Of course! Its hard to imagine what meteor can't do! First you need something to handle the remote http requests. In your meteor directory in the terminal run meteor add http to add the Meteor.http package, also npm install cheerio (have a look at another SO question on how to install npm modules to see exactly where to install external npm modules.

Here is an example that might help you out a bit, it scrapes the current time.

Server js

require = __meteor_bootstrap__.require; //to use npm require must be exposed.
var cheerio = require('cheerio');

Meteor.methods({
    getTime: function () {
        result = Meteor.http.get("http://www.timeanddate.com/worldclock/city.html?n=136");
        $ = cheerio.load(result.content);
        CurrentTime = $('#ct').html();
        return CurrentTime;
    }
});

Client side script:

Meteor.call("getTime", function(error, result) {
    alert("The current time is " + result); 
});

I hope this is helpful. amongst with Cheerio there are also other node frameworks such as node.io

Answer 2

The following code is used in this project to scrape a tweetstorm:

if (Meteor.isClient) {

  Meteor.call('getTweets', function (error, result) {
    if (error) {
      console.log("error", error);
    };

    Session.set("tweets", result);
  });

  Template.tweets.helpers({
    rant: function () {
      return Session.get("tweets");
    }
  });

}

Server side

  if (Meteor.isServer) {
      Meteor.startup(function () {
        var cheerio = Meteor.npmRequire('cheerio');

    Meteor.methods({
      getTweets: function () {
        result = Meteor.http.get("https://twitter.com/Royal_Arse/status/538330380273979393");
        $ = cheerio.load(result.content);
        var body = $('#stream-items-id > li:nth-child(n) > div > div > p').text();
        return body;
      },

    })

  });
}

Answer 3

You can have a look to http://casperjs.org/ which is very useful. You can also do screenshots, automated test, etc...

Answer 4

now you should use meteorhacks npm package https://github.com/meteorhacks/npm and require this with :

var cheerio = Meteor.npmRequire('cherio');

worked for me :)