SEO with Node.js's Express

Question

I'm looking at serving content to bots in HTML, and then for all other requests just rendering index.html and letting my Backbone.js app kick in.

Am I best to try and create an array of known bots for Google, Facebook, etc...and then if the user agent isn't in that array, render index.html, or is there something in Node's Express that can say "this is a real user, not a bot of any kind"?

app.use(function(req, res) {
  var ua = req.headers['user-agent'];
  // TODO
  res.render('index.html');
});

Answer 1

Consider prerendering dynamic pages to static HTMLs and serve to bots using express in similar approach you've described.
You can use phantomjs and some ready libraries to handle page rendering.

Bear in mind that google dislikes when you serve different thing to bots dramatically, and they do adopt their bots to deal with dynamic content as single-page experience is getting more popular around the web.

Answer 2

We have middleware written for node (expressjs) that lets you determine if the user is a bot, and then forward the request on to a server that serves up the prerendered HTML. The code can be found here: https://github.com/prerender/prerender-node

It works by supporting the _escaped_fragment_ protocol, so we only check the user agent for services that don't support the protocol (facebook, twitter, linkedin, etc).

It's currently built to forward requests to our Prerender.io service, which handles all of the work of turning javascript pages into static HTML, but we also have our phantomjs server open sourced: https://github.com/prerender/prerender. That way you can host everything on your own if you'd like.

This is not considered as cloaking because you're really serving what the user sees. Just don't change the static HTML to be any different than what the user actually sees when visiting those pages and you'll be fine. Our service serves over 1 million pages to search engines every day and none of our users have been penalized for cloaking.

Check it out and let me know if you have any questions.