Zombie.js browser not returning full html?
Question
I'm trying to figure out Zombie.js.
I have this script:
var Browser = require("zombie");
var assert = require("assert");
Browser.visit("http://web.mit.edu", function (e,browser) {
console.log(browser.html());
});
That just visits the page and logs the html, but I'm getting html that doesn't match what the source file I get in a normal browser.
Zombie.js output:
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>MIT - Massachusetts Institute of Technology</title>
<meta name="keywords" content="Massachusetts Institute of Technology, MIT" />
<meta name="description" content="MIT is devoted to the advancement of knowledge and education of students in areas that contribute to or prosper in an environment of science and technology." />
<meta name="robots" content="index,follow,noodp,noydir" />
<meta name="allow-search" content="yes" />
<meta name="language" content="en" />
<meta name="distribution" content="global" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta http-equiv="Expires" content="Tue, 01 Jan 1980 1:00:00 GMT" />
<meta http-equiv="Pragma" content="no-cache" />
<meta http-equiv="cache-control" content="no-store" />
<link rel="canonical" href="http://web.mit.edu" />
<link rel="image_src" href="http://web.mit.edu/img/MIT_logo.gif" />
<link rel="search" type="application/opensearchdescription+xml" href="http://web.mit.edu/opensearch.xml" title="MIT - Massachusetts Institute of Technology" />
<link href="https://plus.google.com/104984516469461796485/" rel="publisher" />
<!-- icons -->
<link rel="shortcut icon" href="/favicon.ico" />
<!-- rss -->
<link rel="alternate" type="application/rss+xml" title="MIT - Home Page News" href="http://feeds.feedburner.com/mit/news-homepage" />
<!-- style sheets - global -->
<link href="styles/style3536.css" rel="stylesheet" type="text/css" media="all" />
<script src="http://dnn506yrbagrg.cloudfront.net/pages/scripts/0011/6778.js" type="text/javascript"></script>
<script type="text/javascript" src="http://www.google-analytics.com/ga.js"></script>
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(
['_setAccount', 'UA-1592615-11'],
['_trackPageview','/'],
['rollup._setAccount', 'UA-31439876-1'],
['rollup._trackPageview', '/mit/www/']
);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
<style type="text/css" media="screen">#flashcontent {visibility:hidden}#flashcontent {visibility:visible}</style>
</head>
<body>
<a href="http://events.mit.edu/">Tuesday, October 23, 2012</a>
</body>
</html>
And the real source is really long, check it out at http://web.mit.edu
What's going on?
Solution
If you use assert
to compare the source code to zombie's result you'll get an error because of the following points:
First of all browser.html()
doesn't give you the doctype.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Second and more important MIT uses javascript files which are loaded after a one second timeout
<script type="text/javascript">
setTimeout(function(){var a=document.createElement("script");
var b=document.getElementsByTagName('script')[0];
a.src=document.location.protocol+"//dnn506yrbagrg.cloudfront.net/pages/scripts/0011/6778.js";
a.async=true;a.type="text/javascript";b.parentNode.insertBefore(a,b)}, 1);
</script>
or asynchronously
var _gaq = _gaq || [];
_gaq.push(
['_setAccount', 'UA-1592615-11'],
['_trackPageview','/'],
['rollup._setAccount', 'UA-31439876-1'],
['rollup._trackPageview', '/mit/www/']
);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
The source code shows you the code to load them whereas zombie
gives you the results
<script src="http://dnn506yrbagrg.cloudfront.net/pages/scripts/0011/6778.js" type="text/javascript"></script>
and
<script type="text/javascript" src="http://www.google-analytics.com/ga.js"></script>
Depending on the content of those scripts they might further change the DOM and zombie shows even more deviations. Unfortunately I'm not aware of a workaround.