Programmatic access to HTTP traffic after CLI call to page (via wget, or urllib2, etc.)

Question

Maybe an odd question, but I'm looking to capture the same data you find in Firebug's Net panel after loading a page (e.g. all the subsequent HTTP calls to resources needed to fully load the page). However, I want to get that data in an automated way, rather than via Firefox / Firebug.

Say I have 100 URL's I want to loop through and for each one grab the many HTTP requests required for each of those pages - all that same data you see in Firebug's Net panel.

Any thoughts?

Just to add some detail to the accepted solution below, PhantomJS works for this described use case. One of the examples on the Quick Start page, in fact, does the trick:

var page = require('webpage').create();
page.onResourceRequested = function(request) {
  console.log('Request ' + JSON.stringify(request, undefined, 4));
};
page.onResourceReceived = function(response) {
  console.log('Receive ' + JSON.stringify(response, undefined, 4));
};
page.open(url);

the onResourceReceived callback returns a JSON object for each resource; one of the contained elements is "url". A simple grep for that, or other way to isolate it as you see fit, and you can accomplish what I describe above.

Answer 1

You should use a headless web browser if you want to see all the extra requests made by a particular page (CSS, JS, AJAX calls, etc.). PhantomJS is a good choice.

Answer 2

You could use Selenium IDE to parse the page and check for the URLs delivering the advert content. To do so you'll need to record what you do manually to create a script you can then run automatically.