How to capture all requests made by page in webdriver? Is there any alternative to Browsermob?

Question

I am using Selenium2/WebDriver to test my web applications. All the tests are written in Java and run with Maven.

While opening a page with webdriver I'd like to capture all the requests made by page (images, js and css files, etc). I use this data mainly for two reasons

• checking for 404 (and other errors) in calls
• checking if analytics code is working (checking if it's sending proper requests)

Depending on the project I use either Firebug with Netexport or Browsermob proxy. In both cases I can easily obtain a HAR (Html ARchive) file, parse it and extract the data I want.

Here's the problem: I am not happy with neither of these solutions. I have especially problems with getting HAR file when a page contains video that is being loaded too long. I am looking for something more stable.

So, the questions are:

Is there any alternative to Browsermob? I know about FiddlerCore but it's a .NET library and my tests are written in Java. I've also heard about Ajax DynaTrace and I know that there is some way to integrate it with Selenium but the documentation I found was for Selenium-RC not WebDriver.

Is there any way to integrate DynaTrace with WebDriver or use FiddlerCore with Java?

Is there any other way to achieve the goals I mentioned? I am looking for a proxy that I can easily control from my code. Exporting data to HAR would be a great plus.

Answer 1

I found a google groups discussion on the topic. These links look like promising alternative to Browsermob:

• A Selenium CaptureNetworkTraffic Example in Java
• HOWTO: Collect WebDriver HTTP Request and Response Headers
• Automating the Capture of Web Timings with Selenium 2

Answer 2

There is an alternative with firefox ver 42+, there is addon called Firefox HarExport

File harExportApi = new File(System.getProperty("user.dir")
                     + "/src/main/resources/firebug/harexporttrigger-0.5.0-beta.7.xpi");

netExportProfile.addExtension(harExportApi);
netExportProfile.setPreference("extensions.netmonitor.har.enableAutomation", true);
    netExportProfile.setPreference("extensions.netmonitor.har.contentAPIToken", "test");
    netExportProfile.setPreference("extensions.netmonitor.har.autoConnect", true);

cap.setCapability(FirefoxDriver.PROFILE, netExportProfile);

and running following script will give us all the request responses

 String getHarLogScript = "var options = {\n" +
                "    token: \"test\",\n" +
                "    getData: true,\n" +
                "    title: \"my custom title\",\n" +
                "    jsonp: false,\n" +
                "  };\n" +
                "\n" +
                "  HAR.triggerExport(options).then(result => {\n" +
                "    var har = JSON.parse(result.data);\n" +
                "\n" +
                "    // Use performance.timing to provide onContentLoad\n" +
                "    +
                "     +
                "    var t = performance.timing;\n" +
                "    var pageTimings = har.log.pages[0].pageTimings;\n" +
                "    pageTimings.onContentLoad = t.domContentLoadedEventStart - t.navigationStart;\n" +
                "    pageTimings.onLoad = t.loadEventStart - t.navigationStart;\n" +
                "\n" +
                "    window.HarEntries=har.log.entries\n" +
                "\n" +
                "    console.log(\"HAR log (\" + result.data.length + \") \", har.log);\n" +
                "  }, err => {\n" +
                "    console.error(err);\n" +
                "  });"

LOG.info("Loading HAR log entries object into browser HarEntries object");
SeleniumUtils.executeScript(driver, getHarLogScript);

harEntries = ((List<Object>) SeleniumUtils.executeScript(driver, "return window.HarEntries"));

Answer 3

I've been working recently on this kind of proxy. Project is pretty fresh, I'm still working on documentation but it may be worth checking. Sources and examples are here

1. Add dependency to your project

    <dependency>
       <groupId>com.moxproxy</groupId>
       <artifactId>moxproxy.core</artifactId>
       <version>1.0.2</version>
    </dependency>

2. Start proxy

    MoxProxy proxy = LocalMoxProxy.builder()
                .withPort(89)
                .build();
    proxy.startServer();

3. Setup selenium webdriver to use proxy on localhost with port 89 and run test

4. Collect traffic

    List<MoxProxyProcessedTrafficEntry> requestTraffic = proxy.getAllRequestTraffic();
    List<MoxProxyProcessedTrafficEntry> responseTraffic = proxy.getAllResponseTraffic();

Beside collecting traffic proxy provides posibility to modify requests and responses - details on github