is it possible to make serverside ad requests instead of in-browser using a headless browser?

Question

The motivation for this idea is that browser ad requests are often painfully slow and can block page rendering, not to mention consume the browser javascript thread. If a page loads several ads, that's a lot of unreliable, blocking third party calls. By using an event driven headless server side browser, would it be possible to make those requests and pass the ad assets up to the browser all at once? In particular, would this work with doubleclick, whose ad networks often make a series of requests that document.write script tags that then fetch more script, may load a flash swf that then requests some additional gifs? It sounds like a bit much, but ads are the devil, and if that can be offloaded from the client, it would improve sitespeed tremendously, since a server solution can make ad requests and process javascript in parallel.

EDIT: Responding to @dthorpe:

All good points to consider.

As for ad serving latency, a server side solution should help for the following reasons:

First, multiple ad requests can be made in parallel. While there are methods of loading resources in parallel in the browser, there is no good solution I know of that is completely non-blocking (i.e. both in terms of page rendering AND with respect to the onload event), unless you lazy load all your ads. I'm looking for a way to load ads in parallel and in a non-blocking fashion while the rest of the page is loading.

Second, and perhaps more significantly, loading doubleclick ads usually involves loading multiple intermediate scripts. Javascript in the browser is single threaded. This directly implies that my page's javascript will block the ad-javascript, slowing down the ad impression. The ad's javascript, in turn will block my page's javascript, slowing down my page. I may have other third party calls in play, for tracking or other purposes. I also have multiple ads on the page, all helping slow down page rendering and javascript execution. By offloading the ad-loading to the server, my page's javascript will not compete with the ad's. The latency contributed by the ads will be a function of one ad (the longest one), but the loading behavior for one ad will not be influenced by the other ads or my page, and in theory should be faster as a result.

The goal is to disassociate the ads from the base page in such a way that the ads do not block page rendering, the page onload event, or slow down the page or each other. I don't know of any client side solution that can achieve perfect disassociation as long as in-browser javascript is single threaded. While long-loading ads are being fulfilled via javascript, the rest of the page has to wait.

I am aware of various lazy loading mechanisms, some involving iframes, and some that override document.write to accommodate deferred loading, but none of these have had satisfactory outcomes, purely due to the serial behavior of interfering javascript execution.

I wonder if I can get the benefits of server side ad loading using java applets in the browser and avoid the cookie/preloading situation you mentioned? I am actually considering these non-conforming approaches due to the serious degradation we see with doubleclick ads. The way those ads kill page load speed is the true abomination :)

Answer 1

In order to fully represent the client machine accurately, your headless server browser would need to be able to present the browser cookies residing on the client for the ad network domain(s). This will be difficult since your server's domain name won't be in the ad network domain(s), so the client browser won't give you the ad network cookies.

You could probably capture the ads in your server browser to "pre-load" them on behalf of the client, but without the client's browser cookies you would lack continuity and tracking that the ad networks rely heavily on. The ads selected for display by the ad networks will be based on what pages your server browser has viewed on behalf of the client, not what the client has viewed.

Also: If loading the ad content is what is blocking the rendering of pages in the client browser, how is rendering the same pages on your server going to speed anything up? Your server browser will suffer from the same network and script lag as the client browser, and the client will still be sitting there waiting for "something", but will show nothing until the page has fully loaded and has been processed in the server browser and relayed down to the client browser? Moving the problem "upstream" doesn't solve anything.

Finally: even though your intentions may be honorable (to improve client browsing experience), the ad networks will quite likely consider your server browser ad "preloader" an abomination and a violation of their ad network terms of service precisely because it interferes with their ability to track the browsing activity of the actual client.

Answer 2

Using a java plugin on client side would likely be even more wasteful of the client machine's resources, as the Java plugin must be loaded into the browser which can often take several seconds.

Keep in mind that with the regularity of Java 0-day exploits in the wild (http://java-0day.com/). The Java plugin is recommended to be disabled in all web browsers. Macs now come with Java plugins disabled by default. Also, no Java plugins exist on mobile devices such as smartphones and tablets. So you would be unable to deliver advertising to anything beside desktop browsers, and likely only the desktop browsers used outside of a corporate setting.

If lazy loading ads is not acceptable, you will likely have to accept the loading times of DoubleClick as they stand. The ad networks will not take kindly to the interference from a contractual perspective, and if you ever plan to sell your ad inventory directly, you will encounter resistance from the advertisers themselves as they attempt to deliver the in-vogue rich media ads. (videos, expandable ads, interstitials, etc)

You'll end up spending more time attempting to get your ad emulation layer perfect for every new ad type that comes along, than the amount you will ever make from the cost per thousand rates on the display advertising.

Perhaps you should reconsider if ads even make sense, or if your business model for monetizing your website needs adjustment. Display advertising has become a commodity market, and the cost of building this emulation layer in your time appears higher than the benefit of increased page speed, and the slightly higher impressions that page speed might bring.

Hats off to you for caring about your end users though, DoubleClick advertising requests can be a real drag on page loading times. I'd love for someone to come along and provide a way to defer ad loading in a robust and reliable way.