Wiring up Selenium for Continuous Integration on Windows

Question

Background

I have been working towards automating my Selenium test suite that I developed under Selenium 1.0 (RC) and wiring this up to a continuous integration system (TeamCity in this case) on my build machine. I quickly ran into the fact that I could run the suite manually on the build machine but it failed when run by TeamCity.

For completeness, here is what the selenium server log looks like when successful:

18:26:09.025 INFO [12] org.openqa.selenium.server.SeleniumDriverResourceHandler - Command request: getNewBrowserSession[*iexplore, http://dev2mx:8080/, ] on session null
18:26:09.041 INFO [12] org.openqa.selenium.server.BrowserSessionFactory - creating new remote session
18:26:09.072 DEBUG [12] org.openqa.selenium.server.browserlaunchers.BrowserLauncherFactory - Requested browser string '*iexplore' matches *iexplore
18:26:09.072 INFO [12] org.openqa.selenium.server.BrowserSessionFactory - Allocated session 1c8363f1edbc40b9b251e3bf4bd3d74f for http://portaldev2:80/, launching...
18:26:09.103 DEBUG [12] org.openqa.selenium.server.browserlaunchers.ResourceExtractor - Extracting /core to C:\DOCUME~1\devadmin\LOCALS~1\Temp\5\customProfileDir1c8363f1edbc40b9b251e3bf4bd3d74f\core
18:26:09.244 INFO [12] org.openqa.selenium.server.browserlaunchers.HTABrowserLauncher - Launching Embedded Internet Explorer...
18:26:09.260 DEBUG [12] org.openqa.selenium.browserlaunchers.locators.BrowserLocator - Discovering Internet Explorer...
18:26:09.260 DEBUG [12] org.openqa.selenium.browserlaunchers.locators.BrowserLocator - Checking whether Internet Explorer launcher at :'C:\Program Files\Internet Explorer\iexplore.exe' is valid...
18:26:09.260 DEBUG [12] org.openqa.selenium.browserlaunchers.locators.BrowserLocator - Discovered valid Internet Explorer launcher  : 'C:\Program Files\Internet Explorer\iexplore.exe'
18:26:09.275 INFO [12] org.openqa.selenium.server.browserlaunchers.HTABrowserLauncher - Launching Internet Explorer HTA...
18:26:09.291 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for window 'null' local frame 'null' for 1800 more secs
18:26:09.291 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for condition for 1000 more ms
18:26:10.307 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - got condition? : false
18:26:10.307 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for window 'null' local frame 'null' for 1799 more secs
18:26:10.307 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for condition for 1000 more ms
18:26:10.994 DEBUG [13] org.openqa.jetty.http.HttpConnection - new HttpConnection: Socket[addr=/127.0.0.1,port=3255,localport=4444]
18:26:11.010 DEBUG [13] org.openqa.jetty.http.HttpConnection - REQUEST from SocketListener0@0.0.0.0:4444:

The failure mode manifested in the log by never getting to those last two lines. The waiting lines just above kept waiting and waiting and waiting:

12:19:16.191 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for window 'null' local frame 'null' for 1800 more secs
12:19:16.191 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for condition for 1000 more ms
12:19:17.191 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - got condition? : false
12:19:17.191 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for window 'null' local frame 'null' for 1799 more secs
12:19:17.191 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for condition for 1000 more ms
12:19:18.207 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - got condition? : false
12:19:18.238 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for window 'null' local frame 'null' for 1798 more secs
12:19:18.238 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for condition for 1000 more ms
12:19:19.254 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - got condition? : false
12:19:19.254 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for window 'null' local frame 'null' for 1797 more secs
12:19:19.254 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for condition for 1000 more ms
12:19:20.269 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - got condition? : false

I thought of a variety of angles to examine the problem, starting with upgrading from Selenium RC to Selenium 2.0b2 and just swapping out the server jar and the library references. Same result.

Next I thought about further changeover to WebDriver and Selenium 2.0 by replacing

selenium = new DefaultSelenium("localhost", port, browserType, pageToOpen);

with

selenium = new WebDriverBackedSelenium(driver, pageToOpen);

What I encountered immediately is that WebDriver is more strict about frames! My tests failed until I specified more rigorously what frame the subsequent objects were supposed to be in. This rework is still in progress so no conclusion from it yet regarding my original problem, but I suspect I know the answer, because...

In parallel, I started reading about Selenium and headless configuration and continuous integration. It seems that a continuous integration server must, almost by definition, run everything headless, but selenium needs a display. Conflict. (I do not want to use HtmlUnitDriver because I want to test against real browsers; not to mention that HtmlUnitDriver does not exist yet for .NET in 2.0b2!) I read about xvfb as a solution for linux, but I am on Windows Server 2003. The most likely solution seems to be to use Selenium Grid, which could launch real browsers on other machines. Unfortunately, Grid does not yet exist for Selenium 2, so that means sticking with Selenium RC for the time being. Not ideal, but I could live with that.

Questions

I have not yet looked at Grid so probably a naive question: would Grid (running headless in TeamCity) have any problems opening real browsers on remote machines?

Is there any way to wire up Selenium 2 (having no Grid support) with continuous integration?

Answer 1

naive answers are:

Grid should not have any problem with opening real browsers on remote machine.

You should be able to use Selenium 2 with grid support, it should be same as using Selenium 1

n.b. I should have just read "Questions" and answered ;-)

Answer 2

I have run into a similar problem with regards to the endless error messages such as:

12:19:16.191 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for window 'null' local frame 'null' for 1800 more secs
12:19:16.191 DEBUG [12] org.openqa.selenium.server.FrameGroupCommandQueueSet - waiting for condition for 1000 more ms

As it turns out, running the build agent as a windows service will cause the process for the browser to run, but not an actual browser with which selenium can interact. If you stop the service, navigate to the build agent's bin directory, and launch the build agent manually (agent.bat start), you should see selenium tests functioning against your chosen browser when built by TeamCity.

Hope that helps!