Functional testing for our site

Question

I'm working with Mink / Sahi to write a functional test suite for my site.

I have a set of tests working through them with Firefox and Chrome, which I'm happy with. They're running nightly on our Jenkins box, and working well.

However, because our Jenkins box is a server and Chrome/Firefox are GUI apps, I have had to make the tests run on my desktop PC. This is a pain because it means I have to leave it switched on every night, which is bad for environmental and cost reasons. Plus if it has any problems with power or network or software, then the tests fail.

So I would like some advice on switching the tests to use a headless browser on the Jenkins box itself.

It seems I have three options: Goutte, Zombie and Phantom (unless anyone can recommend another, of course). The following summarises my progress so far:

• Goutte: This is PHP-driven, so would run inside of Mink, removing the need for Sahi. This sounds great, as the Jenkins box has limited resources, so the less I need to install and run on it the better. However, I need to run JS code as part of the tests, and I understand Goutte isn't capable of this. Does that rule it out?

• Zombie: Runs under Node.js. Unfortunately, I haven't been able to make this work at all. I've installed Node, NPM and Zombie, but I can't get Mink to recognise it. Can anyone give me some clearer instructions than the Mink site on how to get this running?

• Phantom: Unfortunately, Mink doesn't have a driver for Phantom, so I'd have to run it via Sahi. As I said, I'd prefer not to have to install Sahi on the Jenkins server, especially as it also needs to run continuously as a server. But it is the only one I've had any success with so far. Running it under Sahi, I can get my tests to run successfully (though not consistenly, which is a worry - it seems to timeout randomly, about one in three times). Can anyone suggest a way to run this without needing Sahi (or any other middle-tier server) installed? Or if I do need Sahi, can anyone tell me how to configure Jenkins to start Sahi at the begining of the test suite and stop it at the end?

I'd really appreciate any advice on how to proceed. None of these options seem to have a clear win, for one reason or another. But functional testing is important, so this must be a solved problem. What is the best solultion for me?

(I know there's also the option of re-writing my scripts in Javascript to talk directly to Zombie or Phantom. I'd prefer not to do that, as when they fail I will still need to see them running in Firefox to see what's going wrong, so a cross-browser interface like Mink is ideal - not to mention the fact that I've already written the all tests in PHP!)

Thanks for any advice. :)

Answer 1

This answer is specifically for

can anyone tell me how to configure Jenkins to start Sahi at the begining of the test suite and stop it at the end?

Using ant you can start Sahi using the following target

<target name="sahitests" description="start the server and run sahi tests">
    <parallel>
        <antcall target="start"/>
        <sequential>
            <waitfor maxwait="3" maxwaitunit="minute" checkevery="100">
                <http url="http://${urlbase}/demo/index.htm"/>
            </waitfor>
            <antcall target="runietests"/>
            <antcall target="stopsahi"/>
        </sequential>
    </parallel>
</target>

<target name="start" description="starts proxy">
    <java classname="net.sf.sahi.Proxy" fork="true">
        <classpath location="lib/sahi.jar">
            <pathelement location="extlib/rhino/js.jar"/>
            <pathelement location="extlib/apc/commons-codec-1.3.jar"/>
            <pathelement location="extlib/license/truelicense.jar"/>
            <pathelement location="extlib/license/truexml.jar"/>
            <pathelement location="extlib/db/h2.jar" />
            <pathelement location="extlib/poi/dom4j-1.6.1.jar"/>
            <pathelement location="extlib/poi/excelpoi.jar"/>
            <pathelement location="extlib/poi/poi-3.7-20101029.jar"/>
            <pathelement location="extlib/poi/poi-ooxml-3.7-20101029.jar"/>
            <pathelement location="extlib/poi/poi-ooxml-schemas-3.7-20101029.jar"/>
            <pathelement location="extlib/poi/xmlbeans-2.3.0.jar"/> 
            <fileset dir="extlib" includes="*.jar"/>
        </classpath>
        <arg value="." id="basePath"/>
        <arg value="userdata" id="userdataPath"/>
    </java>
</target>

<target name="runietests">
    <antcall target="clean-tests">
    </antcall>
    <sahi suite="../userdata/scripts/demo/demo.suite"
          browserType="ie"
          baseurl="http://${urlbase}/demo/"
          sahihost="localhost"
          sahiport="9999"
          failureproperty="sahi.failed"
          haltonfailure="false"
          threads="6"
            >
        <report type="html"/>
        <report type="junit" logdir="${userdata.dir}/temp/junit/tests"/>
    </sahi>
    <antcall target="report-gen" />
    <antcall target="failsahi"/>
</target>

<target name="report-gen">
    <delete dir="${userdata.dir}/temp/junit/reports">
    </delete>
    <mkdir dir="${userdata.dir}/temp/junit/reports"/>
    <junitreport todir="${userdata.dir}/temp/junit/reports">
        <fileset dir="${userdata.dir}/temp/junit/tests">
            <include name="TEST-*.xml" />
        </fileset>
        <report format="frames" todir="${userdata.dir}/temp/junit/reports/sahi-html" />
    </junitreport>
</target>

<target name="failsahi" if="sahi.failed">
    <antcall target="stopsahi"/>
    <fail message="Sahi tests failed!"/>
</target>


<target name="stopsahi" description="stop sahi server">
    <sahi stop="true" sahihost="localhost" sahiport="9999"/>
</target>

The important bits are

1. "sahitests" target which starts Sahi and runs tests in parallel.
2. "start" target which starts Sahi without the dashboard.

You could post the random failure problem in Sahi+PhantomJS on the Sahi forums for an answer.

Sahi's overhead/footprint as a proxy server is fairly small.