specflow plugin with nunit and TFS build

Question

I am in the middle of trying to get specflow UI tests working with TFS build and I am very very close.

I wanted to be able to run the same test on a number of browsers, and so after some research I found this:

http://www.baseclass.ch/blog/Lists/Beitraege/Post.aspx?ID=4&mobile=0

And it works like a dream for local test runs. the next step was to get this to call a selenium grid during a TFS 2010 build.

After getting the nunit community build task into my build template, I am now stuck with an error I can't seem to resolve. The build reports the following error:

Error: The system cannot find the file specified. Stack Trace: at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo) at System.Diagnostics.Process.Start() at TfsBuildExtensions.Activities.CodeQuality.NUnit.RunProcess(String fullPath, String workingDirectory, String arguments) in d:\Projects\CodePlex\teambuild2010contrib\CustomActivities\Legacy\VS2010\Source\Activities\CodeQuality\NUnit\NUnit.cs:line 339 at TfsBuildExtensions.Activities.CodeQuality.NUnit.PublishMSTestResults(String resultTrxFile, String collectionUrl, String buildNumber, String teamProject, String platform, String flavor) in d:\Projects\CodePlex\teambuild2010contrib\CustomActivities\Legacy\VS2010\Source\Activities\CodeQuality\NUnit\NUnit.cs:line 394 at TfsBuildExtensions.Activities.CodeQuality.NUnit.PublishTestResultsToTFS(ActivityContext context, String folder) in d:\Projects\CodePlex\teambuild2010contrib\CustomActivities\Legacy\VS2010\Source\Activities\CodeQuality\NUnit\NUnit.cs:line 387 at TfsBuildExtensions.Activities.CodeQuality.NUnit.InternalExecute() in d:\Projects\CodePlex\teambuild2010contrib\CustomActivities\Legacy\VS2010\Source\Activities\CodeQuality\NUnit\NUnit.cs:line 299 at TfsBuildExtensions.Activities.BaseCodeActivity.Execute(CodeActivityContext context) in d:\Projects\CodePlex\teambuild2010contrib\CustomActivities\Legacy\VS2010\Source\Common\BaseCodeActivity.cs:line 67.

however, this seems to be masking a deeper error as to me it simply says that nunit can't load a custom assembly. So I had a look through the binaries folder on the build server for the nunit logs, and sure enough in my test run .xml file I find another error:

System.Configuration.ConfigurationErrorsException : The type 'OpenQA.Selenium.Remote.RemoteWebDriver, Baseclass.Contrib.SpecFlow.Selenium.NUnit.SpecFlowPlugin' could not be found. It may require assembly qualification, e.g. "MyType, MyAssembly".

at Autofac.Configuration.ConfigurationRegistrar.LoadType(String typeName, Assembly defaultAssembly) at Autofac.Configuration.ConfigurationRegistrar.RegisterConfiguredComponents(ContainerBuilder builder, SectionHandler configurationSection) at Autofac.Configuration.ConfigurationRegistrar.RegisterConfigurationSection(ContainerBuilder builder, SectionHandler configurationSection) at Autofac.Configuration.Core.ConfigurationModule.Load(ContainerBuilder builder) at Autofac.Module.Configure(IComponentRegistry componentRegistry) at Autofac.ContainerBuilder.Build(IComponentRegistry componentRegistry, Boolean excludeDefaultModules) at Autofac.ContainerBuilder.Build(ContainerBuildOptions options) at RegistrationForm.Tests.Acceptance.Features.UserRegistrationFeature.FeatureSetup() in c:\Builds\1\Testing\RegistrationForm - Nightly - Main\Sources\Testing\RegistrationForm\Main\RegistrationForm.Tests.Acceptance\Features\UserRegistration.feature.cs:line 0

From this I started to look at the path to the custom assembly. This is specified in the app.config file for the project (which is transformed during the build. I also checked to ensure the file is being transformed, and in the binaries directory on the build server it is). the relevant section of the file is this.

 <specFlow>
<stepAssemblies>
  <stepAssembly assembly="SpecFlow.Assist.Dynamic" />
  <stepAssembly assembly="Baseclass.Contrib.SpecFlow.Selenium.NUnit.Bindings" />
</stepAssemblies>
<unitTestProvider name="SeleniumNUnit" />
<plugins>
  <add name="Baseclass.Contrib.SpecFlow.Selenium.NUnit" path="..\packages\Baseclass.Contrib.SpecFlow.Selenium.NUnit.1.2.0\tools" />
</plugins>

Which works locally. I figured I just need to change the path in the transformed file. At first I assumed the working directory was the binaries directory so in the transform file I have this:

<specFlow>
<plugins xdt:Transform="Replace">
  <add name="Baseclass.Contrib.SpecFlow.Selenium.NUnit" path="." />
</plugins>

The dll is in the same dir (binaries) so I tried ".", "", ".\" - none of these worked. So after thinking a little more and reading the errors more carefully I thought I needed to look in the Sources folder on the build server. The Dll is in the packages folder (nuget package restore), and so the path should be the same as I use locally. this didn't work either. So what about a path relative to the ".feature" file throwing the error? this simply needed an extra "..\" - still no luck.

So i am at a bit of a loss, I feel I have tried all the paths i can think of, but my knowledge of specflow plugins and TFS build is letting me down. can anyone give me any pointers?

Answer 1

I got there in the end!

So somewhere along the lines of setting this up I endded up with something like this in my app.config:

 <component
          name="IE"
          type="Baseclass.Contrib.SpecFlow.Selenium.NUnit.RemoteWebDriver, Baseclass.Contrib.SpecFlow.Selenium.NUnit.SpecFlowPlugin"
          service="OpenQA.Selenium.Remote.RemoteWebDriver, WebDriver"
          instance-scope="per-dependency">
    <parameters>
      <parameter name="browser" value="InternetExplorer" />
      <parameter name="url" value="http://192.168.1.3:4444/wd/hub" />
    </parameters>
  </component>

The problem (as it says in the error) is that it could not find OpenQA.Selenium.Remote.RemoteWebDriver. I belive I saw this in one of the examples I found on the bassclass page. After changing it to OpenQA.Selenium.IWebDriver everything worked fine, and nUnit was able to run my specFlow tests on my TFS 2010 build server. I did hit another snag in that when the build tried to parse the results, it tried to use MSTest 11 rather than 10 (prob because I was using VS 2012). However, this TFS machine was just a proof of concept and was a windows 2008 32bit edition running on Virtual PC on windows 7. (I could therfor not run 64bit, and so could not install VS2012 on my build server). I resolved this by creating a new environment variable for MStest 11 and pointing it at MSTest 10. The entire process now works perfectly.

Answer 2

But the initial error is basically saying the TFSs nunit support cant find a file when it calls CreateProcess, and as you say it works for local test runs, so is it as simple as NUnit isn't installed on the machine that is running the test for you?