Test Automation - TeamCity + Calabash + JIRA + SpiraTest

Question

I'm following up your question from this Calabash-iOS group thread. Can't tell you much TeamCity or Selenium, we're using Bamboo and JIRA and I just started looking into integration of Calabash tests with JIRA.

The (A) Part

There shouldn't be any big difference between TeamCity, Bamboo or Jenkins. All of them support running custom shell scripts. You need Ruby to be installed on your CI box and all the gems required (cucumber, calabash-cucumber and the rest). Here's an example of Bamboo CI plan we use for testing

Checkout iOS app code
Add calabash framework to the app (shell script using calabash-ios setup)
Build calabash test target (shell script using xcodebuild)
Checkout Cucumber test code (we keep it in separate repo)
Run cucumber tests agains iOS app test target

The last part is a simple shell script, something like

cucumber -p smoke-test

This will run the smoke test profile and create test report in HTML format. Then you can use Cucumber Test Report plugin. I know for sure there's one for Jenkins and Bamboo, so should be one for TeamCity, or simply pick up the HTML and publish it as test results.

The (B) Part

I just started looking into that, here're some findings.

Atlassian CLI

Atlassian have a set of CLI for all their products, here's the link on their marketplace. This is complete set of tools, you can search on the marketplace for JIRA-only package if you don't want to download all of them.

The tools have .sh scripts for UNIX-like systems and .bat files for Windows. Here's documentation for jira command, here are some examples. Install tools on your CI box, update all the paths and you'll be able to use Atlassian commands from your CI tasks and even from Cucumber steps and scenario hooks.

For Mac OS X there are a couple of custom homebrew taps for atlassian and jira cli, but each installs tools in a different way, also, both are outdated and install old version of cli.

Also, there's a Ruby gem called jira-ruby, you'll have to install atlassian-plugin-sdk homebrew tap on Mac OS X. I didn't try to use this gem yet, but it looks really promising.

Use Tags and Scenario Hooks

This is the thing I am trying to do right now.

With JIRA you sure have those User Stories, Bugs and other types of issues for the project. Normally Cucumber Scenario will relate to a certain User Story, or you can have multiple scenarios for one User Story, or even have one scenario shared between multiple User Stories.

We also have a number of scenarios that are regression tests for particular bugs in JIRA.

Anyway, you can use tags to link Calabash/Cucumber scenarios to JIRA issues and projects. Tag each feature with project name, like this @JIRA-PROJECT-<PROJECT-NAME>, also tag scenarios with JIRA issue names @JIRA-ISSUE-<ISSUE-NAME>.

Here's an example

@JIRA-PROJECT-X
Feature: Some Feature for project X
  Feature Description

@JIRA-ISSUE-X-123
Scenario: Some Scenario for User Story or Bug X-111
  <steps>

Then use After Scenario Hooks. When the hook is executed you have scenario object of Scenario class. This object has all the information you need in order to update JIRA. It has passed? and failed? methods, and what's more important, all the tags.

puts scenario.source_tag_names.join(", ")
# => @JIRA-ISSUE-X-123, @JIRA-PROJECT-X

Now you can figure out JIRA project and all the related issues from the tags, and create Test Session for each issue. Test Session is just another issue of "test" type with proper parent. See the rough example below, it needs to be modified to create test session for each issue related to scenario.

After do |scenario|
  project = scenario.source_tag_names.find { |t| t.start_with?("@JIRA-PROJECT") }.gsub("@JIRA-PROJECT-", "")
  issue = scenario.source_tag_names.find { |t| t.start_with?("@JIRA-ISSUE") }.gsub("@JIRA-ISSUE-", "")
  summary = "<Test Summary, passed or failed, etc..."

  # use proper jira executable name depending on your installation
  system("jira.sh -s <jira-server> -u <jira-user> -p <jira-password> -a createIssue --project #{project} --type test --summary '#{summary}' --parent #{issue}")
  # the same can be done using jira-ruby gem
end

Update JIRA After Tests Complete

The other approach would be to let Calabash run it's course first, then parse HTML report and update JIRA with test results in a single batch. JIRA CLI has a special action for that called run (also runFromCsv and others).

Instead of parsing HTML, you can use scenario hooks to write JIRA actions to a file and use this file with JIRA CLI run action after the tests are complete.

P.S.

Since I'm doing this for the first time (the whole Calabash & JIRA integration), I can't be sure this is the right way to do things. There is some plugin for JIRA, called Behave, might be that's what you need. I don't know if it plays well with Calabash, though.