Is there a standard way to test scripts/executables in Jenkins?

Question

We have a project that contains a library of Python and Scala packages, as well as Bourne, Python and Perl executable scripts. Although the library has good test coverage, we don't have any tests on the scripts.

The current testing environment uses Jenkins, Python, virtualenv, nose, Scala, and sbt.

Is there a standard/common way to incorporate testing of scripts in Jenkins?

---

Edit: I'm hoping for something simple like Python's unittest for shell scripts, like this:

assertEquals expected.txt commandline
assertError commandline --bogus
assertStatus 11 commandline baddata.in

Answer 1

I do not know how 'standard' this is, but if you truly practice TDD your scripts also should be developed with TDD. How you connect your TDD tests with Jenkins then depends on the TDD framework you are using: you can generate JUnit reports for example, that Jenkins can read, or your tests can simply return failed status, etc.

Answer 2

Have you looked at shunit2: http://code.google.com/p/shunit2/

It allows you to write testable shell scripts in Bourne, bash or ksh scripts.

Not sure how you can integrate it into what you're describing, but it generates output similar to other unit test suites.

Answer 3

If your script requires another project, then my inclination is to make a new jenkins project, say 'system-qa'.

This would be a downstream project of the python project, and have a dependency on the python project and the in-house project.

If you were using dependency resolution/publishing technology, say apache ivy http://ant.apache.org/ivy/, and if these existing projects were to publish a packaged version of their code (as simple as a .tar.gz, perhaps), then system-qa project could then declare dependencies (again, using ivy) for both the python package and the in-house project package, download it using ivy, extract/install it, run tests, and exit.

So in summary, the system-qa project's build script is responsible for retrieving dependencies, running tests against those dependencies, and then perhaps publishing a standardized output test format like junit xml (but at a minimum returning 0 or non-0 to clue Jenkins in on how the built went).

I think this is a technically correct solution, but also a lot of work. Judgement call required if it's worth it.