Staged deployment of snapshot artifacts with Maven and Jenkins

Question

Problem Description

We have several multi-module projects which are depending on each other. So something like this:

• messaging
• framework
• othercomponent

They have separate source control repository, and lets say, that submodules inside messaging and othercomponent are using bundles from the submodules of framework. All of the projects are OSGI based multi-module maven projects. All of them have a server part, and a single sourced GUI part (Eclipse RAP+RCP). So this ends up in a three step maven based build for all of these multi-module projects (since Tycho builds cannot be mixed with plain old Maven Builds):

• Building server part
• Building RAP GUI part
• Building RCP GUI part

And finally there is an end product multi-module maven project (lets call it ourproduct), which uses messaging, framework and othercomponent. The project ourproduct has different version number than the other three, which are having a common version number.

We build the whole thing with Jenkins, and the jobs trigger each other depending on the dependency tree. The company decided to use snapshots to get more direct and fast feedback between the 3 framework project developers and the ourproduct developers. This can be a nice idea, however there is a big problem.

If during the building chain something is broken, than the snapshot repository contains snapshots of messaging, framework and othercomponent, which cannot work together. This way the developers of ourproduct have to wait for a working snapshot set (otherwise they cannot even compile some time). An other problem is that during the build the set of snapshots are also not consistent.

Idea

For the messaging, framework and othercomponent there is a definite end job in Jenkins. If that finishes, then the set of snapshots must be working, so the ourproduct team could use it. So i would need to COLLECT somehow the snapshots created by the build chain, and deploy them only if the whole build chain was successful.

• Is there any existing possibility to do this?

My idea was to simply change the jobs to make only install only and not deploy. Then at the end I could look for the built snapshots in the local maven repo, and deploy them by a script.

• There is some Staging Concept for Maven (maybe only with Nexus pro). Does it say anything about snapshots?

Any idea is welcome. However i cannot change the fact of using snapshots. So there is no use in convincing me to use releases and no snapshots.

Answer 1

Solution

I have found a nice workaround. Steps to take:

1. The build steps have to deploy to a temp folder instead of the normal target repository using the altDeploymentRepository parameter (see http://maven.apache.org/plugins/maven-deploy-plugin/deploy-mojo.html)

   mvn deploy -DaltDeploymentRepository=stagingFolder::default::file:///c:/mytempfolder

2. Run any number of build steps with this setting, and the artifacts are collected in the folder. You can even resume the builds without problem. You can overwrite artifacts (i do not recommend it however)

3. You have to run the following command after completing all build steps (see http://mojo.codehaus.org/wagon-maven-plugin/copy-mojo.html). This uploads all artifacts which were collected temporarly in the folder:

   mvn org.codehaus.mojo:wagon-maven-plugin:copy -Dwagon.source=file:///c:/mytempfolder -Dwagon.target=http://somerepository.com/repositories/snapshots -Dwagon.targetId=idreferredinsettingsxmltogetauthorization

Important Note

The wagon goal should be run in a folder, where there is no pom file (so it must be run without project). Otherwise there is a weird error with the fromDir parameter.

Known Limitations

1. The build steps should run using the same local repository, since if the steps need the artifacts produced by other artifacts, they can find it in the local repository.
2. This solution does not read the POM to get the repository to which the artifacts should be uploaded. This is wired today into the wagon command. But I can live with it now :)

Answer 2

Jenkins allows to collect artifacts of other builds (if they mark these in their project configuration). Your integration tests could use that construct to collect required from the snapshots builds and only upload a set of bundles to Nexus once integration tests are completed. You might start by inspecting the manifests from the OSGI bundles.I suggest to use jar-signing as a staging-marker, because it is easy in forensic analysis by devops.

btw Nexus Pro has a staging concept.