Post Commit Hook - Jenkins: Starting jobs after Jenkins downtime

Question 1

If you are concerned about missed commit, you need to buffer the notifications to Jenkins. One possible scenario would be the following:

Write the info to a file (database) when Jenkins is not available. You need a second process that reads the file (DB) periodically and pushes the info to Jenkins when Jenkins is available.

A completely different option is to ignore the case when Jenkins is not available. When there are frequent changes to the source, then the next build will be triggered soon anyway. For the jobs where there are only infrequent changes you can use the Startup Trigger Plugin. You can also have a job triggered at startup that reads the files (db) from the previous example.

In general, just go through the Jenkins Plugin list and let your imagination go wild with what the plugins can do for you. You might come up with very creative solutions.

Question 2

Jenkins will poll your source repository down to once per minute in order to see if something needs to be built.

When we used CVS, I used a post-commit hook to force Subversion builds because polling a CVS repository to see if a change has occurred was a long, and processor intensive activity. If you do a dozen or more project in Jenkins with CVS, you'll slow Jenkins and your CVS repo to a crawl. Using a post-commit trigger was the only way. However, one of the issues we ran into is that if Jenkins was down, it wouldn't do the missing builds.

When we moved to Subversion, I simply set Jenkins to poll our Subversion repositories every minute. Finding whether a change took place in Subversion is pretty fast and takes few resources. If the Jenkins server is down, Jenkins will immediately poll the various Subversion projects and will start building practically where it left off.

There are a few downsides with this behavior and the reason why some sites don't allow Jenkins to poll the repository:

It could take as long as 59 seconds for a build to happen. If a build takes two minutes to do, the developer could be waiting 50% longer to find out if their change was successful.
If two developers make a commit in that one minute period, both of their changes will be built. If a failure happens, we won't be able to say which change caused the problem.

To me, these are minor issues, and I like the fact that Jenkins will take care of itself without me worrying about it. However, if you need to have Subversion trigger builds in Jenkins, you can do it with a post-commit hook issuing a build command, but as you figured out, if Jenkins is down, it doesn't learn it has to do a build.

One way around this is to have Jenkins both do polling and use a post-commit hook to trigger Jenkins builds. There's a file system SCM plugin that will poll a file or a directory, and trigger builds off of that. There is also a command line interface to Jenkins too. You could combine these in your post-commit hook.

Use the command line interface to see if Jenkins is up and running. If it is, you can trigger the build via the URL as you normally would. If Jenkins is not up and running, update the polling file.

Jenkins can poll this file every 5 minutes or so (no need to do a minute-by-minute trigger), and if it finds this file has been changed, it will automatically do a new build. This way, if Jenkins is down, you can use the polling mechanism as a backup.

Question 3

Our solution to the issue:

Instead of having the post-commit hook call the jenkins server directly, we call our central Quality application (controls all our Jenkins servers, jobs etc.). This application then push the build on the respective Jenkins server. If the Jenkins server is down - the application tries again later.