Uploading files from last svn revision to SFTP in Jenkins CI

Question

I am configuring Jenkins job for uploading files from Subversion to SFTP. Publish Over SSH is doing quite well, but it uploads all files on every build.

For some projects we have thousands of files and upload cost more than 1 hour so that's not an option.

Can anyone suggest a way to upload only files changed in last revision?

Answer 1

You can use Subversion Plugin to poll for SVN changes and run a job that does nothing in particular. Let's call it YourPollingJob. Then call Jenkins API via http like this:

http://[jenkins_server]/api/xml?depth=2&xpath=/hudson/job[name='YourPollingJob']/build[id='BUILD_ID']/changeSet

where BUILD_ID is the actual build id of the job you've just run (the usual format is something like 2012-02-21_16-15-49). Examine the result. Note that all the information about the files that have changed since the previous build is there - you just need to parse it out.

So now you can do the following: YourPollingJob will call another job - let's call it CopyJob - and pass to it its BUILD_ID as a parameter (via the Parameterized Trigger Plugin; make sure to call CopyJob as a post-build step, not as a build step). CopyJob will then query Jenkins via http (as above, the best way is to do it via wget), parse the results, and do the copy.

You can do it all in one job, but it's a bit more complicated and hard to debug.

Also, it is probably prudent to copy the whole repository once a day (nightly).

Answer 2

You could keep those files in a separate repository. It sounds like a distributed VCS like mercurial or git would be best for that in your case. You could either set it up in the workspace and add a build step to commit changed files and push the commit to the server where you want to publish it (which should work as long as you don't wipe out the workspace or build on several machines) or add additional steps to clone/pull the repo with the artifacts from the server you keep them on.