Using mercurial for storing binaries w.r.t regression testing

Question

I have a project using mercurial as its dvcs

I have written a regression testing tool in python

Every now and then i tag a changeset (say for a customer release)

The regression test scripts are in the same repo as the src code and I ensure they all pass for a given tag

If I make a bunch of changes that break the regression tests i can hg up to the last tag changeset and have a look at whats different - but it requires me to rebuild the application

My question is whether i should include the application binary in my repo so my regression tests automatically use an appropriate version of the binary or, alternatively, should i have my regression test tool automatically build the binary each time it is run

My gut instinct is to just include the binary in the repo to reduce the need to rebuild every time i run my regressions, but I have a nagging feeling you're not supposed to do that sort of thing

Would be very interested to hear what best practice is in this area

Answer 1

If you want to version the built binaries, I would suggest storing them in a subrepo. However, you do not want the subrepo to be a subrepo of the main repository.

Instead, I would recommend configuring as follows:

umbrella_repo/
    .hgsub
    source_repo/
    binaries_repo/

Then you can clone source_repo/ completely independently and develop in it. Your regression setup has a clone of umbrella_repo, which automatically clones source_repo and binaries_repo, and keeps their working directories at the revision they were in when you committed to umbrella_repo.

When you want to make a new regression checkpoint, in your regression setup:

1. Do a clean update in umbrella_repo, and probably a purge:

   cd umbrella_repo
   hg update -C tip
   hg purge --all
   

2. Update to the latest into umbrella_repo/source_repo:

   cd source_repo
   hg pull
   hg update -C <revision you want>
   hg purge --all
   

3. Build.

4. Copy the binaries to binaries_repo and commit.

5. Commit in umbrella_repo.

Now, when you update to the revision you just committed in umbrella_repo, it will automatically update to the matching revisions in source_repo and binaries_repo.

For more information, look at the Mercurial help on subrepositories. Note in particular that most commands do not recurse into subrepos by default.

Answer 2

From a version control point of view, it's not recommended versioning these secondary files since they can be rebuilded at any time. Besides, they would quickly bloat the repository, compromising cloning and backup.

If the rebuild time is an issue, I suggest you to keep the all binaries in another location.