Continuous build infrastructure recommendations for primarily C++; GreenHills Integrity

Question

I need your recommendations for continuous build products for a large (1-2MLOC) software development project. Characteristics:

• ClearCase revision control
• Approx 80% C++; 15% Java; 5% script or low-level
• Compiles for Green Hills Integrity OS, but also some windows and JVM chunks
• Mostly an embedded system; also includes some UI pieces and some development support (simulation tools, config tools, etc...)
• Each notional "version" of the deliverable includes deployment images for a number of boards, UI machines, etc... (~10 separate images; 5 distinct operating systems)
• Need to maintain/track many simultaneous versions which, notably, are built for a variety of different board support packages
• Build cycle time is a major issue on the project, need support for whatever features help address this (mostly need to manage a large farm of build machines, I guess..)
• Operates in a secure environment (this is a gov't program) (Edited to add: This is a classified program; outsourcing the build infrastructure is a non-starter.)

Interested in any best practices or peripheral guidance you might offer. The build automation issues is one of several overlapping best practices that appear to be missing on the program, but try to keep your answers focused on build infrastructure piece and observations directly related.

Cost is not the driving concern. Scalability and ease of retrofitting onto an existing infrastructure are key.

(Edited to address @Dan's comment. ;-)

Answer 1

From my experience with similar systems, there are approximately two parts to this problem:

• A repeatable method for checking out sources, building the software, and testing it (if you want to do continual testing as well as building), using a small number of command-line invocations.

• A means of calling these command lines on various servers in the build farm.

For the latter, we've been using BuildBot, which seems to work pretty well.

For the former, we have a homegrown solution that started out as a simple bash shell script and grew ... rather substantially. From experience, I'd suggest starting out in python rather than bash -- you'll spend far more code in handling setup and configuration than in actually invoking programs. (Also, it's probably easier to run it on Windows if you're doing that.)

The things I've found to be really key in our script's usefulness are:

• Ironclad repeatability. We have a standard set of build tools, and the scripts start out by scrubbing environment variables. There are very few command-line options; everything goes into configuration files, and those go in version control.

• Logging. We produce a log of every command that the build script executes.

• Configuration file inheritance. Each variant of our software gets a configuration file, and those files can include more-general settings (which include even-more-general settings).

• Extensibility. When we add a new source component, it's pretty easy to add a set of instructions for building that component (and the instructions can be arbitrary bash code). The "can be arbitrary code" part is probably key here; no way is a pre-existing product going to be able to do all of the quirky things that you need for a large complex real-world system.

You can get started with a reasonably simple script and let it grow organically as the need arises; honestly, although ours is a bit messy, I think we got a much more usable result that way than we would have with heavy top-down design.

Answer 2

Cost isn't an object? I've worked for GreenHills, and they've solved these issues for their in-house build/test farms. Ask them to do the same for you.

Answer 3

When I see emphasis on things like scalability and security in a build system, I start thinking that you might be a candidate for the enterprise class build systems / CI systems. Conveniently, it sounds like you can afford them as well. A year old SD Times article provides a basic breakdown between the enterprise and team level build tools.

My company makes AnthillPro and we've worked with a number of companies on large embedded projects as well as highly secure projects. IBM is probably the largest other player in the space with BuildForge.

AnthillPro puts some extra emphasis on what you do with the images in the minutes/hours/days post build (do you install them onto simulators / hardware and run automated tests? stage them? promote them?) but we also see folks using it for just build.