How to organize the build system of a project as it starts including multiple languages across multiple operating systems and compilers?

Question

I wrote a Music Player and Library in Java for GNU/Linux and Windows.

My build process is currently in ANT, but I intend to migrate away from that to something more modern after the next release. I've heard good things about gradle, but if there's something more suitable you'd like to recommend, I'm open to that.

The project was originally in pure Java, but in order to get access to certain features, I've had to start special-casing different native features and linking them into my program using JNI.

Here's an outline of the current parts:

• Hypnos Base - written in Java, compiles to a jar.
• JXGrabKey - An existing open source hotkey library, modified by my project to support javafx. Compiles to a jar that is included in the project via manifest.
• GNU/Linux Native Launcher - written in C++, compiled with g++.
• Windows Native Launcher - written in C++, compiled with cl.exe.
• GNU/Linux GTK Tray Icon - written in C, compiled with gcc.

In addition, after the various items are built, I bundle together the pieces and then I run a NSIS script to generate a windows installer, and an AppImage program to generate an AppImage for Linux.

For the natively-compiled code, I was compiling it by hand as needed, and then providing it to ANT in binary form to do my packaging.

As I'm adding more and more native code, I'm starting to want to be able to recompile the entire project on demand with my build script rather than having to manually do each module before starting ANT. It feels irresponsible to have an open source project where the build process is arcane.

My primary development system is Ubuntu 18.04. Prior to today, all of the build code was doable on that system. Now I've added the native launcher for windows, which is currently compiled with cl.exe and targets 64-bit Windows. It's relatively simple code (~60 lines) with a single function, WinMain, some basic library linking work and then handing things over to jvm.dll.

Answer 1

Build Systems are Complex

Build systems are by necessity quite complex and internally arcane. The important part is, like with your music player, how it simplifies that complexity to provide some simple rational actions to the build system user. How configurable those actions are will largely depend on how customisable the build needs to be.

The build system is usually responsible for:

• Analysing source for issues (linting, dependency audits, etc...)
• Compiling source into binaries.
• Unit testing + code coverage.
• Some light-weight module/integration testing.
• Packaging
• Release
• Deployment (depending on what is being built)

Organise it by deliverable.

Literally, treat each part as if it were a separate project. Give each one its own build script. This way you can pick the most suitable build system for that aspect of your project.

You would normally have a single master build script that orchestrates the sub-builds to be built in some order without conflicts etc... This master script would be the equivalent of you: building part 1, move x to y, build part 2, ....

Considering that you have a cross-platform project it may not make sense to have one single master build script. In this case organise the sub-projects by a mix of platform (for platform-dependent code) and sub-system (for cross-compilable/platform-agnostic code). Use a platform-master build script to orchestrate building the sub-systems and platform dependent code.

It may make sense to breakout sub-system and the master-platform build scripts into separate "pseudo" projects as they may depend on binaries not built locally.

Organising this way will help you modularise and later ease the process of breaking this into separate repositories. Additionally the individual build scripts will make tooling a build pipeline for a build server much easier.

Binary Repository

Because you have a cross-platform system, and it appears you can not hope to cross-compile everything on a single box, I would investigate having a binary repository. This could be as simple as a dedicated Git repository that you check-in build artifacts to, a file-share, or some package system. The only real requirement is that you can easily identify the latest appropriate releasable build, and retrieve it.

Now when you build a releasable version, the build script uploads it to your binary repository. When another build script needs a releasable dependency it retrieves it, warning or failing when not found.

The Build Tools

There are many build-tools out there for orchestrating projects. The lowest entry tools are the shell scripts (bash/powershell) I'd argue for using these even as just wrappers around the actual build tooling.

As for actual build tooling, that really depends on your needs. Take a look at the different build tooling out there and find the ones that offer you the best experience. You will need to balance the benefits of these tools, against their costs such as installing/versioning/configuring in comparison to writing your own shell/cross-platform scripts.

General Advice

Personally don't throw Ant out the door just yet. Instead give those smaller projects build scripts, then orchestrate them (even roughly) with master-platform scripts. While you are doing this experiment with a few different build tooling sets. When you are happy with the new build tooling set, then migrate your ant based build.

Don't worry overly much if you end up using three (or more) separate build-systems. You can largely abstract those details away in the build scripts, and provide a follow me document to setup the environment once off. If you really want it effortless for the end-users, script a verify/install/update script for these build tools on the platform.

The most important part though is to simplify the entire build process down to something like build on platform A, then build on platform B.