Build pipeline, dependency on unreleased package

Question

My team is currently making some changes to our solution structure. Before the change we basically had a single solution file with about 40 different projects. Most of these projects are libraries that are consumed by a handful of applications.

To improve the load and build times, we decided to split the solution into several smaller solutions. We started with 2 of the biggest libraries, lets call them Core and DAL. Both got their own solution with a single project.

We created 2 build pipelines for both projects; one for pull-requests and one for building a NuGet package after a merge to the main branch. We have a policy that each PR should have 2 reviews and a successful build/test result in it's pipeline.

Now let's say we have a hypothetical bug in the DAL (v1.0.0).

Previously we would fix the bug in the library, run some applications locally (which would call the updated DLL) and if it works commit the change, start a PR which would kick off the PR pipeline and pretty much be done with it.

But in the new scenario, in order to test the change, the developer has to publish a local NuGet package let's say 1.0.1-pre. Then he updates the package in one or more applications and if everything is fine, he publishes a local 1.0.1 package, updates the consuming applications and makes a single PR for the bug-fix and the version increase in the consuming applications.

But here's the kicker; the PR pipeline for those consuming applications will fail, because the new package is not yet published to our feed. In order to publish the package, the PR has to be completed, but that's not possible because the changes to the library are in the same PR as the version increases to the apps.

We're at the point of instructing all devs to not include changes to packages with other changes in the same PR (even if they're very related) because of this issue, but as a last resort I'm asking here if there's a better approach.

Answer 1

A PR should be for one project only

Updates to a package should be separate to updates to the consumer of the package.

If there is a bug in DAL, then it should be fixed in DAL, a PR triggers the build and its version bump, and a new package appears.

Now the consuming application can pull the package, and perform any associated updates. This is passed off in a PR which bumps its version and any subsequent packaging.

The reason is that the consuming application might not be able to transition to the new package just yet. This could be due to regulatory reasons, or to additional code changes in the consumer.

Separate Project <-> Separate Repositories

Decide whether or not the DAL is a separate project or not.

If its not, ditch the package and consume the library as built by the current check-in. Otherwise your application will be chasing its historical tail and depend on the product of an earlier commit.

If it is, re-home it in a new repository, and actually give it the independence implied by having its own package.

Quality Gates and Red Flags

Another issue is your quality gates. The quality gate for your DAL package appears to be works in the consuming app. As far as quality gates go, undoubtedly that is the end goal, but it does get you into a situation where the work done to accommodate testing, is almost the same work to incorporate the new package.

This raises a few red flags in my mind:

• The Unit/Module/Semantic Testing for DAL is not self-contained alongside the code for DAL, but spread across repositories.
• Any change to a consuming application other than bumping the version number implies:
  • Functionality has been deleted, when it is still in use.
  • Functionality has meaningfully changed, which is identical to having deleted the functionality and created something new with similar signatures. A source of insidious bugs.
  • The consuming application was always incorrect, and as the consuming application is being used as a test, then the test was always wrong and should be fixed first (ergo the consuming application should be fixed first, by effectively making it not work with the previous package version).
• Any change to the consuming application will potentially invalidate a previously built and ratified DAL package.

Clarifying Quality Gates and Automation Goals

I would clearly define the quality gates, and potentially automate updating the dependent package.

To start with find all locations in the consuming applications that interact with the DAL, and distil those interactions into Module/Unit Tests. Make this the quality gate for building a DAL package (alongside the human quality measures such as reviews).

Once the DAL package is created, the second quality gate is to checkout the consuming application code. Update the dependency, test, and automatically generate a pull request. If your architecture permits it this could also be checked after each accepted PR to the consuming application so that if the consuming application is "fixed" it will automatically get the superior package suggested.

Devs will need to perform a PR once for non-breaking changes which should include (most) bug fixes, and extended behaviour. These correspond to patch and minor semver bumps. Simply updating the DAL will propose the update to the downstream consumers.

Breaking changes will be necessarily more involved largely because the semantics are still being defined/redefined. This however should be the exception, not the rule. These changes should be occurring under a major version bump so that the package isn't automatically applied to consuming applications.

To assist developers in updating the package, provide a means to build/test each downstream system from the main/their fork with a built version of the DAL from their fork. Extra points if it also provides build/test-results against the current published package too. This would allow the developer to know when a package is a breaking change, or not. It would also allow them to make decisions about whether or not their PRs are dependent on the new package, and thus when they can push the PR for that particular project.

Answer 2

When separating a big project into smaller ones, you HAVE to think about how that project will be tested. In practice, by separating big project into separate development/build projects, you have to assume those projects need to be independent of where they are used. And that they have completely separate lifecycle from the upstream dependencies.

Basically, your split into CORE and DAL is simply not correct. Go back the the drawing board and re-architect your solution so that changes to each project can be tested without need for upstream dependencies.