When would you choose *not* to update a third-party library to a newer version?

Question

Using third party libraries for productivity gains in software development is common. Unfortunately, along with the library's functionality we also import its bugs. Some of them get fixed in subsequent releases. So, to upgrade or not to upgrade, this is the question.

I am interested in learning from experiences when upgrading to a newer version of the library was desirable, but after a cost/benefit analysis the conclusion was that upgrading was not a good solution "in the grand scheme of things". I am interested in finding out what forces influence the decision towards not upgrading.

Answer 1

The short answer is that the new version could introduce new bugs, as Robert Harvey mentioned in the comments.

To be honest, I think you are approaching this the wrong way. In my experience, the better approach is to default to sticking with the version you have. Presumably, your team has already tested the version you use and how it integrates with your system. If not, then your clients have.

Sure, when a new version comes out, your team should see if it addresses any bugs you have encountered, provides features you could take advantage of, or improves overall performance. Only after running it through testing again should you actually start using it.

The problem with approaching upgrades as something which is done by default and only halted if someone finds a problem is that it is too easy to become lax on the vetting process. Eventually, this will bite you when a new version introduces new issues.

If you stick with the version you have, you know that the overall system behavior will not change.

Answer 2

To answer, let me first turn the question around on the need to upgrade.

If you never upgrade, over time you are stuck on an outdated version and will find bugs in your system that results from bugs in the library that have been fixed in later versions.

Assuming that you are continuing to develop your system, and the relevant library is of any size of complexity, sooner or later it will be worth upgrading. I.e. the cost of retesting justifies the benefit of the remaining on the latest version.

So, you may not need to upgrade if these assumptions fail.

I.e.:

1. You are not actively your application.
2. The library has minimal changes, perhaps only changes to specific problems that you know are not relevant to you.

It's more complex that this, as (2) is a moving target. With each library version that comes out, the accumulated change increases and hence the likely value (as well as risk and testing requirements).

Note that if you delay upgrade a long time "because so far none of the fixes matter to me", then when you do upgrade you may find it much harder due to accumulated impact.

Further, you may have multiple third party libraries that may require upgrade and retest from time to time. One approach is to review and consider upgrade of all third party libraries once per year (or whatever period), and execute the upgrade within a single project, allowing a single full system regression test. This might include upgrade to compiler or operating as well.

Finally, any risk in an upgrade will be inverse to the amount of automated testing in place. If you have very little (relevant to the library) and need to do a lot of manual testing you need more benefit to justify the upgrade cost.

So for example, upgrading a user interface library which requires extensive GUI retest is different to upgrading an XML lib which reads in a few files amenable an automated test.