How to prevent bugs in business-level configurations with similar discipline as in source code?

Question

We have a system that allows our clients to coordinate people (shoppers) so that they can delivery groceries within 45 minutes from the order creation.

Each client has a set of stores where the orders are processed by the shoppers
and each shopper has an schedule that specifies in which store and at what times he/she will be working.

However, we have also a feature that only allows them to create orders if the store is already configured in our systems and if there are enough shoppers at the time we expect to deliver the order.

As you can see, checking if a given order would be accepted in our system depends on the current time, configuration of stores and shopper's schedule.

We've done a pretty good job managing our source code, we do code review, unit testing and integration testing. However we've not found a satisfactory way of managing the configurations of the stores and the schedules and they are as sensitive as source code if there is a wrong configuration, for the client it would be almost the same as if we had a bug in our source code because we won't be letting them to create orders and their sales would decrease.

Thus we would like to find a way to manage the configuration of the stores and the schedules with similar discipline to what we do with our source code. we would like to have a way of testing if the configuration is correct. Right now we do this configuration via SQL scripts which are testable but the inputs might not be correct and we don't have a way of managing them or testing if they are correct.

Have you had a similar problem? Do you know any tools, languages or processes that will prevent us from making mistakes on the client configuration and to have a centralized way of knowing what is the actual configuration besides of looking at database tables?

By the way, we receive these configurations as spreadsheets from business people that talk with the client.

Answer 1

The iron rule of software is:

Garbage in, garbage out

To cope with this hard fact of life, you need to address the requirements that you've discovered.

Configuration process

The configuration process is a preparatory activity of your system deployment. If you want it to be reliable and repeatable, automate it with a configuration generator. This piece of software would support the configuration process as seriously as your core software supports the business process.

Analyse the requirements of a valid configuration and how input data might impact them. Then design the generator that would reliably automate these manual error-prone deployment activities. Eventually replace Excel with a proper configuration assistant that would guide the operator in the data entry.

For the sake of curiosity: system configuration was one of the first real-world use of rule based experts systems in the 80s. But your needs will not necessarily require such an advanced engine ;-)

Schedules

Schedules seem to be business data. Business actors should be able to change these based on business reality. And wrong data or wrong decisions are a business reality.

If some schedules could pose a risk to your customer's process (for example because the other rules would prevent any delivery) a business requirement should be to prevent such risks. For example inform the business-actor in charge at the moment of data entry or schedule validation of the potential risk.

As soon as the responsible business actor is aware that there are not enough shoppers in some slots, it is no longer a question of data inconsistency but a business reality that needs to be addressed by the business-actors (in this example, with staff reinforcement).

Conclusion

You seem to have a strong development process for addressing the business needs. May be it's time to enlarge the scope and include also deployment and operational requirements, in the spirit of DevOps?

Answer 2

You wrote

we do code review, unit testing and integration testing

(and I guess you also use source control). All those techniques can be applied to configuration files (or schedules) as well - at least, in a comparable fashion:

• source control = proper versioning of the configuration files, if necessary with a detailed changelog and "diff tools". It is not necessary that the people managing those files use the same SVCS tools as the dev team, they could also do the versioning manually, or using zip files or a DMS, if they prefer.

• code review = proof reading by a second person. Actually, the term "proof reading" is probably much older than "code review", but it is essentially the same thing.

• unit testing - this may not map 100% to configuration files, but somewhere in your system there should be at least some sanity checks for the configuration parameters or rules, producing detailed error messages when the configuration is syntactically wrong or inconsistent.

• integration testing - this is probably the biggest challenge. One approach is actually not different from what you need for an integration test with your software elsewhere - a test environment where a new configuration can be tested with almost-real data, maybe manually, maybe automatically. Another approach could be to implement something like a "dry run" into your production system - so if the business people change a config file, they get the opportunity to test it on the live system without actually changing real data.

As you see, there are no fancy "new tools" necessary, all the usual means of quality assurance can be applied to configuration files as well. You just have to convince people to apply them.

Answer 3

Some people don't realize this, but handling configuration is a software problem of its own, with it's own set of design challenges. Sadly, because your software is unique to your company, your company is the only one that can tool for your sofware.

The need arise for a back office. At first, it can simply be an internal web interface where the business people can upload a fixed format spreadsheet. Your backend could do some basic checks before inserting it in your DB, and raise errors for the business people to fix what's broken. It's easy to setup and can give you a quick start, but that's not really friendly to use and can be time-consuming to fill if the configuration is complex.

In the long run, what you truly need is to have your own web-based configuration editor with real-time validation. Once you are there, you can put the back office in the hands of the clients to fill their configuration directly. Not only this would make the configuration more robust, it would considerably improve experience. It could be development-cost related choice, but I find surprising that your company didn't anticipate for this.