Finding Feature Bugs As Quickly As Possible

Question

Currently, whenever I am developing a feature, I write the behaviour tests for it using mocks for our downstream services. These mocks are dumb always serve the same output. I faced something grave today. My BDDs were passing in my local and build environment. Therefore, I went ahead and merged my changes into the 'master' branch.

However, while testing the feature manually in my dev environment, I discovered a bug. One of the downstream services was responding with a 4xx. I know that this is bad - the master branch now contains a feature that is buggy.

If I would have written an integration test using actual services and made it a part of my build, this problem would have been solved. But say, if one of the downstreams goes down, then my build won't pass. I don't want this.

Note that we don't have a QA branch as we don't have a separate QA tester. We are all expected to test our features by ourselves.

How should I mitigate this issue?

Answer 1

Testing with stubs/mocks/fakes is good, but it can never fully replace system and integration tests where the real thing gets used.

If you don't have enough control over the external service to reliably use it in an automated integration test, then you should perform a manual integration test before you merge your work to the 'master' branch to prevent just the kind of embarrassment you got this time.

Another improvement to catch these error earlier (assuming the 4xx response was caused by a change you made in your code) is to make the mock for the external service smarter. Instead of always blindly serving the pre-configured data, the mock should also be able to check if the received request is as expected. If the request is not as expected, then there should be a failing testcase.

A test that validates that the correct request is sent still does not remove the need for an integration test with the real external service, because there can always be a different interpretation of the API between what you write in your testcase and what the real service implements.

Answer 2

Not having separate QA is a very bad thing. At the most basic level, the software developer wants the software to work to demonstrate they did a good job, and the QA person wants the software to not work to demonstrate they did a good job. You can't do both at the same time.

Answer 3

When you are writing mocks for the upstream services you depend on, you are matching the service's behavior for a limited number of cases.

If you get a 4xx error from the service in production, it means your mocks are not accurate to the service's actual behavior or the upstream service introduced a breaking change.

The best way around this is to checkout the services documentation and have your mocks more closely match it.

You can use something like OpenAPI Specification to document APIs.

Answer 4

If you are concerned about an integration test that interfaces with an unreliable external system, then one thing you can do is have a separate testing phase on your automated tests for the external test. This should not fail the whole build, if this phases fails (it may indicate a bug or it may indicate the downstream system has gone down), but indicates something needs investigation.

This requires a degree of discipline that you do investigate these issues and you must ensure your tests using your mocks are as good as possible, because you're not getting the same level of safety from external tests anymore, because they don't fail the build, but it can work reasonably well.