Should 'test' environments consume 'test' external services?

Question

We manage an application that consumes a number of external services as part of its general operation. Some services are Soap Services, others Restful Apis. Some services are also managed by us, others are third party services. Some services are central to the application's functionality, others are more auxiliary/non-mandatory.

Each external service exposes a 'test' and 'live' environment. We currently follow the policy that when our application is under test (that's development, testing and staging phases), it should consume the test version of the external service. It is only in our live environment that the live versions of the services are consumed.

There is a not-insignificant amount of overhead in managing which version of the service to consume between environments, but this is not the issue. My question is whether or not this policy is a good idea? Would we be better served instead by always consuming live versions of external services? Have we made the mistake of exposing the test versions of the external services we manage ourselves, i.e. should test environments remain private?

We have not (yet) been burned by not pointing to live external services until the application reaches 'live' but I accept that part of our problem is that we lack the granularity in our environments - by grouping development, testing and stage under the 'test' umbrella, we lose the ability to test against live external services.

All I realise at the moment is that there is little to be gained by consuming the test services in the test environments. There is negligible cost involved in consuming live third-party external services. Also, there is potential impact for our own services to be aware that they are being consumed by a client in the 'test' phase, but this could probably be accommodated.

I understand that the scenario is this somewhat open-ended, but there only seems to be 2 ways to go?

Answer 1

My concern would be accidentally modifying production data when running non-production instances of your application. As soon as you do one SetX(), POST/PUT, insert into/update, what have you, you are up the creek. That's a sneaky kind of bug that can be very hard to find.

If you're strictly consuming, then in theory it doesn't make a difference. In practice, I'd still be concerned. In your position, I'd probably be quite happy to have a non-live option. Otherwise I'd be thinking about stubbing out all those external services.