Best-practice: automated web API testing

Question

I've written a program in Python, which works with two distinct API to get the data from two different services (CKAN and MediaWiki). In particular, there is a class Resource, which requests the data from the above mentioned services and process it.

At some point I've come to conclusion, that there is a need for tests for my app. And the problem is that all examples I've found on web and in books do not deal with such cases.

For example, inside Resource class I've got a method:

def load_from_ckan(self):
    """
        Get the resource
        specified by self.id
        from config.ckan_api_url 
    """
    data = json.dumps({'id': self.id})
    headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
    url = config.ckan_api_url + '/action/resource_show'
    r = requests.post(url, timeout=config.ckan_request_timeout, data=data, headers=headers)
    assert r.ok, r
    resource = json.loads(r.content)
    resource = resource["result"]
    for key in resource:
        setattr(self, key, resource[key])

The load_from_ckan method get the data about resource from the CKAN API and assign it to the object. It is simple, but...

My question is: how to test the methods like this? OR What should I test here?

I thought about the possibility to pickle (save) results to HDD. Then I could load it in the test and compare with the object initialized with load_from_ckan(). But CKAN is community-driven platform and such behavior of such tests will be unpredictable.

If there exist any books on philosophy of automated tests (like what to test, what not to test, how to make tests meaningful etc.), please, give me a link to it.

Answer 1

With any testing, the key question really is - what could go wrong?

In your case, it looks like the three risks are:

• The web API in question could stop working. But you check for this already, with assert r.ok.
• You, or someone else, could make a mistaken change to the code in future (e.g. mistyping a variable) which breaks it.
• The API could change, so that it no longer returns the fields or the format you need.

It feels like you could write a fairly simple test for the latter two, depending on what data from this API you actually rely on: for example, if you're expecting the JSON to have a field called "temperature" which is a floating-point Celsius number, you could write a test which calls your function, then checks that self.temperature is an instance of 'float' and is within a sensible range of values (-30 to 50?). That should leave you confident that both the API and your function are working as designed.

Answer 2

Typically if you want to test against some external service like this you will need to use a mock/dummy object to fake the api of the external service. This must be configurable at run-time either via the method's arguments or the class's constructor or another type of indirection. Another more complex option would be to monkey patch globals during testing, like "import requests; request.post = fake_post", but that can create more problems.

So for example your method could take an argument like so:

def load_from_ckan(self, post=requests.post):
    # ...
    r = post(url, timeout=config.ckan_request_timeout, data=data,
        headers=headers)
    # ...

Then during testing your would write your own post function that returned json results you'd see coming back from ckan. For example:

 def mock_post(url, timeout=30, data='', headers=None):
     # ... probably check input arguments
     class DummyResponse:
         pass
     r = DummyResponse()
     r.ok = True
     r.content = json.dumps({'result': {'attr1': 1, 'attr2': 2}})
     return r

Constructing the result in your test gives you a lot more flexibility than pickling results and returning them because you can fabricate error conditions or focus in on specific formats your code might not expect but you know could exist.

Overall you can see how complicated this could become so I would only start adding this sort of testing if you are experiencing repeated errors or other difficulties. This will just more code you have to maintain.

Answer 3

At this point, you can test that the response from CKAN is properly parsed. So you can pull the JSON from CKAN and ensure that it's returning data with the attributes you're interested in.