How would you mock a web app in Python (for testing a Django project)

Question

My app in Django scraps and imports data from another application's HTML. I tested each parsing function and would like to test the crawler that will go through the other application, too. After this, I'd like to make some integration tests. For making the tests as easy to run as possible, I want to mock the imported web application by creating a little web app that serves some hardcoded HTML and has all the paths I am going to go through.

EDIT: Also, my mock has to have some little dynamic behaviors - for example, for testing both failed and successful logins. So I cannot provide only static files.

How would you create such an mock application? Would you subclass BaseHTTPServer? CGI? Use some framework (as does twill, using Quixote)? Or would be reasonable to use Django for it? That is the solution I am cogitating to use, but Django seems to be too complex for such problem; OTOH, another framework would be a too heavy dependency for such little need, and BaseHTTPServer is just too raw to use.

2nd EDIT: I am not interested on mocking classes, requests etc. etc. That is not the approach I want to use, and a suggestion to use such approach is not an answer to me (Although I am grateful to the nice people who kindly suggested me that until now). If it is too hard to think about my question, just forget that I talked about tests - how would you crudely simulate a web application using Python in general?

Answer 1

I tried to follow @Gagandeep Singh solution. This seemed to be the best one, and probably is a good solution in other situations, but it did not work for me.

The problem is that I had a Django app inside the test directory of another Django app. When I ran the tests of my app with manage.py test myapp, the used settings.py was the one from the whole project, not the file for my mocking app. I was starting Django through the management API and using multiprocessing, so I bet part of my problem came from such a complex interaction. Maybe I could solve it, but I just decided for another strategy.

I decided to override BaseHTTPServer and got some acceptable results. This is not an easy task but I was successful on starting my mocking app.

Answer 2

I think you're mocking at the wrong level. Your unit test shouldn't have to depend on an external webserver at all, even if you're running it specifically for the test. You should be replacing the urllib2.Request object (or whatever you're using that does the actual HTTP call) with one that just returns pre-canned data, including the relevant responses for invalid logins.

Answer 3

I would download the reference pages with wget -r (recursive download), and then made the downloaded pages available as static pages with Apache, Nginx or whatever you're using as a webserver.

Unless you require to see the dynamic changes from your web application...

Answer 4

Sounds like you need to use python mock. This allows you to for example, patch an existing command (which may be calling an external url) and add your own test data to it.

For tests, I feel that you should never be hitting an external service. You should instead provide the data you would expect from that service within a fixture of your own and test that your response handler is doing its job.