Web server using python

Question

I am trying to develop a multithreaded web server, it has the following task:

1. Collect data from various data sources (API calls), I was planning to do this using multiple threads.

2. Store the collected data in a memory data structure

3. Do some processing on the data structure using another thread

4. This data structure would be queried by the multiple clients; maybe I could also make separate threads for each client request.

Now regarding language and platform, I was considering either python or JAVA. I did some research on Flask framework for python, but I do not know how it will accommodate the multithreaded nature of web server. Please suggest how I could achieve the above functionality in my project.

Answer 1

Flask, with some of the available addons, is very suited for what you want to do. Keep in mind that flask is pure python, and therefore you can access any of the excellent available python libraries.

As far as I understand what you have in mind, you can:

1- define a url that, when visited, executes the data gathering from external sources by means of, e.g. python-requests (http://docs.python-requests.org/en/latest/)

2- do the same periodically by scheduling the function above

3- store the collected data in a (e.g.) Redis database (which is memory based) or one of the many available databases (all of the nosql dbs have python bindings that you can access from a flask application)

4- define urls for the visiting clients to access the latest versions of the data. You will just need to define the data extraction functions (from redis or whatever you decide to use) and design a nice template to show them.

Flask/Werkzeug will take care of the multithreading necessary to handle simultaneous requests from different clients.