Monitoring a Twitter stream on a webserver

Question

I wrote a program using Tweepy that I can run on my home computer to monitor a Twitter stream. The way Tweepy does this is basically to have a while(true) loop always running. Now what I want to do is link this program into my Django app. One way I could do this is to leave the Twitter program on the local computer and have it update the database of the server when it receives a message. However, I was wondering what I need to do run the Twitter program in the background on my server. I am using Django.

Answer 1

As you have discovered with your comment, celery is not ideal - you would need another long running process to monitor the existing long running process (your client).

Ideally you need a socket that is always open (like the infinite while loop in your client); so wheverver there is data you get a "realtime" view of the twitter feed on the web page.

Node.js combined with SocketIO is designed to solve this, and since a twitter feed realtime viewer is a common case many examples are floating about - streamie is one of them.

That combination gives you a javascript client that automatically updates based on data coming; no refresh, ajax polling, cron, etc. required.

Answer 2

You can use Celery which is a task queue that can perform tasks in the background (i.e. outside the request/response cycle). It also has django integration with django-celery.

You can set it up to perform periodic tasks, i.e. check the twitter stream every 5 minutes using tweetpy and save the results to the db. Alternatively you could set up a task that fires on an operation i.e. when a user clicks something on the webapp, start a background task to scrape twitter.

If you don't want to go to that kind of trouble, you could alternatively just use Cron to perform an operation every X minutes, or a simplified queue app. That said, Celery is well tested and reliable and probably worth the extra effort