Rss aggregator on Azure - best way

Question

My application(backend) runs on Windows Azure. Application must read rss feeds, and write information to the database.A lot of questions, but these basic.

1.What is the best way for organisation application architecture?

2.How can I get updates from the feeds(periodic request or use PubSubHubbub)?

3.How do I know that the channel support PubSubHubbub?

Thanks for any information

Answer 1

The question comes down to what RSS feeds you are aggregating and I suspect you might have RSS channels that do and don't support PuSH.

If you are a large RSS aggregator then I would say you should support both PuSH and getting the requests periodically but while you are small it comes down to one thing, your time to code and maintain.

If you have the possibility of non-PuSH compatible RSS feeds then I say code only for the periodic update. You can code in PuSH later if needed.

In terms of architecture it depends on your scalability requirements. I personally like this approach

1. Azure Scheduler > Message to Storage Queue

   • This initiates a process that determines all the sites that need their RSS feed updated.
   • Push to Azure Service Bus the site that needs checking

2. Azure Service Bus > Process (goes to site and updates RSS feed)

Keeping all of this on service bus means you can expand out your servers as needed and they all read off the same queue but it splits the workload.

In your database keep an status of which ones support PuSH. At a later date you can then change your method of getting these RSS feeds and only put in the queue ones that don't support PuSH.