What is the recommended implementation architecture for concurrent data input/output in Azure

Question

The worker role takes an approx 10k - 100k data processes it as a unit and produces an also 10k - 100k data (as a unit). I would like to make this concurrent, and scale it by running this worker role in multiple instances.

As far as I know the best approach to provide the input data in a queue, and the worker roles will concurrently consume the messages from this queue. However I am not sure what is the best solution for the output data?

The input data is available in a Windows desktop computer. I suppose I can write a simple .NET application for "upload" the data to the queue (send a message). (Please confirm)

Can this approach work also in the reverse direction for the output data? (I mean the worker role put the output data to the "output" queue and a .NET application running on the Windows desktop "downloads" (read the message from the queue) it?

Thx in advance

Answer 1

That's exactly what queue's are for. I can't think of any platform that isn't supported by Azure Queue's (directly, or indirectly by using the REST interface).

So, yes, "uploading" data from a desktop app is recommended and the same or similar .NET application to "download" data is also recommended.

There are a few examples (e.g. http://www.windowsazure.com/en-us/documentation/articles/storage-dotnet-how-to-use-queues-20/) that show desktops accessing a queue in both directions.