How to transfer content from httpWebRequest from Web Role to a Worker Role in Microsoft Azure

Question

Help a newbie out please:

What I have in mind is this:

1. Web role receives an http request. (done)
2. the url supposed to help determine particular instance of the worker role. (done)
3. the content of the request supposed to go to a worker role for [processing (it can be quite large and likely of type xml or json or blob or something else)
4. worker role receives the data
5. worker role parses/processes the data from the content and does some calcs. (done)
6. worker role sends a response back to the web role. (likely a string or something similar to the data in the request: can be quite large as well).

Items #3 and #4 is where I have most of the problem with: not sure what do do here.
Item #6 i believe i have SOME idea on what to do, but would love to hear your suggestions

What I know is this:

• I cant use queue: there is a size limitation I believe, which will prevent me from using it.
• I dont think I will be able to use shared storage because how the web role is right now - long story
• That leaves me with using endpoint: unless someone tells that this is absolutely impossible, then I will have to think of using the shared storage.

Appreciate all your inputs in advance.

Answer 1

You can certainly find out an internal endpoint of a worker role instance from a web role instance using code. See this thread for more details: How to get web role input endpoint in worker role?.

However I see one problem with your approach. You mentioned that web role will communicate with worker role using an internal endpoint to send the data and then wait for worker role to process that data and send the response back. While technically its possible to do so, it would cause some scalability issues because you're now tying up your web role. Furthermore, you have to decide in your code which worker role instance should process the data (assuming there're more than one instances).

My recommendation would be to get the request and persist the request data in blob storage. You will get a URL of the blob. Then write a message in a queue with the URL of the blob. Once the message is written to the queue, your web role can notify the user that the request has been submitted. Worker role instances would poll this queue periodically and work on the message. They will get the blob URL from message contents, fetch the contents and work on it. Once the work is processed, worker role could write to a queue/table indicating that the work is completed. Web role could constantly poll this table or queue and once the task is completed it can notify the user.