multiple webrequests asynchronously in asp.net?

Question

I have a page that needs to combine data from four different webrequests into a single list of items. Currently, I'm running these sequentially, appending to a single list, then binding that list to my repeater.

However, I would like to be able to call these four webrequests asynchronously so that they can run simultaneously and save load time. Unfortunately, all the async tutorials and articles I've seen deal with a single request, using the finished handler to continue processing.

How can I perform the four (this might even increase!) simultaneously, keeping in mind that each result has to be fed into a single list?

many thanks!

EDIT: simplified example of what i'm doing:

var itm1 = Serialize(GetItems(url1));
list.AddRange(itm1);
var itm2 = Serialize(GetItems(url2));
list.AddRange(itm2); 

string GetItems(string url)
{
     var webRequest = WebRequest.Create(url) as HttpWebRequest;
     var response = webRequest.GetResponse() as HttpWebResponse;

     string retval;
     using (var sr = new StreamReader(response.GetResponseStream()))
     { retval = sr.ReadToEnd(); }
     return retval;
} 

Answer 1

This should be really simple since your final data depends on the result of all the four requests.

What you can do is create 4 async delegates each pointing to the appropriate web method. Do a BeginInvoke on all of them. And then use a WaitHandle to wait for all. There is no need to use call backs, in your case, as you do not want to continue while the web methods are being processed, but rather wait till all web methods finish execution.

Only after all web methods are executed, will the code after the wait statement execute. Here you can combine the 4 results.

Here's a sample code I developed for you:

class Program
    {
        delegate string DelegateCallWebMethod(string arg1, string arg2);

        static void Main(string[] args)
        {
            // Create a delegate list to point to the 4 web methods
            // If the web methods have different signatures you can put them in a common method and call web methods from within
            // If that is not possible you can have an List of DelegateCallWebMethod
            DelegateCallWebMethod del = new DelegateCallWebMethod(CallWebMethod);

            // Create list of IAsyncResults and WaitHandles
            List<IAsyncResult> results = new List<IAsyncResult>();
            List<WaitHandle> waitHandles = new List<WaitHandle>();

            // Call the web methods asynchronously and store the results and waithandles for future use
            for (int counter = 0; counter < 4; )
            {
                IAsyncResult result = del.BeginInvoke("Method ", (++counter).ToString(), null, null);
                results.Add(result);
                waitHandles.Add(result.AsyncWaitHandle);
            }

            // Make sure that further processing is halted until all the web methods are executed
            WaitHandle.WaitAll(waitHandles.ToArray());

            // Get the web response
            string webResponse = String.Empty;
            foreach (IAsyncResult result in results)
            {
                DelegateCallWebMethod invokedDel = (result as AsyncResult).AsyncDelegate as DelegateCallWebMethod;
                webResponse += invokedDel.EndInvoke(result);
            }
        }


        // Web method or a class method that sends web requests
        public static string CallWebMethod(string arg1, string arg2)
        {
            // Code that calls the web method and returns the result

            return arg1 + " " + arg2 + " called\n";
        }

    }

Answer 2

How about launching each request on their own separate thread and then appending the results to the list?

Answer 3

you can test this following code:

Parallel.Invoke(() => {
//TODO run your requests... });

You need reference Parallel extensions : http://msdn.microsoft.com/en-us/concurrency/bb896007.aspx

Answer 4

@Josh: Regarding your question about sending 4 (potentially more) asynchronous requests and keeping track of the responses (for example to feed in a list). You can write 4 requests and 4 response handlers, but since you will potentially have more requests, you can write an asynchronous loop. A classic for loop is made of an init, a condition, and an increment. You can break down a classic for loop into a while loop and still be equivalent. Then you make the while loop into a recursive function. Now you can make it asynchronous. I put some sample scripts here at http://asynchronous.me/ . In your case, select the for loop in the options. If you want the requests to be sent in sequence, i.e. one request after the previous response (request1, response1, request2, response2, request3, response3, etc.) then choose Serial communication (i.e. sequential), but the code is a bit more intricate. On the other hand, if you don't care about the order in which the responses are received (random order), then choose Parallel communication (i.e concurrent), the code is more intuitive. In either case, each response will be associated with its corresponding request by an identifier (a simple integer) so you can keep track of them all. The site will give you a sample script. The samples are written in JavaScript but it's applicable to any language. Adapt the script to your language and coding preferences. With that script, your browser will send 4 requests asynchronously, and by the identifier you'll be able to keep track of which request the response corresponds to. Hope this helps. /Thibaud Lopez Schneider