ASP.NET HttpWebRequest freeze other page

Question

A little introduction:

The HTTP protocol is a request/response protocol. This means that the client sends a request, the server performs some processing and returns a response.

If in the course of processing the request the server needs to make another request to a remote resource and use the results of this request into its response, you have no way but waiting.

This being said you could optimize it by taking advantage of asynchronous pages and I/O Completion Ports. This basically will allow to free the worker thread during the execution of the remote HTTP request.

Don't be fooled into thinking though that this will make the response come faster. It will just not jeopardize your worker threads if the remote server is slow. This means that your application will be able to service other requests.

In order to achieve that you could use the BeginGetResponse/EndGetResponse methods of the HttpWebRequest. They allow for an asynchronous execution taking advantage of I/O Completion Ports.

In the article linked previously in my answer you will find how to make your ASP.NET Page (or if you are using a handler) asynchronous.

The first step is to mark your WebForm async:

<%@ Page Async="true" ... %>

and then you could register asynchronous tasks using the AddOnPreRenderCompleteAsync method:

using System;
using System.IO;
using System.Net;
using System.Web;

public partial class WebForm1 : System.Web.UI.Page
{
    private HttpWebRequest request;

    protected void Page_Load(object sender, EventArgs e)
    {
        this.AddOnPreRenderCompleteAsync(
            new BeginEventHandler(this.BeginAsyncOperation), 
            new EndEventHandler(this.EndAsyncOperation)
        );
    } 

    private IAsyncResult BeginAsyncOperation(object sender, EventArgs e, AsyncCallback cb, object state) 
    {
        string url = "http://example.com";
        string method = "GET";

        this.request = (HttpWebRequest)WebRequest.Create(url);
        this.request.Method = method;

        this.request.Accept = "text/javascript, text/html, application/xml, text/xml, */*";
        this.request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
        this.request.Host = "steamcommunity.com";
        this.request.UserAgent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11";

        return this.request.BeginGetResponse(cb, state); 
    } 

    private void EndAsyncOperation(IAsyncResult ar) 
    { 
        using (WebResponse response = this.request.EndGetResponse(ar))
        using (StreamReader reader = new StreamReader(response.GetResponseStream())) 
        { 
            string result = reader.ReadToEnd(); 
            // TODO: do something with the result here => like binding
            // it to some Web Control or displaying it somewhere on the page
        } 
    }
}

As an alternative to this approach you might consider loading your page faster, and then trigger an AJAX request to another handler which will perform the remote request and return the result. In this case, the initially request page will load faster and not be freezed, but the results will come later, once the remote request has finished processing.

But in both cases I would recommend you taking advantage of asynchronous pages to avoid blocking your worker thread which could be catastrophic to your application especially if for some reason the remote server you are accessing is slow.