WebClient with credentials still not downloading file

Question

I am trying to download files from a website with username/password. You need to pay for a registered account in order to download files - which we have done. I am attempting to pass in the username/password and download a file as follows:

if (docUrl != null)
            {
                if (!string.IsNullOrEmpty(username) && !string.IsNullOrEmpty(password))
                    this.WebClientInstance.Credentials = new NetworkCredential(username, password);

                fileData = this.WebClientInstance.DownloadData(docUrl);
                this.WebClientInstance.Dispose();
                isDataDownloaded = true;
            }

WebClientInstance is a System.Net.WebClient. I debugged and verified that it is hitting the line to set credentials. Instead of downloading the PDF, I end up with an HTML page that prompts me to log in to get access to the file. I have verified that the username/password is correct. I use the same credentials to scrape the website with WatiN.

Is there something else that I'm supposed to be doing here?

UPDATE

Okay, I've done some sniffing around and found some useful info on this issue. I still haven't gotten it to work, but I think I'm closer. First, you need to create a cookie aware WebClient that extends the WebClient class, as follows:

public class CookiesAwareWebClient : WebClient
{
    public CookieContainer CookieContainer { get; private set; }

    public CookiesAwareWebClient()
    {
        this.CookieContainer = new CookieContainer();
    }

    protected override WebRequest GetWebRequest(Uri address)
    {
        var webRequest = base.GetWebRequest(address);

        if (webRequest is HttpWebRequest)
            (webRequest as HttpWebRequest).CookieContainer = this.CookieContainer;

        return webRequest;
    }
}

Next is to use the WebClient.UploadValues() method to upload the login info to the target website. The full process of authenticating and downloading the target resource is as follows:

using (var webClient = new CookiesAwareWebClient())
                    {
                        var postData = new NameValueCollection()
                        {
                            { "userId", username },
                            { "password", password }
                        };

                        webClient.UploadValues(docUrl, postData);

                        fileData = webClient.DownloadData(docUrl);
                    }

I was wrong about the site using forms auth. It is a JSP website and uses a JSESSIONID. I have verified that I am getting a cookie back with what appears to be a valid 32-byte JSESSIONID value.

However, when I call WebClient.DownloadData() it is still only returning the redirected login page. I've tried to fix this by setting the AllowAutoRedirect property on the HttpWebRequest to false, but then it returns 0 bytes.

Is there something else that I need to do so it won't redirect and will take me to the resource once I have authenticated?

Answer 1

(Answered in a question edit. Converted to a community wiki answer. See Question with no answers, but issue solved in the comments (or extended in chat) )

The OP wrote:

Solved. So the problem was between my ears. I was passing in the URL for the secure resource to the .UploadValues() method, knowing that it would redirect to the login page. However, I really needed to pass in the URL from the login form (where it goes upon submitting) - not the login page itself. Once I did that, it worked correctly. I think I'm going to go find a career in food service now.

LINKS

There were already a few questions posted on SO that addressed this issue. I just didn't know what I was looking for at first so I didn't see those... Anywhere here are a couple good resources that I came across when working on this issue:

how to maintaine cookies in between two Url's in asp.net

Trying to get authentication cookie(s) using HttpWebRequest