C# Link Analyzer getting Bad Request Errors?

Question

I have a rather simple program which takes in a URL and spits out the first place it redirects to. Anyhow, I've been testing it on some links and noticed gets 400 errors on some urls. I tried testing such urls by pasting it into my browser and that worked fine.

    static string getLoc(string curLoc, out string StatusDescription, int timeoutmillseconds)
    {
        HttpWebRequest x = (HttpWebRequest)WebRequest.Create(curLoc);            
        x.UserAgent = "Opera/9.52 (Windows NT 6.0; U; en)";
        x.Timeout = timeoutmillseconds;            
        x.AllowAutoRedirect = false;
        HttpWebResponse y = null;
        try
        {
            y = (HttpWebResponse)x.GetResponse(); //At this point it throws a 400 bad request exception.

Answer 1

I think something weird is happening with cookies. It turns out that due to the way I was testing the link, the necessary cookies for it to work were in my browser but not the link. It turns out some of the links I was testing manually (when the other links failed) were generating cookies.

It's slightly convoluted what happened but the short answer is that my browser had cookies, the program did not, maintaining the cookies between redirects did not solve the problem.

The underlying problem is caused by the fact that the link I am testing requires either an extra parameter or a cookie or both. I was trying to avoid both in my tests since the parameter/cookie were for tracking and I didn't want to break tracking.

In short, I know what the problem is but it's not a solvable problem.