C# program stops processing while making Web Request

Question

I'm very new to C# and am playing around with writing a basic program. The goal is to grab an image and save it to the disk given a URL.

This is where my code essentially times out:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(m.Value.Trim()); // url string passed in from Regex function
HttpWebResponse resp = (HttpWebResponse) req.GetResponse(); // times out here

The code is in a class containing one function, which runs as such:

 String url; // passed in as a parameter
 String folder = @"C:\SMBC";
 // create directory if not exists to save comic
 if(!Directory.Exists(folder)) 
     Directory.CreateDirectory(folder);

     // visit the site and check for comics
     HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
     request.Method = "GET";
     HttpWebResponse response = (HttpWebResponse)request.GetResponse();
     using (StreamReader reader = new StreamReader(response.GetResponseStream()))
        {
            while (reader.Peek() >= 0)
            {
                // if line contains "/comics/" then I'm in the right spot, we're at a URL now
                String line = reader.ReadLine();
                if (line.Contains("/comics/"))
                {
                    // pull out the address of the image // example result: http://www.smbc-comics.com/comics/20020905-2.gif
                    Regex linkParser = new Regex(@"\b(?:https?://|www\.)\S+\b", RegexOptions.Compiled | RegexOptions.IgnoreCase);
                    foreach (Match m in linkParser.Matches(line))
                    {
                        // new local file in folder, use original file name
                        String name = @"" + folder +"\\" + m.Value.Substring(m.Value.LastIndexOf("/") + 1);
                        Console.WriteLine(m.Value.Trim()); //http://www.smbc-comics.com/comics/20020905-2.gif
                        Uri uri = new Uri(m.Value.Trim());
                        
                        Console.WriteLine("Making request"); // works 
                        HttpWebRequest req = (HttpWebRequest)WebRequest.Create(m.Value.Trim());
                        Console.WriteLine("getting response"); // works
                        HttpWebResponse resp = (HttpWebResponse) req.GetResponse();
                        Console.WriteLine("opening stream"); // never shows
                        using (Stream inputStream = resp.GetResponseStream())
                        using (Stream outputStream = File.OpenWrite(name))
                        {
                            byte[] buffer = new byte[4096];
                            int bytesRead;
                            do
                            {
                                bytesRead = inputStream.Read(buffer, 0, buffer.Length);
                                outputStream.Write(buffer, 0, bytesRead);
                            } while (bytesRead != 0);
                            //outputStream.Close();
                            //inputStream.Close();
                            //resp.Close();
                        }
                    }
                }
            }
        }

I am running this in the console and here's what I get:

After a while, the operation just times out, I know that the address is valid, because I visit it.

Is there something I am missing?

Answer 1

I think you're hitting the ServicePoint.ConnectionLimit. This can be increased by tweaking ServicePointManager.DefaultConnectionLimit before you start. Don't forget to .Dispose of everything that can be disposed in the smallest possible scope... using statements are good. If you don't close/dispose of WebRequest related stuff, you can eat into your connection limit even if you believe that the request is finished.

Answer 2

As a newbie, I was unaware that the framework let a limit on the number of connections allowed at any given time.

So you have to override this in case you have loops within loops of connections.

ServicePointManager.DefaultConnectionLimit = 4;