ConcurrentQueue Operation is giving OutofMemory Error.

Question

I understand this question may be too general. But I tried many things and I am not able to figure out how to resolve this.

I am using ConcurrentQueue for multithreading operation. One thread is downloading Images from server and saving it to queue. Here is the code for that:

 public static void DownloadImage()
    {
        string baseUrl = "http://someurl";
        //int numIterations = 5;

        HttpWebRequest request = null;
        foreach (var fileName in fileNames)
        {
                string url = string.Format(baseUrl, fileName);
                request = (HttpWebRequest)WebRequest.Create(url);
                request.Method = "GET";
                request.ContentType = "application/x-www-form-urlencoded";
                var response = (HttpWebResponse)request.GetResponse();
                Stream stream = response.GetResponseStream();
                img = Image.FromStream(stream);
                ImageFileName FileNameImage = new ImageFileName(fileName, img);
                ImageQueue.Enqueue(FileNameImage);
                Console.WriteLine("Count after Enqueue: {0}", ImageQueue.Count);

         }

And another thread takes images from Queue and saves them on destination folder. Here is the code for that:

public static void SaveImage()
    {
        while (true)
        {
            if (!ImageQueue.IsEmpty)
            {
                foreach (var newobject2 in ImageQueue)
                {

                    Image img2 = newobject2.Image;
                    img2.Save("C:\\path" + newobject2.ImageName);
                    ZoomThumbnail = img2;
                    ZoomSmall = img2;
                    ZoomLarge = img2;

                    ZoomThumbnail = GenerateThumbnail(ZoomThumbnail, 86, false);
                    ZoomSmall = GenerateThumbnail(ZoomSmall, 400, false);
                    ZoomLarge = GenerateThumbnail(ZoomLarge, 1200, false);

                    ZoomThumbnail.Save("C:\\path" + newobject2.ImageName + "_Thumb.jpg");
                    ZoomSmall.Save("C:\\path" + newobject2.ImageName + "_ZoomSmall.jpg");
                    ZoomLarge.Save("C:\\path" + newobject2.ImageName + "_ZoomLarge.jpg");
                    ImageFileName imgobject3 = new ImageFileName();
                    ImageQueue.TryDequeue(out imgobject3);
                    Console.WriteLine("Count after Deque: {0}", ImageQueue.Count);

                }


            }


        }

    }

I am calling these two threads from Button_Click() like this:

Thread DownloadThread = new Thread(DownloadImage);
  DownloadThread.Start();
  Thread SaveThread = new Thread(SaveImage);
  SaveThread.Start();

I am getting MemoryFull error whenever queue reaches count of 68. I am not sure how I can avoid that. I have tried using Thread.Sleep to avoid this. For Example, I tried: Thread.Sleep(500) after foreach loop. Whenever I try it inside foreach it works totally fine as at any given point ImageQueue.Count = 1. Where I am getting wrong?

Answer 1

You never actually remove your images from the queue. You iterate through them, but you never dequeue them. You also have the problem that when enumerating the queue the enumerable will stop whenever there are currently no more items in the queue. You don't want to do this, you may not be done yet.

What you want to do is use a BlockingCollection instead of a ConcurrentQueue, so that if there are currently no more items you wait for more, rather than stopping. Once you do this, you can use foreach(var item in queue.GetConsumingEnumerable()). This makes all of the needed changes. It removes the items from the queue as you iterate it, and it waits for more items if there are currently none.

In addition to making this change, as is mentioned in Guffa's answer, you need to dispose of your image objects. You can dispose of them at the end of your loop body, once you're done with the image.

Answer 2

You should dispose after you are done with the image so after the ZoomLarge statement because after that you dont make any use of it anymore. With the Generate statements you create new images.

Answer 3

You might want to look at the BlockingCollection class (here), which I've successfully used for this kind of "producer-consumer" pattern.

Something (a producer) puts items into the collection, with one or more "consumer" b/g threads taking the next available item(s) out of the collection and doing something with them. You can also configure the number of consumer threads to use (e.g. for a multi-core PC).

Answer 4

You are creating a lot of Image objects that you don't dispose. You should call Dispose on each of the thumbnails that you create.

Also, the object that you dequeue contains an Image object that you should dispose when you don't need it any more.

---

Side note: You shouldn't enumerate the items in the queue and deqeue the item inside the loop. Instead use the TryDequeue method in the loop itself:

ImageFileName imgobject2;
while (ImageQueue.TryDequeue(out imgobject2)) {
  ...
}

---

Another side note: Your code will go into a tight loop when the queue is empty, using a lot of CPU power to repeatedly make the same check very rapidly. You should use a mechanism to suspend the thread until something happens with the queue, or at least sleep for a while before checking the queue again.

Answer 5

1.It seems that your are keeping a reference to all your images which keeps them in memory, not allowing the garbage collector to collect them.

Also, after you've finished using your objects, you can explictly dispose them, like this:

Image img2 = newobject2.Image;
img2.Dispose();

2. What your are doing is enumerating over all your images everytime you are in a while loop, i am not sure this is what you want. What you should do is try to dequeue an item:

FileImageName myImage;
ImageQueue.TryDequeue(out myImage);

if (myImage != null)
{
// Do work on the image
}