How do I avoid out of memory exceptions when using PLINQ?

Question

Hi and thanks for looking!

Background

I have a computing task that requires either a lot of time, or parallel computing.

Specifically, I need to loop through a list of about 50 images, Base64 encode them, and then calculate the Levenshtein distance between each newly encoded item and values in an XML file containing about 2000 Base64 string-encoded images in order to find the string in the XML file that has the smallest Lev. Distance from the benchmark string.

A regular foreach loop works, but is too slow so I have chosen to use PLINQ to take advantage of my Core i7 multi-core processor:

Parallel.ForEach(candidates, item => findImage(total,currentWinner,benchmark,item));

The task starts brilliantly, racing along at high speed, but then I get an "Out of Memory" exception.

I am using C#, .NET 4, Forms App.

Question

How do I tweak my PLINQ code so that I don't run out of available memory?

Update/Sample Code

Here is the method that is called to iniate the PLINQ foreach:

private void btnGo_Click(object sender, EventArgs e)
{
    XDocument doc = XDocument.Load(@"C:\Foo.xml");
    var imagesNode = doc.Element("images").Elements("image"); //Each "image" node contains a Base64 encoded string.
    string benchmark = tbData.Text; //A Base64 encoded string.
    IEnumerable<XElement> candidates = imagesNode;

    currentWinner = 1000000; //Set the "Current" low score to a million and bubble lower scores into it's place iteratively.
    
    Parallel.ForEach(candidates, i => {
        dist = Levenshtein(benchmark, i.Element("score").Value);
        if (dist < currentWinner)
        {
            currentWinner = dist;
            path = i.Element("path").Value;
        }
    });
}

. . .and here is the Levenshtein Distance Method:

public static int Levenshtein(string s, string t)    {
            int n = s.Length;
            int m = t.Length;
            var d = new int[n + 1, m + 1];

            // Step 1
            if (n == 0)
            {
                return m;
            }

            if (m == 0)
            {
                return n;
            }

            // Step 2
            for (int i = 0; i <= n; d[i, 0] = i++)
            {
            }

            for (int j = 0; j <= m; d[0, j] = j++)
            {
            }

            // Step 3
            for (int i = 1; i <= n; i++)
            {
                //Step 4
                for (int j = 1; j <= m; j++)
                {
                // Step 5
                int cost = (t[j - 1] == s[i - 1]) ? 0 : 1;

                // Step 6
                d[i, j] = Math.Min(
                    Math.Min(d[i - 1, j] + 1, d[i, j - 1] + 1),
                    d[i - 1, j - 1] + cost);
                }
            }
            // Step 7
            return d[n, m];
            }

Thanks in advance!

Answer 1

Update

Ran into this error again today under different circumstances. I was working on a desktop app with high memory demand. Make sure that you have set the project for 64-bit architecture to access all available memory. My project was set on x86 by default and so I kept getting out of memory exceptions. Of course, this only works if you can count on 64-bit processors for your deployment.

End Update

After struggling a bit with this it appears to be operator error:

I was making calls to the UI thread from the parallel threads in order to update progress labels, but I was not doing it in a thread-safe way.

Additionally, I was running the app without the debugger, so there was an uncaught exception each time the code attempted to update the UI thread from a parallel thread which caused the overflow.

Without being an expert on PLINQ, I am guessing that it handles all of the low-level allocation stuff for you as long as you don't make a goofy smelly code error like this one.

Hope this helps someone else.