Parallel.ForEach() missing list items

Question

In my tests without using Parallel.ForEach(), 3500 records are processed.

But with Parellel a random lower number of records are processed each attempt. (3402, 3375, 3471, etc). I figured out that DBSet didn't support parellel programming, so I am moving my items into a list, then dealing with the list later. (not ideal) Some of the records that are missed, are added to the List as null objects.

What am I missing?

       public int Execute(CalculateAutoClassRatesCommandRequest request)
        {
            int recordCount = 0;
            var autoClassCalculations = new List<AutoClassCalculation>();
            var options = new ParallelOptions { MaxDegreeOfParallelism = 2 };
            Parallel.ForEach(request.BaseCalculations, options, baseCalculation =>
                {
                    foreach (var autoClass in request.AutoClasses
                        .Where(autoClass => baseCalculation.BaseClassId == autoClass.BaseClassId)
                        .Where(autoClass => autoClass.AutoClassForms != null))
                    {
                        foreach (var form in autoClass.AutoClassForms)
                        {
                            recordCount++;
                            var calcRate = CreateAutoClassCalculation(autoClass.AutoClassRoundingRules, form, baseCalculation); 
autoClassCalculations.Add(calcRate);
                        }
                    }
                });

            foreach (var autoClassCalculation in autoClassCalculations)
            {
                request.AutoClassCalculationsDbSet.Add(autoClassCalculation);
            }
            request.ratingContext.SaveChangesWithChangeTracking(request.Identity);

            return recordCount;
        }

private static AutoClassCalculation CreateAutoClassCalculation(
    IEnumerable<AutoClassRoundingRule> autoClassRoundingRules, 
    AutoClassForm autoClassForm,
    BaseCalculation baseCalculation)
{
    var autoClassCalculation = new AutoClassCalculation()
    {   
        BaseCalculation = baseCalculation, 
        AutoClassForm = autoClassForm, 
        BaseCalculationId = baseCalculation.BaseCalculationId, 
        AutoClassFormId = autoClassForm.AutoClassFormId
    };

    var roundingRule = autoClassRoundingRules != null ? autoClassRoundingRules.FindMatchingAutoClassRoundingRule(autoClassForm.AutoClassId, autoClassCalculation.CalcRate) : new AutoClassRoundingRule();
    autoClassCalculation.CalculateRate(roundingRule);

    return autoClassCalculation;
}

Answer 1

There is one problem and one potential problem that I see in your code:

---

All threads are accessing recordCount variable directly. Although num++ may look like an atomic operation, it is not. It is essentially equivalent to num = num + 1. Since this is executed in several CPU instructions, it is possible that a thread takes a value, and gets suspended before updating the value. In mean time, other threads increase the value, and once initial thread is resumed, it just places value it calculated based on old starting value. Instead of using incrementing operator, you should use Interlocked.Increment() function.

Interlocked.Increment(ref recordCount);

---

Enumerator of your collection may not be thread safe. If you are using a custom collection type parallel access to elements may iterate over them faster than you expect. This shouldn't cause problem you described though. It would be wise that you use BlockingCollection<T>.