Running multiple threads, starting new one as another finishes

Question

I have an application that has many cases. Each case has many multipage tif files. I need to covert the tf files to pdf file. Since there are so many file, I thought I could thread the conversion process. I'm currently limiting the process to ten conversions at a time (i.e ten treads). When one conversion completes, another should start.

This is the current setup I'm using.

private void ConvertFiles()
{
  List<AutoResetEvent> semaphores = new List<AutoResetEvet>();
  foreach(String fileName in filesToConvert)
  {
    String file = fileName;

    if(semaphores.Count >= 10)
    {
      WaitHandle.WaitAny(semaphores.ToArray());
    }


    AutoResetEvent semaphore = new AutoResetEvent(false);
    semaphores.Add(semaphore);

    ThreadPool.QueueUserWorkItem(
      delegate
      { 
        Convert(file);
        semaphore.Set();
        semaphores.Remove(semaphore);
      }, null);
  }

  if(semaphores.Count > 0)
  {
    WaitHandle.WaitAll(semaphores.ToArray());
  }
}

Using this, sometimes results in an exception stating the WaitHandle.WaitAll() or WaitHandle.WaitAny() array parameters must not exceed a length of 65. What am I doing wrong in this approach and how can I correct it?

Answer 1

There are a few problems with what you have written.

1st, it isn't thread safe. You have multiple threads adding, removing and waiting on the array of AutoResetEvents. The individual elements of the List can be accessed on separate threads, but anything that adds, removes, or checks all elements (like the WaitAny call), need to do so inside of a lock.

2nd, there is no guarantee that your code will only process 10 files at a time. The code between when the size of the List is checked, and the point where a new item is added is open for multiple threads to get through.

3rd, there is potential for the threads started in the QueueUserWorkItem to convert the same file. Without capturing the fileName inside the loop, the thread that converts the file will use whatever value is in fileName when it executes, NOT whatever was in fileName when you called QueueUserWorkItem.

This codeproject article should point you in the right direction for what you are trying to do: http://www.codeproject.com/KB/threads/SchedulingEngine.aspx

EDIT:

var semaphores = new List<AutoResetEvent>();
        foreach (String fileName in filesToConvert)
        {
            String file = fileName;
            AutoResetEvent[] array;
            lock (semaphores)
            {
                array = semaphores.ToArray();
            }
            if (array.Count() >= 10)
            {
                WaitHandle.WaitAny(array);
            }

            var semaphore = new AutoResetEvent(false);
            lock (semaphores)
            {
                semaphores.Add(semaphore);
            }
            ThreadPool.QueueUserWorkItem(
              delegate
              {
                  Convert(file);
                  lock (semaphores)
                  {
                      semaphores.Remove(semaphore);
                  }
                  semaphore.Set();
              }, null);
        }

Personally, I don't think I'd do it this way...but, working with the code you have, this should work.

Answer 2

Are you using a real semaphore (System.Threading)? When using semaphores, you typically allocate your max resources and it'll block for you automatically (as you add & release). You can go with the WaitAny approach, but I'm getting the feeling that you've chosen the more difficult route.

Answer 3

Looks like you need to remove the handle the triggered the WaitAny function to proceed

if(semaphores.Count >= 10)
{
  int index = WaitHandle.WaitAny(semaphores.ToArray());
  semaphores.RemoveAt(index);
}

So basically I would remove the:

semaphores.Remove(semaphore);

call from the thread and use the above to remove the signaled event and see if that works.

Answer 4

Maybe you shouldn't create so many events?

// input
var filesToConvert = new List<string>();
Action<string> Convert = Console.WriteLine;

// limit
const int MaxThreadsCount = 10;

var fileConverted = new AutoResetEvent(false);
long threadsCount = 0;

// start
foreach (var file in filesToConvert) {
    if (threadsCount++ > MaxThreadsCount) // reached max threads count 
        fileConverted.WaitOne();          // wait for one of started threads

    Interlocked.Increment(ref threadsCount);

    ThreadPool.QueueUserWorkItem(
        delegate {
            Convert(file);

            Interlocked.Decrement(ref threadsCount);
            fileConverted.Set();
        });
}

// wait
while (Interlocked.Read(ref threadsCount) > 0) // paranoia?
    fileConverted.WaitOne();