Parallel functions when I need thread-specific resource

Question

How do I use System.Threading.Tasks.Pararallel functions when I want each thread to initialize and use its own resource? For example, the work I want to do requires an SqlConnection, each thread must have its own. How do I initialize and use such a resource?

A bad solution is:

const int PARALLELISM = 8;
SqlConnection[] connections = ...; // init array of connections
JobObject[] jobs = ...; // init array of jobs
Parallel.For(0, PARALLELISM, mod => {
    for(int j=0; j<jobs.Length;j++)
        if (j % PARALLELISM == mod)
            jobs[j].Do(connections[mod]);
    }
);

The bad thing about this solution is that it doesn't give work to the next available thread, it doesn't let the system decide the degree of parallelism and frankly it's an abuse of the Parallel library.

What I want is:

JobObject[] jobs = ...; // init array of jobs
Parallel.For(0, PARALLELISM, i => {
    SqlConnection connection = // get the connection for the current thread
    jobs[i].Do(connection);
}

Answer 1

First, before I go in to how to do it, let me say that SqlConnection is specifically designed to be created and destroyed quickly and repeatedly. When you dispose of the connection it doss not immediately release the resources, it instead waits to shut down and if a new connection is created using the same connection string before a timeout expires it will re-use the old connection. This is called "Connection Pooling" and it is on by default (you must explicitly set special settings in your connection string to disable it.)

I recommend you just create a new connection inside the function (I also corrected a few errors about how to set the degree of parallelism, you need to pass in a ParallelOptions parameter)

JobObject[] jobs = ...; // init array of jobs
Parallel.For(0, 
             jobs.Length, 
             new ParallelOptions { MaxDegreeOfParallelism = PARALLELISM},
            (i) => 
            {
                using(SqlConnection connection = new SqlConnection(_connectionString)
                {
                    jobs[i].Do(connection);
                }
            });

This also could be re-written as a Parallel.ForEach if you don't need the index i other than to access the array.

JobObject[] jobs = ...; // init array of jobs
Parallel.ForEach(jobs, 
             new ParallelOptions { MaxDegreeOfParallelism = PARALLELISM},
            (job) => 
            {
                using(SqlConnection connection = new SqlConnection(_connectionString)
                {
                    job.Do(connection);
                }
            });

However, to answer your question if you wanted a re-useable resource that was per-thread you need to use this overload that gives you two more functions you pass in that creates and destroys the thread local resources.

JobObject[] jobs = ...; // init array of jobs
Parallel.For(0, 
             jobs.Length,
             new ParallelOptions { MaxDegreeOfParallelism = PARALLELISM},
             () => new SqlConnection(_connectionString), //Thread local init
             (i, loopstate, connection) => 
             {
                 jobs[i].Do(connection);
                 return connection; //Passes the object to the next function that will re-use it.
             },
             (connection) => connection.Dispose()); //thread local finally.

Once again, I do not recommend you do it this way and instead do it the first way using the short lived SqlConnection, the class is designed to be used that way.