c++ Jobqueue: Is there a rule of thumb for the number of worker threads?

Question

I just implemented a very first simple multithreaded jobqueue in c++ and I thought (and read) that it's a good idea to use one worker thread for each hardware thread (in my case that would be 4). Basically my app just loads alot of images (at the same time) from the internet right now, and I notice that I get a huge speed up if I increase the number of worker threads to 8 or even 16 instead of 4.

Is there a general rule of how many threads to use in such a jobqueue? My guess would be that 4 would be a greate number if I was creating new jobs each frame, and the worker threads had a constant workload each frame, while if I want to process alot of stuff at once (like loading 50 images or so) more threads than that can give a great speed up. Still, is there a rule of thumb for the right number in different scenarios?

Thanks

Answer 1

The ideal number of worker threads is equal to the number of CPU cores in the system. However, in reality this is suboptimal because sometimes tasks can block on the network, disk I/O, etc.. resulting in underutilization. That sounds like what is happening here.

Often times thread pools will "over schedule" to compensate for this. Sometimes there is built in kernel support for notifying you when threads block, so you know to spin up another one (completion ports in Win32) and still hit the optimal number of active worker threads.

Answer 2

Microsoft chose to provide a threadpool implementation (available from the OS, and likewise available in .net) with the number of threads starting at 1.5 times the number of processors. The idea is that a thread that blocks (for disk i/o, etc) can swap out for another thread that is not blocked. The threadpool is also configurable so that you can demand a minimum number of threads, should you decide that having 2x, 3x, or more threads are necessary.

I suspect all of this applies to your situation.

Answer 3

Some while ago I was looking for the answer of that question and I came across an msdn article which stated that it is best to use max 16 thread per core. Unfortunately I can't find the article anymore, but for loading images, 16 threads per core make sense.