Parallel execution: 1 thread pool or N thread pools?

Question

I have a tree generated by configuration.

Each leaf of the tree is a long running task (like DB query, reading from file, etc.) that gives a result I want to store in a mirrored tree with only the results; non-leaf nodes don't have any computation (the structure and height of the tree is given by configuration).

In the current version of the software we are creating a fixed thread pool executor (Java) for each non-leaf nodes and executing the visit inside the thread pool, regardless if the node is a leaf or not (i.e. visiting in parallel multiple nodes). Then for each non-leaf node the visit wait for its own children to end. Writing in pseudocode is a bit like (skipping the saving the result part)

void visit(NonLeafNode node){
    pool = new FixedThreadPool(node.poolSize) //this number is part of the configuration of the node; 
    for (child in children){
         pool.submit(visit(child))
    }
    while(pool is not empty){
         wait()
    }
}

void visit(LeafNode node){
     task()
}

I think that is approach has 2 flaws:

1. There is no real need to visit a tree in parallel (the size is between 10 and 100, but even if it was 1000...)

2. We are creating a dynamic number of threads and executors based on the configuration and that number is probably more than what is needed.

My personal solution is to have one executor, collect all the leaves and then submit them all together (only in the first call and not recursively).

void visit(Root node){
    pool = new FixedThreadPool(poolSize)  //this is global configuration
    tasks = new List<Tasks>()
    for(child in children){
         visit(child, tasks)
    }
    for(task in tasks){
        pool.submit(task)  
    }
    wait
}

void visit(NonLeaf node, tasks){
     for(child in children){
         visit(child, tasks)
     }
}

void visit(Leaf node, tasks){
     tasks.add(() -> task()) // only using the reference of the task
}

I got the comment that

1. In this way we cannot configure how many threads there will be per node

2. This can create deadlocks because the number of threads is limited.

So, in this particular case, but also in general, is better N pools or 1 fixed pool?

Answer 1

There are two basic reasons to run a thread pool.

1. Creating threads as you go is expensive. With a pool, you create them only once, and re-use them as needed, saving the overhead.

2. The optimum number of threads is driven mostly by characteristics of the hardware, often computed as a multiple of the number of CPU cores. There's a reason for this. If the hardware supports 20 threads, it supports 20 threads, regardless of how many you think you need-- adding more can actually be counterproductive. (I could see maybe allocating a dynamic number of tasks based on what your logic requires, but the number of threads that those tasks run on should stay relatively fixed, as there is no point in increasing them beyond their optimum number).

The architectures I'm familiar with involve the creation of one or two thread pools when the application starts, and re-using threads from that pool as needed.

The architecture you are showing me, where pools are allocated dynamically as needed, seems contrary to common practice, and would only make sense if there is something extraordinary about how your application operates. Is there?