I suspect the problem is due to one thread running at 100% on one core. Due to hyper-threading this is really consuming two threads. You need to find the core that is causing this and try and exclude it. Let's assume it's threads 20 and 21 (you said it starts at 20 in your question - are you sure about this?). Try something like this
GOMP_CPU_AFFINITY = 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 22 23
I have never used this before so you might need to read up on this a bit to get it right.
OpenMP and CPU affinity You might need to list the even threads first and then odd (e.g. 0 2 4 ... 22 1 3 5 ...) in which case I'm not sure what to exclude (Edit: the solution was: export GOMP_CPU_AFFINITY="0-17 20-24
. See the comments).
As to why 26 threads would not have the problem I can only guess. OpenMP can choose to migrate the threads to different cores. Your system can run 24 logical threads. I have never found a reason to set the number of threads to a value larger than the number of logical threads (in fact in my matrix multiplication code I set the number of threads to the number of physical cores since hyper-threading gives a worse result). Maybe when you set the number of threads to a value larger than the number of logical cores OpenMP decides it's okay to migrate threads when it chooses. If it migrates your threads away from the core running at 100% then the problem could go away. You might be able to test this by disabling thread migration with OMP_PROC_BIND