Does parallelGCThreads actually limit the amount of threads such that Amdahl's law should be used with P = 8 and not P = 24?
No. The number of GC threads does not directly affect the performance of your program while doing "real" work. (It might affect it if the program generated lots of garbage, but the analysis would be rather complicated ... and it certainly would not replace P
in the Amdahl equation.)
You could try tweaking the parallelGCThreads parameter to see what effect it has, but there is probably something else (not GC) that is impacting your multi-threaded performance resulting in lower speed-up ratios than you anticipated. Most likely it is something to do with the application; e.g. memory bandwidth contention, or Java lock contention.
FWIW - it is unusual for a multi-threaded program to get close to the theoretical limit set by Amdahl's Law.