In the end, this turned out to be related to the combination of our AV and a specific dbghelp.dll version.
Tested combinations
- AV Enabled with dbghelp version 6.0 is way to slow
- AV Disabled with dbghelp version 6.0 is no problem
- AV Enabled with dbghelp version 6.3 is no problem