Sometimes using -fdebug-types-section can make things a bit faster. It isn't guaranteed though.
Several minutes to load ... I wonder how big this executable is. If I were desperate I might try only compiling selected modules with debug info. Or perhaps look to see if it is a gdb bug. If it is split into an executable and some shared libraries, and some parts don't change very often, you could also look into using the "gdb index" feature (see the manual) to speed up the loading of debuginfo for those modules.