There are a few methods.
First is to lower the number of frames decoded - either by lowering the frame rate on the encoder side, or to decode only keyframes and skip the rest.
Second is to increase the decoding power - add CPUs or add decoding hardware. AFAIK a graphics card can only decode one h264 stream at a time; not sure if this is the software or hardware restriction. So you're unlikely to gain much here.