You have 6 frustums, but note that
- you don't need to check the back or front faces, just whether it is in the infinite pyramid
- each face is shared by two neighbouring frustums
- each face is mirrored by the frustum that faces backwards
This means that you only have to check the pixel against 6 planes IN TOTAL to see which shadow map you need to access.