I encountered this problem in the past with the .p7b extension. There are two ways I found you can solve this. In the end I ended up using number 1. Number 2 is something you already found out by exporting to a .cer. You can also try to use option 3 but I am not sure if that will fully work.
1. Use SignedCms instead of the X509Certificate class.
See for more details Enveloped PKCS #7 Signatures
2. Loading a .p7b only includes the certificate file, which probably doesn't include the private key. Install the private key on the server where it was generated and then export it to as a .pfx file and move it to the server you want to use.
3. Since a .p7b file contains the whole certificate chain and not just one certificate you can try the follow method to add this to the windows store.
X509Certificate2Collection certCollection = new X509Certificate2Collection();
certCollection.Import(@"C:\test_public_cert.p7b");
X509Store store = new X509Store(StoreName.AddressBook, StoreLocation.LocalMachine);
store.Open(OpenFlags.ReadWrite);
store.AddRange(certCollection);