I believe I've figured out what to do:
Dim PrivateKeyPath As String = "C:\Users\xxxxx\privatekey.p12"
Dim CertificatePassword As String = "notasecret"
Dim cert As New X509Certificate2(PrivateKeyPath, CertificatePassword,
X509KeyStorageFlags.Exportable)
Dim certData As Byte() = cert.Export(X509ContentType.Pkcs12, CertificatePassword)
Dim keyString As String = Convert.ToBase64String(certData)
As @Jim Mischel pointed out, I just needed to encode the key data in base-64.
By the way, adding the X509KeyStorageFlags.Exportable
flag fixed a CryptographicException (“Key not valid for use in specified state.”) while trying to export the key.