Check that a file is certificate or a key

Question

check and can tell me that the given file is a valid certificate file or private key file

I'm going to answer this in C because OpenSSL is a C library. Others may be able to translate it into pyOpenSSL, in which case they probably have a better answer than mine.

There are two answers here. One is for the certificate, and the second is for the private key. The private key is shown first because it is used to validate the certificate (so it makes sense to visit it first).

Also, its important to call the *_check_key routines because OpenSSL only checks that a key is well encoded; and it does not check that its actually valid. See, for example, Private key generated by openssl does not satisfy n = p * q.

In OpenSSL, you would use the following to verify the the private key:

FILE* file = fopen(...);
EVP_PKEY* pkey = PEM_read_PrivateKey(file, NULL, PasswordCallback, NULL);
unsigned long err = ERR_get_error();

if(pkey)
    EVP_PKEY_free(pkey);

If pkey is NULL, then there was a problem and err holds a reason code. Otherwise, you have a properly encoded private key (but not necessarily valid).

PasswordCallback can simply provide a password in the buffer, or it can prompt the user and return the password in the buffer. For more on PasswordCallback, see Loading a PEM format certificate.

If the key is properly encoded, you can check the type of private key and validate it with the following.

int type = EVP_PKEY_get_type(pkey);
switch (type)
{
case EVP_PKEY_RSA:
case EVP_PKEY_RSA2:
    RSA* rsa = EVP_PKEY_get1_RSA(pkey);
    rc = RSA_check_key(rsa);
    ASSERT(rc);
    RSA_free(rsa);

    break;

case EVP_PKEY_DSA:
case EVP_PKEY_DSA1:
case EVP_PKEY_DSA2:
case EVP_PKEY_DSA3:
case EVP_PKEY_DSA4:
    DSA* dsa = EVP_PKEY_get1_DSA(pkey);
    rc = DSA_check_key(dsa);
    ASSERT(rc);
    DSA_free(dsa);

    break;

case EVP_PKEY_DH:
    DH* dh = EVP_PKEY_get1_DH(pkey);
    rc = DH_check_key(dh);
    ASSERT(rc);
    DH_free(dh);

    break;

case EVP_PKEY_EC:
    EC_KEY* ec = EVP_PKEY_get1_EC_KEY(pkey);
    rc = EC_KEY_check_key(ec);
    ASSERT(rc);
    EC_KEY_free(ec);

    break;

default:
    ASSERT(0);
}

EVP_PKEY_get_type is not part of OpenSSL. Here's how I implemented it:

int EVP_PKEY_get_type(EVP_PKEY *pkey)
{
    ASSERT(pkey);
    if (!pkey)
        return NID_undef;

    return EVP_PKEY_type(pkey->type);
}

In OpenSSL, you would use the following to verify the the certificate:

FILE* file = fopen(...);
X509* x509 = PEM_read_X509(file, NULL, NULL, NULL);
unsigned long err = ERR_get_error();

If x509 is NULL, then there was a problem and err holds a reason code. Otherwise, you have a properly encoded certificate (but not necessarily valid).

You can then verify the certificate with:

/* See above on validating the private key */
EVP_PKEY* pkey = ReadPrivateKey(...);

int rc = X509_verify(x509, pkey);
err = ERR_get_error();

If rc != 1, then there was a problem and err holds a reason code. Otherwise, you have a valid certificate and private key pair. If the certificate is valid, then you can't use err because err is only valid if there's a problem.

If your certificate is signed by an issuer (for example, a CA or intermediate), then you need to use a X509_STORE to verify the issuer's signature on your certificate (a lot of error checking omitted):

const char* serverCertFilename = ...;
const char* issuerCertFilename = ...;    

X509_STORE* store = X509_STORE_new();
ASSERT(store);

static const long flags = X509_V_FLAG_X509_STRICT | X509_V_FLAG_CHECK_SS_SIGNATURE
        | X509_V_FLAG_POLICY_CHECK;
rc = X509_STORE_set_flags(store, flags);
err = ERR_get_error();
ASSERT(rc);

/* Some other object/functions owns 'lookup', but I'm not sure which (perhaps the store) */
X509_LOOKUP* lookup = X509_STORE_add_lookup(store, X509_LOOKUP_file());
/* err = ERR_get_error(); // Does not set error codes. */
ASSERT(lookup);    

/* Cannot load this from memory. No API!!! */
rc = X509_LOOKUP_load_file(lookup, issuerCertFilename, X509_FILETYPE_PEM);
/* err = ERR_get_error(); // Does not set error codes. */
ASSERT(rc);

X509_STORE_CTX* ctx = X509_STORE_CTX_new();
ASSERT(ctx);

X509* serverCert = ReadCertifcate(serverCertFilename);
ASSERT(serverCert);

rc = X509_STORE_CTX_init(ctx, store, serverCert, NULL);
ret = err = ERR_get_error();
ASSERT(rc);

/* Error codes at https://www.openssl.org/docs/crypto/X509_STORE_CTX_get_error.html */
rc = X509_verify_cert(ctx);
err = X509_STORE_CTX_get_error(ctx);

/* Do cleanup, return success/failure */