libssh2 public key authentication only requiring passphrase if key request is accepted

Question

I'm using libssh2 to make a networking program more secure.

I'd like my program to authenticate in as similar way to the OpenSSH client ssh(1) as possible. The OpenSSH client will only ask for passphrases for keys that are actually accepted by the server.

As I understand from this link, an ssh client sends a request to use a public key, and if that is accepted, it can unlock the private key using the passphrase.

libssh2 provides a function libssh2_userauth_publickey_fromfile which takes the private and public key file names and a passphrase. Using this function is very straight forward, but it means I have to obtain the passphrase for private keys even if the public key wouldn't have been accepted by the server in the first place. This is clearly a problem for users that have a lot of different keys (my program currently iterates through key files in the ~/.ssh directory).

I have tried reading the man pages for libssh2 functions, and most of them appear too brief for me to understand without a more detailed knowledge of the ssh protocol. In fact some of them haven't even been written yet.

Can anyone tell me how to only prompt for passphrases for keys that are actually accepted by an ssh server using libssh2?

Answer 1

After RTFM and doing some testing, I discovered that libssh2_userauth_publickey_fromfile will return a different error code depending on whether the key wasn't accepted by the server, or the passphrase was incorrect.

So, here is a pretty inefficient solution (because it calls libssh2_userauth_publickey_fromfile and hence all the key exchange parts of the protocol at least twice).

int nAttempts = 3; // number of attempts the user gets at entering the passphrase

// Try authenticating with an empty passphrase
int err = libssh2_userauth_publickey_fromfile(session, user, pub, priv,"");
if (err == 0)
{
    fprintf(stderr, "You shouldn't use keys with an empty passphrase!\n");
}
else if (err == LIBSSH2_ERROR_PUBLICKEY_UNVERIFIED)
{
    // if we get here it means the public key was initially accepted
    // but the private key has a non-empty passphrase
    char p[BUFSIZ];

    for (int i = 0; i < nAttempts; ++i)
    {
         get_passphrase(p); // assume this gets the passphrase
         err = libssh2_userauth_publickey_fromfile(session, user, pub, priv,p);
         if (err != LIBSSH2_ERROR_PUBLICKEY_UNVERIFIED) break;
    }
}

if (err != 0)
{
    fprintf(stderr, "Authentication using key %s failed!\n", priv);
}

For completeness, the get_passphrase function uses the solution to this question to prompt the user for a passphrase.