The public key in the known_hosts file isn't the public key you need to be using. Rather, you need to be using the public key from authorized_keys.
That said, the fact that libssh2 even requires the public key is stupid. Honestly, I'd recommend you use phpseclib, a pure PHP SSH2 implementation, over libssh2. eg.
<?php
include('Net/SFTP.php');
include('Crypt/RSA.php');
$ssh = new Net_SFTP('www.domain.tld');
$key = new Crypt_RSA();
$key->loadKey(file_get_contents('private.pem'));
if (!$ssh->login('username', $key)) {
exit('Login Failed');
}
print_r($ssh->nlist());
?>