SSHD Gives error could not open Authorized Keys, although permissions seem correct

Question 1

If the permissions are correct, SELinux might still be preventing sshd from opening the file.

Try fixing the labels inside the .ssh directory (and maybe $HOME):

restorecon -FRvv ~/.ssh

If the user account uses non-standard home path, default labels for the path need to be added to the local configuration first:

semanage fcontext -a -t ssh_home_t "/srv/custom/\.ssh(/.*)?"

(I'm intentionally not suggesting disabling SELinux or setting it to the permissive mode.)

Question 2

In case if SELinux enabled:

$ getenforce
Enforcing

to temporary enable pub-key ssl login to non-standard user home directory location run:

$ sudo chcon -t ssh_home_t /srv/jenkins/.ssh/authorized_keys /srv/jenkins/.ssh

$ ls -ldZ /srv/jenkins/.ssh/authorized_keys /srv/jenkins/.ssh/
drwxr-xr-x. jenkins jenkins system_u:object_r:ssh_home_t:s0  /srv/jenkins/.ssh/
-rw-r--r--. jenkins jenkins system_u:object_r:ssh_home_t:s0  /srv/jenkins/.ssh/authorized_keys

See https://linux.die.net/man/8/ssh_selinux for the details.

To make SELinux settings permanent run:

$ sudo semanage fcontext -a -t ssh_home_t /srv/jenkins/.ssh/authorized_keys
$ sudo semanage fcontext -a -t ssh_home_t /srv/jenkins/.ssh
$ sudo restorecon -R -v /srv/jenkins/.ssh/

You hit this if you are on modern RHEL, Oracle Linux, CentOS.

Question 3

I was struggling to use key authentication as well.

Could not open authorized keys '/home/myUserName/.ssh/authorized_keys2': Permission denied

Had checked all the above things when I ended up here (first link on google). I realize that this is an old post but I will add it here in case somebody else has the same problem as me and end up here.

I had owner of the authorized_keys file to "root", so changing it with:

chown myUserName authorized_keys2

Solved it for me.

Question 4

Check the /home directory permissions. It should be

drwxr-xr-x. 9 root root 113 Jun 28 22:57 home

and then your home directory detail:

drwxr----- 5 user group 124 May 18 17:00 User drwx------ 2 user group 29 May 18 12:05 .ssh -rw------- 1 user group 2235 Jun 28 23:09 authorized_keys

My error messages in logs

/var/log/secure > sshd[22565]: error: Received disconnect from X.X.X.X: 14: No supported authentication methods available [preauth]

On client side

ssh user@X.X.X.X Permission denied (publickey). ssh -vvv user@X.X.X.X ... debug2: we did not send a packet, disable method debug1: No more authentication methods to try. Permission denied (publickey).
On server side

service sshd stop
run sshd debug mode:
/usr/sbin/sshd -ddd

... debug1: trying public key file /home/USER/.ssh/authorized_keys debug1: Could not open authorized keys '/home/USER/.ssh/authorized_keys': Permission denied ...

Question 5

I've spent a good couple hours with this crap. Reading loads of illuminated solutions to the issue, here is a boil down of what none of them say clearly.

Sumarizing.

The following applies to these files:

~/.ssh/
~/.ssh/authorized_keys

Ownership

The owner and group must be the logging user.

Permissions

Group and public permissions must be set to 0 (no permissions). Yes! sshd thinks you're stupid by default.

And of course, the owner must be allowed to read.

Note. Permissions such as 777 give the same error as 000, even though it is an application policy to not open the file. This was probably the biggest source of confusion during this whole endeavour.

Users

All logging users must exist. By logging users we mean:

The user sent with the url ssh root@localhost
Or, the user in the public key present on the client machine (~/.ssh/id_rsa.pub in the client, not in the server)

Question 6

A couple ideas to check:

Can you cat authorized_keys? What does the file look like?
Is your sshd configured to allow root login? This is generally frowned upon,
Are you doing it as root or as a sudoer?

Question 7

Don't do chmod on ~/.ssh/.... Try to write the exact path: /root/.ssh/..., since sometimes (when using su etc), the ~ can be setup incorrectly. Check and post the permissions again for the full path without using ~ in the command.
Once you are absolutely sure the permissions are OK, check if your sshd is actually running under user root: ps -A u | grep sshd.

Question 8

A couple of things to double-check:

Are you sure you copied the PUBLIC key to the authorized_keys, not the private key? :-)
Do cat -tv authorized_keys. Any ^M characters at the end of each line? Do a dos2unix on authorized_keys
Did you restart the ssh daemon after making configuration changes?

Question 9

I encountered this same issue and got it solved by changing both .ssh and authorized_keys's owner at the same time: chown MyUsername:Myusername .ssh chown MyUsername:Myusername .ssh/authorized_keys

Thanks to @niclaslindgren.

And BTW, it's no matter with whether there is ^M in authorized_keys or not, I had tested and proved it, it works with both the ways

Question 10

I'm going to throw my answer in here as well since I just wasted the last hour trying to figure out a workaround for another hour I wasted previously on Azure.

This does not work:

AuthorizedKeyFiles ~/.ssh/authorized_keys

Generating server-side logs using an alternate port (i.e. /usr/sbin/sshd -ddd -p 12345) reveals the following when using the above configuration:

debug1: trying public key file /root/.ssh/authorized_keys
debug1: Could not open authorized keys '/root/.ssh/authorized_keys': Permission denied

Either of these are correct, however:

AuthorizedKeyFiles /home/%u/.ssh/authorized_keys

AuthorizedKeyFiles .ssh/authorized_keys

See also: https://www.ssh.com/ssh/authorized_keys/openssh

Question 11

For me it was also changing ownership, but not just of authorized_keys but also of the .sdd directory:

chown -R user:user ./home/user/.ssd

Question 12

In my case, home directories are on NFS, which means that ~/.ssh is also on NFS.

Found this answer, and indeed running the following (as root) solved the problem:

setsebool -P use_nfs_home_dirs 1

Question 13

There is a set of generic steps if one has to find the reason why sshd is refusing to accept a connection or keys. The details below are for a systemd based system but alternative systems users would be able to find their way easily.

How to debug sshd systematically?

Start watching the journal

journalctl -u sshd -f

Set sshd logging to debug mode in /etc/ssh/sshd_config

LogLevel DEBUG

Restart the daemon to let the change take effect

systemctl restart sshd

We are set on the server side.

Try the client connection now

ssh -o IdentitiesOnly=yes -v -i ~/.ssh/key_to_the_kingdom king@kingdom.gov

IdentitiesOnly disables trying other than the identity specified.
-v to increase the client verbosity so one can see if the client is doing what is expected. (Is able to find and use the key on the client system, is able to negotiate the encryption algorithm, etc.)

(Not posting an example for the client.)

On the server (kingdom.gov in our example), we should see something like the following in the debug mode log:

Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: userauth-request for user king service ssh-connection method none [preauth]
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: attempt 0 failures 0 [preauth]
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: PAM: initializing for "king"
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: PAM: setting PAM_RHOST to "75.73.78.71"
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: PAM: setting PAM_TTY to "ssh"
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: userauth-request for user king service ssh-connection method publickey [preauth]
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: attempt 1 failures 0 [preauth]
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: userauth_pubkey: test pkalg rsa-sha2-512 pkblob RSA SHA256:0hjXPXkM8d91W2D8bg3fcapifm5QJd7QV9wwOEMU1 [preauth]
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: temporarily_use_uid: 112233/10 (e=0/0)
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: trying public key file /home/king/.ssh/authorized_keys
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: Could not open authorized keys '/home/king/.ssh/authorized_keys': Permission denied
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: restore_uid: 0/0
Jul 14 12:46:39 kingdom.gov sshd[4665]: Failed publickey for king from 83.69.65.84 port 52756 ssh2: RSA SHA256:0hjXPXkM8d91W2D8bg3fcapifm5QJd7QV9wwOEMU1

In my case, the problem was in SELinux not allowing to use authorized_keys stored in the NFS home directory.

You may be asking: How can I check the permissions? What is the identity sshd uses to access the files?

Look at temporarily_use_uid: 112233/10 in the log above. There should be correct UID and primary GID for the user. In my case, these values were taken from the name service (LDAP) and were as expected.

If the client user identity is incorrect, look into the name service configuration and resolve this issue first.

The directory and file ownership and permissions were as expected (at least u=x for ~ and ~/.ssh directories, at least u=r for authorized_keys, if owned by the user). It was clear the reason must be in something like SELinux.

Let's check it:

getsebool use_nfs_home_dirs
  use_nfs_home_dirs --> off

For that case, the answer by Nadav Aharoni resolves the problem.

 setsebool -P use_nfs_home_dirs 1

Cleanup

Restore LogLevel DEBUG in sshd_config to the previous value. (Or comment it out to restore the default.)
systemctl restart sshd