Hadoop sshfence (Permission denied)

StackOverflow https://stackoverflow.com/questions/21253299

  •  30-09-2022
  •  | 
  •  

Question

I am trying Hadoop 2 High Availability for HDFS. I set up passwordless ssh connection among NameNodes under user hafence. That I verified - and it works. However I am getting following (Permission Denied) when using this sshfence setup.

2014-01-20 12:54:47,101 INFO  ha.NodeFencer (NodeFencer.java:fence(91)) - ====== Beginning Service Fencing Process... ======
2014-01-20 12:54:47,101 INFO  ha.NodeFencer (NodeFencer.java:fence(94)) - Trying method 1/1: org.apache.hadoop.ha.SshFenceByTcpPort(hafence:22)
2014-01-20 12:54:47,101 WARN  ha.SshFenceByTcpPort (SshFenceByTcpPort.java:tryFence(93)) - Unable to create SSH session
com.jcraft.jsch.JSchException: java.io.FileNotFoundException: /home/hafence/.ssh/id_rsa (Permission denied)
    at com.jcraft.jsch.IdentityFile.newInstance(IdentityFile.java:98)
    at com.jcraft.jsch.JSch.addIdentity(JSch.java:206)
    at com.jcraft.jsch.JSch.addIdentity(JSch.java:192)
    at org.apache.hadoop.ha.SshFenceByTcpPort.createSession(SshFenceByTcpPort.java:122)
    at org.apache.hadoop.ha.SshFenceByTcpPort.tryFence(SshFenceByTcpPort.java:91)
    at org.apache.hadoop.ha.NodeFencer.fence(NodeFencer.java:97)
    at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:521)
    at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:494)
    at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:59)
    at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:837)
    at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:900)
    at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:799)
    at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:415)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:596)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
Caused by: java.io.FileNotFoundException: /home/hafence/.ssh/id_rsa (Permission denied)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at java.io.FileInputStream.<init>(FileInputStream.java:97)
    at com.jcraft.jsch.IdentityFile.newInstance(IdentityFile.java:83)
    ... 14 more
2014-01-20 12:54:47,102 WARN  ha.NodeFencer (NodeFencer.java:fence(108)) - Fencing method org.apache.hadoop.ha.SshFenceByTcpPort(hafence:22) was unsuccessful.
2014-01-20 12:54:47,102 ERROR ha.NodeFencer (NodeFencer.java:fence(111)) - Unable to fence service by any configured method.
2014-01-20 12:54:47,102 WARN  ha.ActiveStandbyElector (ActiveStandbyElector.java:becomeActive(807)) - Exception handling the winning of election

My configuration is as following in hdfs-site.xml:

  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>sshfence(hafence:22)</value>
  </property>
  <property>
    <name>dfs.ha.fencing.ssh.private-key-files</name>
    <value>/home/hafence/.ssh/id_rsa</value>
  </property>
  <property>
    <name>dfs.ha.fencing.ssh.connect-timeout</name>
    <value>30000</value>
  </property>

  <property>
    <name>dfs.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>

What are the permission requirements, user etc. in order to make the sshfence working?

Was it helpful?

Solution

As hadoop were installed from RPM packages. We used hdfs user(including password less auth) for sshfence and then sshfence started to work.

OTHER TIPS

Check the permissions on /home/hafence/.ssh/id_rsa. I had this issue. Check the permissions on id_rsa. Your user should have read-write permissions.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top