cassandra snapshot : Unable to create hard link from errno 1

Question

i'm using cassandra 1.2.8 in ubuntu 12.4.

i used snapshot command already and backing up process was ok!

but now when i use this command i get error like below:

   Exception in thread "main" FSWriteError in /var/lib/cassandra/data/english/word_doc  /snapshots/1393155090911/english-word_doc-ic-10-Index.db
at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:80)
at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1063)
at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1566)
at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1611)
at org.apache.cassandra.db.Table.snapshot(Table.java:194)
at org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:2205)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:74)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:277)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:253)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:856)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:805)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1476)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:98)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1317)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1409)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:839)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
  Caused by: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/english/word_doc/english-word_doc-ic-10-Index.db to /var/lib/cassandra/data/english/word_doc/snapshots/1393155090911/english-word_doc-ic-10-Index.db (errno 1)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:174)
at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:76)
... 40 more

note: my current data base is a back up version that i restore in my cassandra!

thanks for your help :)

edit: i have two keyspace,A and B. B is a back up version that i restore and store more data on it, but A is not. i can create back up from A with out any problem,but for B i get above error!

Answer 1

this seems to be an ubuntu 12.04 security issue on hard-links.

use this command to disable that:

sudo sysctl -w kernel.yama.protected_nonaccess_hardlinks=0

and after restore, reset it to its original value:

sudo sysctl -w kernel.yama.protected_nonaccess_hardlinks=1

Answer 2

After struggling with this issue again I found that I had a few root-owned files inside /var/lib/cassandra/data/.../etcetera, which triggered the hard link access restriction.

Recursively chowning all directories and files inside /var/lib/cassandra/data solved the problem.