neo4j 1.6.M02 database seems corrupted; how do I recover it?

Question

I have a neo4j instance on a ubuntu machine that shut down while the service was up. As a consequence, now the service refuses to start:

Starting Neo4j Server... process [3807]... waiting for server to be ready.. BAD.
Neo4j Server may have failed to start, please check the logs.

and the logs...

Feb 16, 2012 4:44:51 PM org.neo4j.server.logging.Logger log
INFO: Store files missing, or not in suitable state for upgrade. Leaving this problem for main server process to resolve.
Feb 16, 2012 4:44:51 PM org.neo4j.server.logging.Logger log
INFO: Starting Neo Server on port [7474] with [20] threads available
Feb 16, 2012 4:44:51 PM org.mortbay.log.Slf4jLog info
INFO: Logging to org.slf4j.impl.JDK14LoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
Feb 16, 2012 4:44:51 PM org.neo4j.server.logging.Logger log
INFO: Using database at /home/dan/personal/neo4j-community-1.6.M02/data/graph.db
Feb 16, 2012 4:44:52 PM org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery
INFO: Non clean shutdown detected on log [/home/dan/personal/neo4j-community-1.6.M02/data/graph.db/nioneo_logical.log.1]. Recovery started ...

any idea how can

1. recover the database ?
2. prevent this to happen again should the machine fail again?

thanks

Update: Tried running sudo ./bin/neo4j-shell -path data/graph.db/ and this is the output

Feb 16, 2012 3:57:16 PM org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery
INFO: Non clean shutdown detected on log [/home/dan/personal/neo4j-community-1.6.M02/data/graph.db/nioneo_logical.log.1]. Recovery started ...
org.neo4j.graphdb.TransactionFailureException: Could not create data source [nioneodb], see nested exception for cause of error
    at org.neo4j.kernel.impl.transaction.TxModule.registerDataSource(TxModule.java:161)
    at org.neo4j.kernel.GraphDbInstance.start(GraphDbInstance.java:104)
    at org.neo4j.kernel.EmbeddedGraphDbImpl.<init>(EmbeddedGraphDbImpl.java:190)
    at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:78)
    at org.neo4j.shell.kernel.GraphDatabaseShellServer.instantiateGraphDb(GraphDatabaseShellServer.java:85)
    at org.neo4j.shell.kernel.GraphDatabaseShellServer.<init>(GraphDatabaseShellServer.java:57)
    at org.neo4j.shell.StartClient.tryStartLocalServerAndClient(StartClient.java:286)
    at org.neo4j.shell.StartClient.startLocal(StartClient.java:219)
    at org.neo4j.shell.StartClient.start(StartClient.java:159)
    at org.neo4j.shell.StartClient.main(StartClient.java:108)
Caused by: java.lang.RuntimeException: Unexpected Exception
    at org.neo4j.helpers.Exceptions.launderedException(Exceptions.java:78)
    at org.neo4j.helpers.Exceptions.launderedException(Exceptions.java:48)
    at org.neo4j.helpers.Exceptions.launderedException(Exceptions.java:43)
    at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.<init>(NeoStoreXaDataSource.java:216)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.neo4j.kernel.impl.transaction.XaDataSourceManager.create(XaDataSourceManager.java:77)
    at org.neo4j.kernel.impl.transaction.TxModule.registerDataSource(TxModule.java:155)
    ... 9 more
Caused by: java.io.IOException: Inject start failed, xid: GlobalId[NEOKERNL|3992629294259512899|39], BranchId[ 52 49 52 49 52 49 ] already injected
    at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.injectStart(XaResourceManager.java:118)
    at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.applyStartEntry(XaLogicalLog.java:486)
    at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.applyEntry(XaLogicalLog.java:445)
    at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.doInternalRecovery(XaLogicalLog.java:815)
    at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.open(XaLogicalLog.java:241)
    at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.open(XaLogicalLog.java:187)
    at org.neo4j.kernel.impl.transaction.xaframework.XaContainer.openLogicalLog(XaContainer.java:120)
    at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.<init>(NeoStoreXaDataSource.java:177)
    ... 15 more
ERROR (-v for expanded information):
    Could not create data source [nioneodb], see nested exception for cause of error

 -host      Domain name or IP of host to connect to (default: localhost)
 -port      Port of host to connect to (default: 1337)
 -name      RMI name, i.e. rmi://<host>:<port>/<name> (default: shell)
 -pid       Process ID to connect to
 -c         Command line to execute. After executing it the shell exits
 -readonly  Connect in readonly mode
 -path      Points to a neo4j db path so that a local server can be started there
 -config    Points to a config file when starting a local server

Example arguments for remote:
    -port 1337
    -host 192.168.1.234 -port 1337 -name shell
    -host localhost -readonly
    ...or no arguments for default values
Example arguments for local:
    -path /path/to/db
    -path /path/to/db -config /path/to/neo4j.config
    -path /path/to/db -readonly

Answer 1

Ok, unfortunately you are running 1.6.M02 which has this known issue which can put your db in this state. 1.6 doesn't have this issue. There was another db which had this problem on the neo4j mailing list, so I created this tool: http://pastebin.com/KuSTcaGM to fix such a broken log. Would you feel comfortable compiling and running that? Otherwise I could put together a bundled jar accompanied a script like:

$ ./fix-double-start-record-issue my/db

and send you somehow.