Repair corrupted neo4j database (2.3.2 community)

Question

My neo4j-2.3.2 community database has run out of disk space (on 80G). I did a clean shutdown, then added more disk and tried to backup it, only to receive critical error messages and refuse to start it.

I closed it in the morning when I realized that the disk is full and it no longer accepts any inserts. I did not return to returning it until tonight. I apologize for the apparent gap in the logs between shutdown and restart. The database is still in prototype / proof of concept stage. I performed tasks to load data in it last week or so. I really, really don't want to start with the data loading process. It is too slow as it is.

I hope someone tells me that there is a command line option "neo4j --force_repair" that will do the trick. My data loader is smart enough to check the last records that it successfully inserted into the database before trying to load new records, so if we lose a few, this should be fine.

Here is what I see in console.log.

2016-01-31 16:25:56.648+0000 INFO  Successfully shutdown Neo4j Server
2016-01-31 16:25:56.810+0000 INFO  Successfully stopped database
2016-01-31 16:25:56.811+0000 INFO  Successfully shutdown database
2016-02-01 01:16:10.591+0000 INFO  Successfully shutdown Neo4j Server
2016-02-01 01:16:10.593+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
    at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:234)
    at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
    at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
    at org.neo4j.server.CommunityBootstrapper.main(CommunityBootstrapper.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase@2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
    at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:194)
    ... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /home/neo4j/neo4j-community-2.3.2/data/graph.db
    at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:143)
    at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:43)
    at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
    at org.neo4j.server.CommunityNeoServer$1.newGraphDatabase(CommunityNeoServer.java:66)
    at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
    ... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery@4ec23a22' failed to initialize. Please see attached cause exception.
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:434)
    at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:66)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:102)
    at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:600)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
    at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:112)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
    at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
    ... 10 more
Caused by: java.lang.IllegalArgumentException: Unknown entry type -1 for version -1. At position LogPosition{logVersion=252, byteOffset=100663356} and entry version V2_1
    at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.entryParser(LogEntryVersion.java:207)
    at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:92)
    at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:54)
    at org.neo4j.kernel.recovery.LatestCheckPointFinder.find(LatestCheckPointFinder.java:77)
    at org.neo4j.kernel.recovery.PositionToRecoverFrom.apply(PositionToRecoverFrom.java:53)
    at org.neo4j.kernel.recovery.DefaultRecoverySPI.getPositionToRecoverFrom(DefaultRecoverySPI.java:135)
    at org.neo4j.kernel.recovery.Recovery.init(Recovery.java:72)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:424)
    ... 19 more

Answer 1

It appears that transaction logs have been partially written. If true, then the data warehouse is unlikely to be damaged.

WARNING: before making any manual changes to the folder, data/graph.dbbe sure to back up it.

Before continuing, read the paragraph above. Backup? If so, try deleting the transaction log files in data/graph.dbon rm neostore.transaction.db.*. Try starting Neo4j. If it still does not start, insert the fragment data/graph.db/messages.logcontaining the last run sequence - this should give more understanding.