cleaning up failed repair db: path: /var/lib/docker/wesfgr/repair/_tmp_repairDatabase_0

Question

We run MongoDB 3.0.6 with MMAP in Docker container (single node). The filesystem was full and I did a repair with other path.

2016-10-03T08:58:51.186+0200 I INDEX    [initandlisten]          building index using bulk method
2016-10-03T08:58:51.186+0200 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T08:58:51.186+0200 I INDEX    [initandlisten] build index on: 6JzeOdlsErBwx4Fc.fb31a88b-cdfc-440b-81f8-d333dcd9527bs properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "6JzeOdlsErBwx4Fc.fb31a88b-cdfc-440b-81f8-d333dcd9527bs" }
2016-10-03T08:58:51.186+0200 I INDEX    [initandlisten]          building index using bulk method
2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten] build index on: 6JzeOdlsErBwx4Fc.fb362dbf-a5d9-445b-a052-1166f0c522a7 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "6JzeOdlsErBwx4Fc.fb362dbf-a5d9-445b-a052-1166f0c522a7" }
2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten]          building index using bulk method
2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten] build index on: 6JzeOdlsErBwx4Fc.fb38d789-61a7-40ff-a63f-146244df4475 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "6JzeOdlsErBwx4Fc.fb38d789-61a7-40ff-a63f-146244df4475" }
2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten]          building index using bulk method
2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten] error: hashtable namespace index max chain reached:1335
2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten] error: hashtable namespace index max chain reached:1335
2016-10-03T08:58:51.349+0200 I STORAGE  [initandlisten] cleaning up failed repair db: 6JzeOdlsErBwx4Fc path: /var/lib/docker/wesfgr/repair/_tmp_repairDatabase_0
2016-10-03T08:58:51.570+0200 I STORAGE  [initandlisten] exception in initAndListen: 10081 too many namespaces/collections, terminating
2016-10-03T08:58:51.570+0200 I CONTROL  [initandlisten] now exiting
2016-10-03T08:58:51.570+0200 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2016-10-03T08:58:51.570+0200 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2016-10-03T08:58:51.570+0200 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2016-10-03T08:58:51.570+0200 I NETWORK  [initandlisten] shutdown: going to close sockets...
2016-10-03T08:58:51.570+0200 I STORAGE  [initandlisten] shutdown: waiting for fs preallocator...
2016-10-03T08:58:51.570+0200 I STORAGE  [initandlisten] shutdown: closing all files...
2016-10-03T08:58:51.579+0200 I STORAGE  [initandlisten] closeAllFiles() finished
2016-10-03T08:58:51.580+0200 I STORAGE  [initandlisten] shutdown: removing fs lock...
2016-10-03T08:58:51.580+0200 I CONTROL  [initandlisten] dbexit:  rc: 100

there are lots of errors, but the MongoDB started successfully after repair. What do this errors mean? Will we have an inconsistent DB long term?

2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten] error: hashtable namespace index max chain reached:1335
2016-10-03T08:58:51.187+0200 I INDEX    [initandlisten] error: hashtable namespace index max chain reached:1335
2016-10-03T08:58:51.349+0200 I STORAGE  [initandlisten] cleaning up failed repair db: 6JzeOdlsErBwx4Fc path: /var/lib/docker/wesfgr/repair/_tmp_repairDatabase_0
2016-10-03T08:58:51.570+0200 I STORAGE  [initandlisten] exception in initAndListen: 10081 too many namespaces/collections, terminating

after restart a lot of those lines in logs:

2016-10-03T07:03:41.458+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.458+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.459+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.459+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.459+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.459+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.459+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.459+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.459+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.459+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.460+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.460+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.460+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.461+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.461+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.461+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.462+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.462+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.463+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.463+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.463+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.463+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.463+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.463+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.463+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.463+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.464+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.464+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.464+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.464+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.464+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.464+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.464+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.465+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.465+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.465+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.465+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.465+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.465+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.465+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.465+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.466+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.466+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain
2016-10-03T07:03:41.466+0000 I INDEX    [initandlisten] warning: hashtable namespace index long chain

Answer 1

It looks like your namespace file is still corrupt, you get similar errors when you hit the max number of collections for a database in MMAPv1. To try to fix this up and get a database that is in a more consistent state you can try recreating your collections in a new database.

If you have auth enabled, you will need to do this in the admin database - note that the real key here is the new database in the to: field:

use admin;
db.runCommand( { renameCollection: "db1.collection1", to: "db2.collection1" });

Repeat that for each collection in the database at present, and make sure you have enough space to recreate your collection on the existing filesystem. Once that is done I would recommend backing up (just in case) and removing the old database.