mysql_install_db equivalent for MongoDB? Error 'handle-open: open: Too many open files'

Question

A developer created millions of collections and indexes, this means millions of collection-* and index-* files.

I have no administrative login (DB user) on this replicaset (3 node) with mongod 3.2.6. I wish to initialize MongoDB data directory only with Linux commands. In MariaDB there is mysql_install_db command.

error message

2017-01-14T05:25:38.674+0000 E STORAGE  [thread2] WiredTiger (24) [1484371538:674300][60390:0x7fcc4bc69700], file:WiredTiger.wt, WT_SESSION.checkpoint: /var/vcap/store/mongodb-data/example/WiredTiger.turtle: handle-open: open: Too many open files
2017-01-14T05:25:38.737+0000 I -        [conn2] Assertion: 13538:couldn't open [/proc/60390/stat] errno:24 Too many open files
2017-01-14T05:25:38.738+0000 I NETWORK  [conn2] end connection 127.0.0.1:39289 (12 connections now open)
2017-01-14T05:25:38.738+0000 I NETWORK  [conn6] end connection 127.0.0.1:39347 (12 connections now open)
2017-01-14T05:25:38.738+0000 I NETWORK  [conn9] end connection 127.0.0.1:39453 (12 connections now open)
2017-01-14T05:25:38.769+0000 I COMMAND  [conn10] command example command: collStats { collstats: "090f42d6-5368-46c6-80
ca-2a8e7e380693" } keyUpdates:0 writeConflicts:0 numYields:0 reslen:9450 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r:
1 } } } protocol:op_query 161ms
2017-01-14T05:25:38.783+0000 E STORAGE  [thread2] WiredTiger (24) [1484371538:783882][60390:0x7fcc4bc69700], checkpoint-server: checkpoint server error: Too many open files
2017-01-14T05:25:38.783+0000 E STORAGE  [thread2] WiredTiger (-31804) [1484371538:783983][60390:0x7fcc4bc69700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger
library panic
2017-01-14T05:25:38.784+0000 I -        [thread2] Fatal Assertion 28558
2017-01-14T05:25:38.784+0000 I -        [thread2]

***aborting after fassert() failure


2017-01-14T05:25:38.784+0000 I -        [WTJournalFlusher] Fatal Assertion 28559
2017-01-14T05:25:38.784+0000 I -        [WTJournalFlusher]

***aborting after fassert() failure


2017-01-14T05:25:38.789+0000 F -        [thread2] Got signal: 6 (Aborted).

 0x1383b32 0x1382c89 0x1383492 0x7fcc50af2330 0x7fcc50753c37 0x7fcc50757028 0x130dba2 0x1109083 0x1ae49dc 0x1ae4e9d 0x1ae5284 0x1a69dbb 0x7fcc50aea184 0x7fcc5081737d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F83B32","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F82C89"},{"b":"400000","o":"F83492"},{"b":"7FCC50AE2000","o":"10330"},{"b":"7FCC5071D000","o":"36C37","s":"gsignal"},{"b":"7FCC5071D000","o":"3A028","s":"abort"},{"b":"400000","o":"F0DBA2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"D09083"},{"b":"400000","o":"16E49DC"
,"s":"__wt_eventv"},{"b":"400000","o":"16E4E9D","s":"__wt_err"},{"b":"400000","o":"16E5284","s":"__wt_panic"},{"b":"400000","o":"1669DBB"},{"b":"7FCC50AE2000","o":"8184"},{"b":"7FCC5071D00
0","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.6", "gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25", "compiledModules" : [ "enterprise" ], "uname" : { "sysna
me" : "Linux", "release" : "3.19.0-64-generic", "version" : "#72~14.04.1-Ubuntu SMP Fri Jun 24 17:59:48 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "bui
ldId" : "88B6197786F38A44F8D924B095DAD84A9E55F2C8" }, { "b" : "7FFF49FFB000", "elfType" : 3, "buildId" : "C89BD46B7CFC47F3E55EF539B3FAF8E450562F6A" }, { "b" : "7FCC52C73000", "path" : "/us
r/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "666B276BD134B0E9579B67D4EE333F2D0FB813CD" }, { "b" : "7FCC52806000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpmibs.s
o.30", "elfType" : 3, "buildId" : "4630C89B4E7BCCAD8B1B4FFB508962666D6663C2" }, { "b" : "7FCC525F7000", "path" : "/usr/lib/x86_64-linux-gnu/libsensors.so.4", "elfType" : 3, "buildId" : "85
9FDBFDD82F0EFDEB44A433D9D8020A232A35E2" }, { "b" : "7FCC523F3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "
b" : "7FCC5218A000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpagent.so.30", "elfType" : 3, "buildId" : "9E02A41B22FEB1F704B60BA109EC5785131A5090" }, { "b" : "7FCC51F80000", "path" : "
/lib/x86_64-linux-gnu/libwrap.so.0", "elfType" : 3, "buildId" : "54FCBC5B0F994A13A9B0EAD46F23E7DA7F7FE75B" }, { "b" : "7FCC51CA6000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmp.so.30",
 "elfType" : 3, "buildId" : "E9B667050A5D6C0C4D58826C32DEEAF38B16EBAA" }, { "b" : "7FCC518CA000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "AAE7CFF83
51B730830BDBCE0DCABBE06574B7144" }, { "b" : "7FCC51683000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "55F72A23CB9C0F7529F0E0BEE43981864B74C4FE"
}, { "b" : "7FCC5137D000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7FCC5111E000", "path" : "/lib/x86_6
4-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "74864DB9D5F69D39A67E4755012FB6573C469B3D" }, { "b" : "7FCC50F16000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "
buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7FCC50D00000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED
7359DBB92" }, { "b" : "7FCC50AE2000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7FCC5071D000", "pa
th" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7FCC52E8E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType"
: 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" }, { "b" : "7FCC50394000", "path" : "/usr/lib/libperl.so.5.18", "elfType" : 3, "buildId" : "73F1E3843DAEC3F6114D138BF002C0A122CC
3707" }, { "b" : "7FCC5017A000", "path" : "/lib/x86_64-linux-gnu/libnsl.so.1", "elfType" : 3, "buildId" : "32E56CFD30B8B4FCD8AA69CED88A9782814A9D18" }, { "b" : "7FCC4FEAF000", "path" : "/u
sr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "77287B3AF8DD293D7367EEF27C652C04353752EC" }, { "b" : "7FCC4FC80000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3
", "elfType" : 3, "buildId" : "49E3D743C2B3741229AD3892B22C4794C646E1F2" }, { "b" : "7FCC4FA7C000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "8D56938ABD
6462C4C29822D8E48A131BE1C61F6A" }, { "b" : "7FCC4F871000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "0B3ABC152466DE0C69954405A0E980B6E0D4B78F" }
, { "b" : "7FCC4F638000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "FDB9E0552092EB7559F27C7A9915665408D930F0" }, { "b" : "7FCC4F434000", "path" : "/lib/x8
6_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "0F03635F97B93D3DACD84F0ED363C56BD266044F" }, { "b" : "7FCC4F219000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType

Answer 1

errno:24 Too many open files

This error indicates the mongod process is unable to open any more file handles because the configured operating system resource limits (aka ulimit on UNIX-like operating systems such as Linux) have been reached. The ulimit settings are meant to constrain individual users or processes from consuming excessive system resources.

Verifying and increasing ulimits is an admin task outside of MongoDB. See: UNIX ulimit Settings in the MongoDB manual for more information.

I wish to initialize MongoDB data directory only with Linux commands. In MariaDB there is mysql_install_db command.

As at MongoDB 3.4 there is no equivalent of "initializing" a MongoDB data directory beyond copying data files from another deployment before starting mongod with the appropriate configuration file. There are also no standard set up scripts that would be analogous to mysql_install_db , which does preparatory tasks such as creating schema and user accounts.

MongoDB production deployments are generally configured as replica sets or sharded clusters, so set up requires context of the overall deployment. Typical solutions for deployment automation include MongoDB Cloud Manager (SaaS) and MongoDB Ops Manager (on-premise). New AWS environments provisioned using MongoDB Cloud Manager will include O/S settings as per the Production Notes in the MongoDB manual.

A developer created millions of collections and indexes, this means millions of collection-* and index-* files.

You can increase ulimit settings to try to support this use case, but with millions of collections you are likely to run into other performance issues (for example, commands like listDatabases would have to iterate stats for millions of files). Unless the millions of collections and indexes were created accidentally, I would consult with the developer and advise them to rethink their data model.