mongo crashing due to too many open files

I recently upgraded an Ubuntu 14 server, running mongo 3.4.11, to Ubuntu 16. I re-installed the identical version of mongodb-org from their PPA, but now when I start mongo, it's doesn't respond to any connections, and I see this error in /var/log/mongodb/mongodb.log:

```
2018-02-06T18:40:15.680+0000 I CONTROL [main] SERVER RESTARTED
2018-02-06T18:40:15.684+0000 I CONTROL [initandlisten] MongoDB starting : pid=15925 port=27017 dbpath=/var/lib/mongodb 64-bit host=proddb1
2018-02-06T18:40:15.684+0000 I CONTROL [initandlisten] db version v3.4.11
2018-02-06T18:40:15.684+0000 I CONTROL [initandlisten] git version: 34f5bec2c9d827d71828fe858167f89a28b29a2a
2018-02-06T18:40:15.684+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2018-02-06T18:40:15.684+0000 I CONTROL [initandlisten] allocator: tcmalloc
2018-02-06T18:40:15.684+0000 I CONTROL [initandlisten] modules: none
2018-02-06T18:40:15.684+0000 I CONTROL [initandlisten] build environment:
2018-02-06T18:40:15.684+0000 I CONTROL [initandlisten] distmod: ubuntu1604
2018-02-06T18:40:15.684+0000 I CONTROL [initandlisten] distarch: x86_64
2018-02-06T18:40:15.684+0000 I CONTROL [initandlisten] target_arch: x86_64
2018-02-06T18:40:15.684+0000 I CONTROL [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } }
2018-02-06T18:40:15.684+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2018-02-06T18:40:15.709+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2018-02-06T18:40:15.709+0000 I STORAGE [initandlisten]
2018-02-06T18:40:15.709+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-02-06T18:40

I have gone through your error and find out that in your case the Open file limit is 990000. Which is sufficient to open the mongos and mongod process. As per MongoDB BOL Here The default net.maxIncomingConnections should be Default: 65536 and by default LimitNOFILE should be Here LimitNOFILE=64000.

The maximum number of simultaneous connections that mongos or mongod will accept. This setting has no effect if it is higher than your operating system’s configured maximum connection tracking threshold.

Do not assign too low of a value to this option, or you will encounter errors during normal application operation.

This is particularly useful for a mongos if you have a client that creates multiple connections and allows them to timeout rather than closing them.

In this case, set maxIncomingConnections to a value slightly higher than the maximum number of connections that the client creates, or the maximum size of the connection pool.

As per Ramon Fernandez from MongoDB jira Blog Here WiredTiger needs at least two files per collection (one for the collection data and one for the _id index), plus one file per additional index in a collection. If your total count of collections and indexes is large you'll need to adjust your open files limit accordingly.

As per MongoDB Recommended ulimit Settings for Linux distributions using systemd is

[Service]
# Other directives omitted
# (file size)
LimitFSIZE=infinity
# (cpu time)
LimitCPU=infinity
# (virtual memory size)
LimitAS=infinity
# (open files)
LimitNOFILE=64000
# (processes/threads)
LimitNPROC=64000

As in your case it's 990000, which should be sufficient for most deployments. Please make sure that you set a high value for this limit at the system level.

For further your ref Here and Here and Resource limit directives, their equivalent ulimit shell commands and the unit used

StackExchange Database Administrators Q#197328, answer score: 4