Mongodb high lock percentage / slow queries

I'm working with an application using MongoDB that is getting a fairly large number of incoming requests and I've noticed that the responses are getting slower as the number of requests get higher.

I'm suspecting that there is some inefficiency with the database. I ran mongostat during one of our higher load periods and the result seems to indicate high lock percentage:

```
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn set repl time
0 1120 479 0 179 472|0 0 72.1g 146g 1.31g 0 server-prod:15.7% 0 0|0 1|0 1m 4m 298 rs0 PRI 15:37:28
0 1192 509 0 178 428|0 0 72.1g 146g 1.3g 0 server-prod:18.9% 0 0|0 0|0 1m 5m 298 rs0 PRI 15:37:29
0 1107 456 0 174 362|0 0 72.1g 146g 1.3g 0 server-prod:20.4% 0 0|0 2|1 1m 5m 298 rs0 PRI 15:37:30
0 1320 556 0 211 532|0 0 72.1g 146g 1.3g 0 server-prod:19.5% 0 0|0 0|1 1m 6m 298 rs0 PRI 15:37:31
0 1094 474 0 179 449|0 0 72.1g 146g 1.3g 0 server-prod:15.7% 0 0|0 0|0 1m 5m 298 rs0 PRI 15:37:32
0 1120 487 0 184 458|0 0 72.1g 146g 1.3g 0 server-prod:20.8% 0 0|0 1|1 1m 5m 298 rs0 PRI 15:37:33
0 807 299 0 108 270|0 0 72.1g 146g 1.3g 0 server-prod:15.3% 0 103|1 0|3 1m 3m 298 rs0 PRI 15:37:34
0 1613 709 0 161 146|0 0 72.1g 146g 1.3g 0 server-prod:63.5% 0 0|0 1|0 2m 7m 298 rs0 PRI 15:37:35
0 1133 472 0 167 341|0 0 72.1g 146g 1.31g 0 server-prod:21.2% 0 1|0 0|1 1m 6m 298 rs0

To see if the hardware is not limiting:

• top/htop => cpu percentage

• iostat -x 1 => sysstat tool to see disk r/w limits (%util)

Concerning locking:

• Mongo 2.6 : database locking

• Mongo 3.0 + MMAPv1 storage engine : collection locking

• Mongo 3.0 + WiredTiger storage engine : document locking

If you have 1 huge collection (server-prod), maybe Sharding is an option to distribute the load, or more cores + less locking with Mongo3.0

Improve indexes:
- More indexes = slower write + faster read
- Less indexes = faster write + slower read

Read from Secondaries, Only write on Primary.

> db.setProvilingLevel(1,4)  ##  save slow logs for that db slower than 4ms
> db.system.profile.find({millis:{$gt:100}}).sort({ts:-1}) ## find queries slower than 100ms, order by timestamp descending
> ....query.explain()  ## find out which indexes it uses

Information:
http://docs.mongodb.org/manual/administration/optimization/

StackExchange Database Administrators Q#104153, answer score: 5