high background flush avg mongodb

similar question: High Global Lock % on Mongodb

Overview

We have a production setup replica set in v2.4.8 mongodb running on five 4-core, 28gb RAM VMs with standard azure datadisk HDDs running on 64bit CentOS 6. We distribute reads across the secondaries at about 600-700 ops/sec/secondary. The CPU usage is around 15% per secondary. CPU usage is ~5-10% on the primary. We are currently having problems with high global write lock and background flush avg on our primary. Our global write lock is between 30-40% on our primary despite only having ~200 insert/update/deletes per second (see MMS output below). We also have noticed that our background flush avg is between 2 and 15 seconds. Unfortunately, this is causing a serious amount of slow queries (up to 50 updates/inserts > 100ms per second). We have considered sharding but feel that mongodb should be performing better than this.

Testing

This says to me that we are having issues writing to our HDDs but running a simple iostat shows that our utilization on sdc (the disk we are writing to) is NOT maxed out and is between 20 and 40%:

$ iostat -x 1

The result for 4 seconds:

```
Linux 2.6.32-279.14.1.el6.openlogic.x86_64 (mongodb3-wus) 05/08/2014 _x86_64_ (4 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
5.28 0.00 1.82 5.50 0.00 87.40

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.05 0.04 0.06 0.11 3.25 1.23 26.13 0.00 18.07 14.87 0.25
sdc 0.02 216.57 1.70 95.83 216.22 3106.45 34.07 9.27 95.07 4.32 42.11
sdb 0.00 11.35 0.01 0.56 0.05 95.25 169.44 0.01 18.44 0.11 0.01

avg-cpu: %user %nice %system %iowait %steal %idle
2.56 0.00 2.05 0.00 0.00 95.38

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %u

Your mongo stat shows higher number of updates vs inserts. One thing that could cause high write lock issues is if your updates typically are increasing the document size and causing the document to move in the data file. We ran into this ourselves, but we were working with mongo support at the time to figure out so I don't remember what metric or stat would tell you this is the case. This would likely only be an issue if your document sizes were very large. We ended up splitting out a sub array that was always being added to into its own collection so that we were just adding new documents instead of modifying an existing one.

The usePowerOf2Sizes flag on the collection can also help alleviate this by giving the documents room for growth. This is apparently the default now on 2.6, but you would need to turn it on if you're not on 2.6 yet. Setting that is described here: http://docs.mongodb.org/manual/reference/command/collMod/

StackExchange Database Administrators Q#64713, answer score: 5