How we can take backup oplog on every hour and apply on top on full backup for mongodb

We have full lvm snapshot backup. We are trying to take oplog backup in every hour and save as a file. We like to apply the oplog backup on top of full backup if we need to. We like to know how we can do this. We are taking everyday LVM snapshot full backup. Our Oplog is 80GB, which can hold 30 days of oplog. But we are facing issue to apply it. I can take the oplog backup in every hour. But question is : how to apply it? We don't want duplicate or any missing data.

The oplog is idempotent, you can run through the operations in it as many times as you want and you won't get duplicates or issues unless you run the operations on a set of data files in an odd state.

However, it should be noted that as long as you have the journal as part of the LVM snapshot, re-running the oplog is not necessary for a consistent backup.

With that said, if you do have a copy of the oplog, you can "replay" it in two ways:

• Start up a new MongoDB instance, use it to access the oplog backup, and then create a custom script to read each operation and apply it in order (has the benefit of allowing you to filter and being rather quick)

• Use mongodump/mongorestore to replay the oplog (see below)

To use mongodump in this way, you first have to dump it out into BSON format so that it can be resored:

mongodump --dbpath /path/to/folder/with/oplog/files -d local -c oplog.rs -o oplogDump

Next you move the bson file out of the folder (this is basically for convenience):

mkdir oplogRestore
mv oplogDump/local/oplog.rs.bson oplogRestore/oplog.bson

Now you can use mongorestore to replay the oplog and apply it to a given running instance:

mongorestore --host host:port --oplogReplay oplogRestore

You can also use the oplogLimit option if you only wish to replay up to a certain point. See Asya's excellent answer here for more on that.

Please note that none of this will be particularly quick, nor is it strictly necessary (as mentioned above).

StackExchange Database Administrators Q#73146, answer score: 4