Modify and reuse MongoDB oplog

Question

Is it possible to modify MongoDB oplog and play it?

The error caused the update to be added to more documents than anticipated by overwriting some of the data. The data was restored from the backup and reintegrated, so nothing was really lost, but I was wondering if there was a way to change the oplog to delete or modify the abusive update and play it back.

I have no in-depth knowledge of MongoDB's internal functions, so the informative answers along the lines of “you don’t understand how this works, this is so” will also be considered for adoption.

Answer 1

One of the serious problems when using these applications or human errors is that the offensive record of the primary will be immediately replicated to the secondary.

This is one of the reasons why users use "slaveDelay" - the ability to start one of your secondary nodes with a fixed time delay (of course, this will help you if you find an error or error during a period of time that is shorter than the delay on this secondary).

If you do not have this setting, you must rely on the backup to recreate the state of the records you need to restore to a preliminary error state.

Perform all operations on a separate, offline copy of your data - only after verifying that everything was correctly recreated if you then transfer the corrected data to your production system.

This requires the last backup copy (say, the backup is X hours), and the oplog on your cluster must store more than X hours. I did not indicate which node the oplog is because (a) each member of the replica set has the same contents in oplog and (b) it is possible that your oplog size is different from different node members, in which case you want to check the "largest "

So, let's say your most recent backup is 52 hours, but fortunately you have an oplog that stores data for 75 hours (yay).

You already realized that all of your nodes (primary and secondary) have bad data, so you can restore this last backup to a new mongod. Here you will restore these records to the point that they were right before the abusive update, and then you can simply transfer them to the current initial one, from where they will be replicated to all secondary ones.

When restoring a backup, create the mongodump of your oplog collection with this command:

mongodump -d local -c oplog.rs -o oplogD

Move oplog to your own directory by renaming it to oplog.bson:

 mkdir oplogR mv oplogD/local/oplog.rs.bson oplogR/oplog.bson 

Now you need to find the operation "insult". You can display oplog in a human- bsondump form using the bsondump command in the oplogR / oplog.bson file (and then use grep or not find a “bad” update). Alternatively, you can query the source oplog in the replica set using the use local and db.oplog.rs.find() in the shell.

Your goal is to find this entry and mark its field ts .

It might look like this:

"ts" : Timestamp( 1361497305, 2789 )

Note that the mongorestore command has two parameters, one of which is called --oplogReplay , and the other is oplogLimit . You will now play this oplog on the restored stand-alone server, but you will stop before this update operation.

The command will be (host and port where your newly restored backup is):

mongorestore -h host --port NNNN --oplogReplay --oplogLimit 1361497305:2789 oplogR

This will lead to the restoration of each operation from the oplog.bson file in the oplogR directory, stopping right before writing to ts with the Timestamp value (1361497305, 2789).

Recall that the reason you did this in a separate copy is because you can check the restoration and playback of the created correct data - after you check it, you can write the restored records to the appropriate location in the real primary (and allow replication to distribute adjusted records to secondary).