HiveBrain v1.2.0
Get Started
← Back to all entries
snippetModerate

How do I (safely) kill long running operations in MongoDB?

Submitted by: @import:stackexchange-dba··
0
Viewed 0 times
killoperationsmongodblongrunninghowsafely

Problem

Occasionally operations get out of control in MongoDB and might end up running for hundreds of seconds, and impact performance until they are killed or completed.

When that happens, I know I have killOp() available to me but how do I kill only targeted long running operations without also killing (for example) the long running operations involved in replication (which can be dangerous)?

Solution

This can be a little tricky, but the fact that the MongoDB shell is basically a Javascript interpreter gives us decent options in terms of filtering. Here is the function I use to accomplish this:

// kills long running ops in MongoDB (taking seconds as an arg to define "long")
// attempts to be a bit safer than killing all by excluding replication related operations
// and only targeting queries as opposed to commands etc.
killLongRunningOps = function(maxSecsRunning) {
    currOp = db.currentOp();
    for (oper in currOp.inprog) {
        op = currOp.inprog[oper-0];
        if (op.secs_running > maxSecsRunning && op.op == "query" && !op.ns.startsWith("local")) {
            print("Killing opId: " + op.opid
            + " running over for secs: "
            + op.secs_running);
            db.killOp(op.opid);
        }
    }
};


This will only kill queries above the maxSecsRunning threshold and will not touch anything running against the local database, which is where the oplog lives (and hence is the database that is involved in the long running replication ops. It is relatively easy to add criteria to the inner if conditional to more precisely target operations as needed based on specific needs.

The code is also available as a gist (where I will remember to update it on an ongoing basis).

Code Snippets

// kills long running ops in MongoDB (taking seconds as an arg to define "long")
// attempts to be a bit safer than killing all by excluding replication related operations
// and only targeting queries as opposed to commands etc.
killLongRunningOps = function(maxSecsRunning) {
    currOp = db.currentOp();
    for (oper in currOp.inprog) {
        op = currOp.inprog[oper-0];
        if (op.secs_running > maxSecsRunning && op.op == "query" && !op.ns.startsWith("local")) {
            print("Killing opId: " + op.opid
            + " running over for secs: "
            + op.secs_running);
            db.killOp(op.opid);
        }
    }
};

Context

StackExchange Database Administrators Q#60029, answer score: 19

Revisions (0)

No revisions yet.