Manually change hidden node to primary when all other nodes fail

We have a setup MongoDB cluster that consists of a total of 5 nodes of which 3 are on-premise and 2 nodes on cloud server. We have configured 2 nodes on cloud servers to be hidden and priority-0 to avoid delays in writes due to latency.

In case of an outage at our DC, the only nodes that will remain in our cluster will be the hidden nodes that are on the cloud. In such a scenario, can we somehow make the hidden nodes primary and secondary? And later add more nodes to it?

Currently you can do this only manually, i.e. write a script and execute it maybe by frequent cron job.

First you must define clear condition when you like to elevate a hidden member and when demote it back. Have a look at output of db.hello() and/or rs.status().members

In my application I have a script like this one:

const newPriority = 10
const members = db.adminCommand({ replSetGetStatus: 1 }).members.filter(x => x.stateStr == 'HIDDEN');

for (let member of members) {
   let cfg = db.adminCommand({ replSetGetConfig: 1 }).config;
   
   // `_id` can be different to position in Array, esp. when you have added and removed members.
   let idx = cfg.members.findIndex(x => x._id == member._id); 

   // Set 'votes' and 'hidden'
   cfg.members[idx].votes = 1;
   cfg.hidden = false;
   cfg.version++;
   var cmd = { replSetReconfig: (({ _id, version, term, members }) => ({ _id, version, term, members }))(cfg), maxTimeMS: 1000 * 60 * 10 };
   cmd.replSetReconfig = cfg;
   db.adminCommand(cmd);

   // Set 'priority'
   cfg = replicaSet.adminCommand({ replSetGetConfig: 1 }).config;
   cfg.members[idx].priority = newPriority;
   cfg.version++;
   cmd = { replSetReconfig: (({ _id, version, term, members }) => ({ _id, version, term, members }))(cfg), maxTimeMS: 1000 * 60 * 10 };
   cmd.replSetReconfig = cfg;
   db.adminCommand(cmd);
}

I set votes and priority in two steps, because I have a PSA-ReplicatSet, see Modify PSA Replica Set Safely. I assume, this is required only when your ReplicaSet has an ARBITER, in your case it should work with a single reconfiguration.

In order to demote members back to HIDDEN, change attributes votes, priority, hidden accordingly.

Have a careful look at rs.status().majorityVoteCount, see Replica Set Elections and Replica Set Deployment Architectures.

If 3 nodes are down, then the remaining 2 nodes in the cloud cannot elect a new PRIMARY. You must elevate the HIDDEN member to {hidden: false, votes: 1, priority: >0 } but also demote the other members to {votes: 0, priority: 0}, then the 2 node in the cloud can elect a new PRIMARY.

Another note, when you connect to the ReplicaSet, then you should use readPreference=primaryPreferred. Otherwise, when you reconfigure a ReplicaSet member then the state can change (for example from PRIMARY to SECONDARY) and your script may fail within the loop.

StackExchange Database Administrators Q#319883, answer score: 2