mongo converting standalone to replica set causes services down

I have standalone node1 already running for a long time. Now I want to convert it to replica set. I followed steps in https://docs.mongodb.org/v2.4/tutorial/convert-standalone-to-replica-set/ to convert standalone to replica set.

After I added a member in mongo client connnected to primary, command line prompt changed from ReplicaSet0:PRIMARY> to ReplicaSet0:SECONDARY>. Then I found my production's services were down.

I check my sentry(an error collection service), and found lots of errors my Ruby code threw:


Moped::Errors::ConnectionFailure: Could not connect to a primary node for replica set #, ]>


These are my operation and mongo output:

ReplicaSet0:PRIMARY> rs.add("node2")
{ "ok" : 1 }
ReplicaSet0:PRIMARY> rs.conf()
{
  "_id" : "ReplicaSet0",
  "version" : 2,
  "members" : [
    {
      "_id" : 0,
      "host" : "node1:27017"
    },
    {
      "_id" : 1,
      "host" : "node2:27017"
    }
  ]
}
ReplicaSet0:PRIMARY> rs.status()
Thu Oct 22 15:40:13.762 DBClientCursor::init call() failed
Thu Oct 22 15:40:13.763 Error: error doing query: failed at src/mongo/shell/query.js:78
Thu Oct 22 15:40:13.763 trying reconnect to 127.0.0.1:27017
Thu Oct 22 15:40:13.764 reconnect 127.0.0.1:27017 ok
ReplicaSet0:SECONDARY>

You can see PRIMARY became to SECONDARY.
Why did it happen? I think it caused my services down. And How can I avoid it? Please help me.

Update0:

mongo.conf(Yeah, that is all.)

dbpath=/data/mongodb
logpath=/var/log/mongodb/mongodb.log
logappend=true
bind_ip = 0.0.0.0
journal=true
replSet=ReplicaSet0

Update1: rs.status()

```
ReplicaSet0:SECONDARY> rs.status()
{
"set" : "ReplicaSet0",
"date" : ISODate("2015-10-22T07:58:14Z"),
"myState" : 2,
"members" : [
{
"_id" : 0,
"name" : "kuankr:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 2463,
"optime" : Timestamp(1445499598, 19),
"optimeDate" : ISODate("2015-10-22T07:39:58Z"),
"self" : true

Basically, the scenario from the comments is what has happened here. You have added a new host to the set (mongo-primary) and this host is not reachable from your original host (kuankr). That means that you have a replica set with 2 hosts, but only one healthy. When that occurs you cannot satisfy the requirement for electing a primary - which is that >50% of the votes (or a strict majority) are required to elect a primary.

In a 2 node set, both nodes must be available and voting to elect a primary. In a 3 node set, you need 2 out of 3, in a 4 node set you need 3 out of 4, in a 5 node set you need 3 out of 5 etc.

This is why it is always recommended to have an odd number of nodes in your set. I would recommend adding an arbiter that can be reached by your original primary so that it can be elected again. Then, with the immediate problem solved, work out why the original primary cannot talk to the new node (most common issues: firewall, routing, incorrect bind IP on new node).

Update based on comments:

To force an addition if the helpers will not work on a secondary, then you can do something like this:

cfg = rs.conf()
// here's what my sample config member array looks like - adjust as necessary
> cfg.members 
[
    {
        "_id" : 0,
        "host" : "mongod_A.example.net:27017"
    },
    {
        "_id" : 1,
        "host" : "mongod_B.example.net:27017"
    }
]
// let's manually add an arbiter
> cfg.members[2] = {
... "_id" : 2,
... "host": "arbiter:27017",
... "arbiter": true
... }
// now our cfg object looks like this
> cfg
{
    "_id" : "rs",
    "version" : 7,
    "members" : [
        {
            "_id" : 0,
            "host" : "mongod_A.example.net:27017"
        },
        {
            "_id" : 1,
            "host" : "mongod_B.example.net:27017"
        },
        {
            "_id" : 2,
            "host" : "arbiter:27017",
            "arbiter" : true
        }
    ]
}
// Finally, reconfigure with force on the secondary
rs.reconfig(cfg, {force : true})

You can also just remove the "bad" node using this similar procedure

StackExchange Database Administrators Q#118875, answer score: 3