Secondary and Arbiter stuck in Startup

I'm using MongoDB version 3.0.0. I'm trying to setup mongodb replication on our machine. Replication was initially setup but due to some changes on the VM the entire thing crashed. When I tried to set it up again, the secondary and Arbiter became stuck in StartUp Mode.

In the mongoDB conf file I have set

replSet=ReplicaSet1

I added 2 machines using the command

rs.add("10.235.96.12:27017")
rs.add("10.235.96.12:27017")

But after this on the primary when I do a rs.status() secondary and arbiter are still shown in StartUp

```
ReplicaSet1:PRIMARY> rs.status()
{
"set" : "ReplicaSet1",
"date" : ISODate("2015-07-31T04:45:57.260Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "BOSPROD9:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 63104,
"optime" : Timestamp(1438257913, 1),
"optimeDate" : ISODate("2015-07-30T12:05:13Z"),
"electionTime" : Timestamp(1438254975, 2),
"electionDate" : ISODate("2015-07-30T11:16:15Z"),
"configVersion" : 7,
"self" : true
},
{
"_id" : 1,
"name" : "10.235.96.12:27017",
"health" : 1,
"state" : 0,
"stateStr" : "STARTUP",
"uptime" : 62663,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2015-07-31T04:45:56.520Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
"pingMs" : 0,
"configVersion" : -2
},
{
"_id" : 2,
"name" : "10.235.96.13:27017",

From "BOSPROD9", try to connect with the mongoshell to the other serves:

$ mongo --host 10.235.96.12 --port 27017

$ mongo --host 10.235.96.13 --port 27017

(Telnet is not the same.) If this doesn't work, it might be firewall or BindIP.

Check bind_ip (should be 0.0.0.0, change in mongodb.conf is it's 127.0.0.1):

$ netstat -nap | grep :27017 | grep LISTEN
tcp        0      0 0.0.0.0:27018           0.0.0.0:*               LISTEN      -

Try to look at the log-files on 10.235.96.12 and 10.235.96.13, why they are stuck. Did they receive the configuration?

Try to reconfigure this way:

mongo> var cfg = {_id:"ReplicaSet1",members:[{_id:0, host:"BOSPROD9:27017"},{_id:1, host:"10.235.96.12:27017"},{_id:2, host:"10.235.96.13:27017",arbiterOnly:true}]};
mongo> rs.reconfig(cfg);

Solution:

Use hostnames instead of ip's: give all servers a hostname, update the hosts-files and use hostnames in the rs-configuration. It seems that mongodb advices not to use ip addresses, but hostnames.

StackExchange Database Administrators Q#108625, answer score: 6