MongoDB replicaSet config - strange behaviour

Question

I'm trying to implement mongoDB replication, made up of 4 nodes in the form of Virtual Machines,

Info: I use virtualbox, and v.machines comunicate with each other throught the host-only adapter. Communication is been tested, and ervery node can ping other nodes.

this is the output of rs.conf() command :

rs0:PRIMARY> rs.conf()
{
    "_id" : "rs0",
    "version" : 4,
    "members" : [
        {
            "_id" : 0,
            "host" : "192.168.56.1:27017"
        },
        {
            "_id" : 1,
            "host" : "192.168.56.101:27018"
        },
        {
            "_id" : 2,
            "host" : "192.168.56.102:27019"
        },
        {
            "_id" : 3,
            "host" : "192.168.56.103:27020"
        }
    ]
}

this is the output of the command rs.status()

rs0:PRIMARY> rs.status()
{
    "set" : "rs0",
    "date" : ISODate("2013-12-14T16:09:36Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "192.168.56.1:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 3207,
            "optime" : Timestamp(1387034904, 1),
            "optimeDate" : ISODate("2013-12-14T15:28:24Z"),
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "192.168.56.101:27018",
            "health" : 1,
            "state" : 6,
            "stateStr" : "UNKNOWN",
            "uptime" : 2542,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2013-12-14T16:09:35Z"),
            "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
            "pingMs" : 1,
            "lastHeartbeatMessage" : "still initializing"
        },
        {
            "_id" : 2,
            "name" : "192.168.56.102:27019",
            "health" : 1,
            "state" : 6,
            "stateStr" : "UNKNOWN",
            "uptime" : 2497,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2013-12-14T16:09:35Z"),
            "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
            "pingMs" : 0,
            "lastHeartbeatMessage" : "still initializing"
        },
        {
            "_id" : 3,
            "name" : "192.168.56.103:27020",
            "health" : 1,
            "state" : 6,
            "stateStr" : "UNKNOWN",
            "uptime" : 2472,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2013-12-14T16:09:36Z"),
            "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
            "pingMs" : 1,
            "lastHeartbeatMessage" : "still initializing"
        }
    ],
    "ok" : 1
}

from the last command it seems the replicaSet is still initializating, but looking at the 4 instances of mongod, i don't know, it seems like here is something not working properly..

im wondering why all the nodes try to contact only the primary instance ignoring the ohers? And why when the connection get accepted, it try again to contact the same node, failing to do that because it say that "Couldn't load config yet".. i really need a thrust for understand the problem, if is necessary any other command output, or information in general just let me know i'll post them.

thanks in advance for any help

Answer 1

solved - i just checked the connections betweens all members in the way of mongo connection like rubenfa suggested in his comment.

mongo --host 192.168.56.103 --port 27020

and every connection between each members work properly using Host-only Adapter.

the main problem i posted at the beginning, was related to the fact that i didn't checked inside the 3 secondary nodes, if there was others local database created from the previous attempts to configure the replicaSet. I just used to check the primary node, and delete the local db from there, without check other nodes.

Also, remember to delete all the local dbs from all the nodes before try to reconfigure replicaSet

Answer 2

For anyone else stumbling on this:

Make sure your primary's hostname is resolvable from the members, or otherwise your replSetConfig is using your primary's IP instead of hostname or fqdn. By default mongo uses the primary's hostname in the "host" configuration field, making all the other members fail to communicate with it if they don't have the information in the /etc/hosts file.