Connect to remote MongoDB replica set, when replica set connect to each other on local IP addresses

Question

We have setup a mongo replica set on a local network interface (i.e. 192.168.1.x). The replica set members are the only ones that know about this interface. The instances are also on an external interface that is addressable from inside our network (10.x.x.x). When trying to connect to the 'master' only with the Java driver[2.10.1] (via gMongo wrapper) we're running into a problem where a connection will not be made because the app cannot determine the other replica nodes. This happens even if I provide all the external interfaces in the sever list.

I also need to be able to setup an SSH tunnel from my machine to the replica set. Again, I run into the same issue because the replica set provides hostnames and IP addresses that are not applicable to me.

How can I connect to a replica set when the replica knows about each other only over local IP addresses?

Here is an example of the code and the errors that result. The example below assume a tunnel has been setup from localhost to the remote server.

@Grab(group='com.gmongo', module='gmongo', version='1.1')

import com.gmongo.*
import com.mongodb.*

def hosts= [
        [host:'127.0.0.1', port:29017],
        //[host:'127.0.0.1', port:29018],  //have tried with and without the others commented
        //[host:'127.0.0.1', port:29019],
        ]

List<ServerAddress> addrs = hosts.collect {it.port?new ServerAddress(it.host, it.port):new ServerAddress(it.host)}
MongoClientOptions clientOptions = new MongoClientOptions.Builder().build()
def mongo = new GMongoClient(addrs, clientOptions)
def db = mongo.getDB("mydb")

def result = db.tickets.findOne([:])

println result

results in this:

WARNING: couldn't resolve host [mongo03:27017]
May 16, 2013 10:54:59 AM com.mongodb.ReplicaSetStatus$UpdatableReplicaSetNode _addIfNotHere

WARNING: couldn't resolve host [mongo03:27017]
May 16, 2013 10:55:01 AM com.mongodb.ReplicaSetStatus$UpdatableReplicaSetNode findNode

WARNING: couldn't resolve host [mongo02:27017]
May 16, 2013 10:55:01 AM com.mongodb.ReplicaSetStatus$UpdatableReplicaSetNode _addIfNotHere

WARNING: couldn't resolve host [mongo02:27017]
May 16, 2013 10:55:01 AM com.mongodb.ReplicaSetStatus$UpdatableReplicaSetNode findNode
com.mongodb.MongoException: can't find a master

    at com.mongodb.DBTCPConnector.checkMaster(DBTCPConnector.java:518)

WARNING: Server seen down: /192.168.1.11:27017 - java.io.IOException - message: couldn't connect to [/192.168.1.11:27017] bc:java.net.SocketTimeoutException: connect timed out

I'm not sure why it's trying to resolve based on the servers' local config rather than the list of servers I provide.

Answer 1

In my experience, replica sets work best when they have DNS names that both the clients and the servers can resolve.

I have used both pymongo and php MongoClient, and if the client machine uses the IP Addresses of the replica set, but the replica set was setup using DNS names, the client will fail to connect (probably because when asking the replica set which node is the primary, the replica set probably returns the DNS name).

If I use DNS names (I just modify the hosts file on the client) then the connections do not fail.

From the documentation on replica sets:

Each member of the replica set must be accessible by way of resolvable DNS or hostnames

Hope this helps you debug your issue.