MongoReplicationSetClient returns an empty set in pymongo 2.6.3

Question

I'm running into an issue with the MongoReplicaClient while trying to connect to a remote replica set. For some reason MongoReplicaSetClient is just returning an empty set for any replica set that is not on my local host. The strange part is that MongoClient works fine. I'm not sure how to debug this any further since it is not erroring out in anyway. Networking appears to be ruled out since connecting via MongoClient works fine. Does anyone have an idea why this could be happening?

[Update]

After further investigation it appears that this issue affects Pymongo 2.6.3 when the local host is unable to resolve the remote hostname without using it's FQDN. The problem is that if the FQDN is provided to MongoReplicaSetClient in this situation it will just return an empty set. Here's how I reproduced the issue:

Environment

python2.7 (2.7.3-0ubuntu3.4)  
DISTRIB_ID=Ubuntu  
DISTRIB_RELEASE=12.04  
DISTRIB_CODENAME=precise  
DISTRIB_DESCRIPTION="Ubuntu 12.04.2 LTS"

Procedure

1) Confirm that remote hostname cannot resolve

05:03:48 [js-dev][503]$ ping -c3 hq-sre-mongodb-01
ping: unknown host hq-sre-mongodb-01

05:03:59 [js-dev][504]$ ping -c3 hq-sre-mongodb-01.eng.nutanix.com
PING hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11) 56(84) bytes of data.
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=1 ttl=63 time=0.222 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=2 ttl=63 time=0.217 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=3 ttl=63 time=0.247 ms

Test on pymongo 2.5

1) Connect to the replica set using hostname (Breaks as expected)

>>> MongoReplicaSetClient('hq-sre-mongodb-01', replicaSet='rs0')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_replica_set_client.py", line 516, in __init__
    raise ConnectionFailure(str(e))
pymongo.errors.ConnectionFailure: hq-sre-mongodb-01:27017: [Errno -2] Name or service not known

2) Connect to the replica set using FQDN (Works)

>>> pymongo.version
'2.5'
>>> MongoReplicaSetClient('hq-sre-mongodb-01.eng.nutanix.com', replicaSet='rs0')
MongoReplicaSetClient([u'sv2-sre-mongodb-03:27017', u'hq-sre-mongodb-01:27017', u'sv2-sre-mongodb-01:27017', u'sv2-sre-mongodb-02:27017', u'hq-sre-mongodb-02:27017'])

Test on pymongo 2.6.3

1) Connect to the replica set using hostname (Breaks as expected)

>>> pymongo.version
'2.6.3'
>>> MongoReplicaSetClient('hq-sre-mongodb-01', replicaSet='rs0')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_replica_set_client.py", line 745, in __init__
        raise ConnectionFailure(str(e))
  pymongo.errors.ConnectionFailure: hq-sre-mongodb-01:27017: [Errno -2] Name or service not known

2) Connect to the replica set using FQDN (Returns an empty set)

>>> pymongo.version
'2.6.3'
>>> MongoReplicaSetClient('hq-sre-mongodb-01.eng.nutanix.com', replicaSet='rs0')
MongoReplicaSetClient([])

Confirmation

You can see that once the localhost can resolve by hostname the issue does not present itself any longer in 2.6.3. It's almost as if MongoReplicaSetClient doesn't know how to handle a FQDN.

1) Confirm remote hostname resolves

05:33:32 [js-dev][501]$ ping -c3 hq-sre-mongodb-01
PING hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11) 56(84) bytes of data.
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=1 ttl=63 time=0.263 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=2 ttl=63 time=0.259 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=3 ttl=63 time=0.281 ms

05:33:36 [js-dev][502]$ ping -c3 hq-sre-mongodb-01.eng.nutanix.com
PING hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11) 56(84) bytes of data.
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=1 ttl=63 time=0.154 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=2 ttl=63 time=0.256 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=3 ttl=63 time=0.275 ms

2) Confirm MongoReplicaSetClient connection using both hostname and FQDN

>>> pymongo.version
'2.6.3'
>>> MongoReplicaSetClient('hq-sre-mongodb-01', replicaSet='rs0')
MongoReplicaSetClient([u'sv2-sre-mongodb-03:27017', 'hq-sre-mongodb-01:27017', u'sv2-sre-mongodb-01:27017', u'sv2-sre-mongodb-02:27017', u'hq-sre-mongodb-02:27017'])
>>> MongoReplicaSetClient('hq-sre-mongodb-01.eng.nutanix.com', replicaSet='rs0')
MongoReplicaSetClient([u'sv2-sre-mongodb-03:27017', u'hq-sre-mongodb-01:27017', u'sv2-sre-mongodb-01:27017', u'sv2-sre-mongodb-02:27017', u'hq-sre-mongodb-02:27017'])

Answer 1

This is a PyMongo bug. I've reported it in PYTHON-608 and I'll fix it for the next release of PyMongo, version 2.7, due out in about a month.

The behavior you saw in PyMongo 2.5 was buggy: The MongoReplicaSetClient connected to the FQDN of the member you provided, but the member wasn't actually reachable using its hostname from the replica set config. Auto-reconnect, failover, and read preferences wouldn't work, so the client should've raised an error instead of allowing you to proceed.

The PyMongo 2.6.3 behavior is buggy too, obviously. In 2.6.3, the client knew it couldn't reach any members using the hostnames it discovered, but it didn't raise an error. Instead, it allowed you to create an unusable client. The correct behavior is to raise an error if none of the hostnames in the replica set config are reachable from the client.

Answer 2

This appears to be related to networking issues between the mongod nodes. If your host is having DNS issues and the stars align you may see this issue. In my testing I've seen MongoReplicaSetClient return an empty list if all nodes are not reachable by both hostname and FQDN. However, after a running a few test scenarios I cannot reliably reproduce the issue so I'll close this out. If you run into this, ensure that DNS connectivity to your mongod nodes is fully functional and you should be all set.