replicaset (+pymongo) reading from slave when master is down

Question

It seems as if with pymongo, connection will always attempt to read from PRIMARY, and when it is down, a socket error is raised until new election process is completed.

Given the fact one of the purposes of replicSet is to balance read load, this seems like a major flaw unless I am missing a key concept here.

I have provided slave_ok thingy, yet, as long as there is no primary available, no read, let alone write are available either.

I have launched 3 instance of mongod at ports 8910,8911, and 8912 and took them down once after another, when the last one remain, there was no way to read from it, even thou mongo (cli) let read.

versions in use:

mongodb: 2.0.2
pymongo: 2.1.1

pymongo Console output

>>> collection = Connection("localhost:8910, localhost:8911, localhost:8912", slave_okay=True).testdb['TEST']
>>> len(list(collection.find()))
0
>>> collection.insert({"a": 1})
ObjectId('4f4a491bb9efb72ec8000045')
>>> len(list(collection.find()))
1

taking down 1 of 3 instances (the primary one), and then...

>>> len(list(collection.find()))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py", line 703, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py", line 666, in _refresh
    self.__uuid_subtype))
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py", line 616, in __send_message
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 880, in _send_message_with_response
    sock = self.__socket()
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 686, in __socket
    sock, from_pool = self.__pool.get_socket(host, port)
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 165, in get_socket
    self.sock = (pid, self.connect(host, port))
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 127, in connect
    s.connect((host, port))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 111] Connection refused
>>> 

a new primary has been elected, then...

>>> len(list(collection.find()))
1

taking down the second instance out of 3, and then...

>>> collection.insert({"c": 3})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/collection.py", line 312, in insert
    continue_on_error, self.__uuid_subtype), safe)
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 811, in _send_message
    sock = self.__socket()
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 674, in __socket
    host, port = self.__find_node()
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 659, in __find_node
    raise AutoReconnect(', '.join(errors))
pymongo.errors.AutoReconnect: could not connect to  localhost:8911: [Errno -5] No address associated with hostname, could not connect to localhost:8911: [Errno 111] Connection refused, could not connect to localhost:8912: [Errno 111] Connection refused, could not connect to  localhost:8912: [Errno -5] No address associated with hostname, localhost:8910 is not primary or master
>>> 
>>> 
>>> collection.insert({"c": 3})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/collection.py", line 312, in insert
    continue_on_error, self.__uuid_subtype), safe)
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 811, in _send_message
    sock = self.__socket()
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 674, in __socket
    host, port = self.__find_node()
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 659, in __find_node
    raise AutoReconnect(', '.join(errors))
pymongo.errors.AutoReconnect: could not connect to  localhost:8911: [Errno -5] No address associated with hostname, could not connect to localhost:8911: [Errno 111] Connection refused, could not connect to localhost:8912: [Errno 111] Connection refused, could not connect to  localhost:8912: [Errno -5] No address associated with hostname, localhost:8910 is not primary or master
>>> 
>>> 
>>> len(list(collection.find()))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py", line 703, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py", line 666, in _refresh
    self.__uuid_subtype))
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py", line 616, in __send_message
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 880, in _send_message_with_response
    sock = self.__socket()
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 674, in __socket
    host, port = self.__find_node()
  File "/usr/local/lib/python2.7/dist-packages/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/connection.py", line 659, in __find_node
    raise AutoReconnect(', '.join(errors))
pymongo.errors.AutoReconnect: could not connect to  localhost:8911: [Errno -5] No 
address associated with hostname, could not connect to localhost:8911: [Errno 111] 
Connection refused, could not connect to localhost:8912: [Errno 111] Connection refused, 
could not connect to  localhost:8912: [Errno -5] No address associated with hostname, 
localhost:8910 is not primary or master
>>> 

mongo (cli) output

SECONDARY> rs.status()
{
        "set" : "myset",
        "date" : ISODate("2012-02-26T15:09:49Z"),
        "myState" : 2,
        "members" : [
                {
                        "_id" : 0,
                        "name" : "localhost:8910",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "t" : 1330268443000,
                                "i" : 1
                        },
                        "optimeDate" : ISODate("2012-02-26T15:00:43Z"),
                        "lastHeartbeat" : ISODate("2012-02-26T15:09:37Z"),
                        "pingMs" : 0,
                        "errmsg" : "socket exception"
                },
                {
                        "_id" : 1,
                        "name" : "localhost:8911",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "optime" : {
                                "t" : 1330268443000,
                                "i" : 1
                        },
                        "optimeDate" : ISODate("2012-02-26T15:00:43Z"),
                        "self" : true
                },
                {
                        "_id" : 2,
                        "name" : "localhost:8912",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "t" : 1330268443000,
                                "i" : 1
                        },
                        "optimeDate" : ISODate("2012-02-26T15:00:43Z"),
                        "lastHeartbeat" : ISODate("2012-02-26T15:09:37Z"),
                        "pingMs" : 0,
                        "errmsg" : "socket exception"
                }
        ],
        "ok" : 1
}
SECONDARY> 
SECONDARY> 
SECONDARY> db.TEST.find().count()
54
SECONDARY> db.TEST.insert({eeee:23232323})
not master
SECONDARY> 
SECONDARY> db.TEST.find().count()
54

Answer 1

There are two factors to solve this, ReplicaSetConnection and ReadPreference.SECONDARY.

i.e.

ReplicaSetConnection("localhost:8910,localhost:8911,localhost8912",  
      replicaSet='myset', read_prefererence=ReadPreference.SECONDARY) 

Answer 2

use the latest PyMongo and instantiate a ReplicaSetConnection instead of a Connection. Use ReadPreference to direct reads to the primary or secondary; the preference can be set at the ReplicaSetConnection, and also overridden per-query:

http://api.mongodb.org/python/current/api/pymongo/index.html#pymongo.ReadPreference