MongoDB+Azure+Android: com.mongodb.WriteConcernException err: "not master" code: "10058"

Question

Background:

Hi, I am running a MongoDB replica set on Azure and have connected to it remotely from within an Android app. I've gotten reads to work great from all instances (Updated: Because they are allowed to read on primary and secondary nodes). But, writes to the database still have intermittent errors with the following error, because writes must be done on the primary node only.

Also, if you could provide any more specific resources for dealing with this issue, then that would be very helpful too. I've already gone through most of the docs and searched quite a bit for this error.

Question:

How to prevent this error and allow writes 100% of the time?

E/AndroidRuntime(): com.mongodb.WriteConcernException: {
        "serverUsed" : "/<my-remote-ip>:27017" , "err" : "not master" , 
        "code" : 10058 , "n" : 0 , "lastOp" : { "$ts" : 0 , "$inc" : 0} , 
        "connectionId" : 1918 , "ok" : 1.0}

Stack trace:

E/AndroidRuntime(13731): FATAL EXCEPTION: Thread-7629
E/AndroidRuntime(13731): Process: com.myapplication.examplemongodb, PID: 13731
E/AndroidRuntime(13731): com.mongodb.WriteConcernException: { "serverUsed" : "/<my-remote-ip>:27017" , "err" : "not master" , "code" : 10058 , "n" : 0 , "lastOp" : { "$ts" : 0 , "$inc" : 0} , "connectionId" : 1918 , "ok" : 1.0}
E/AndroidRuntime(13731):    at com.mongodb.CommandResult.getException(CommandResult.java:77)
E/AndroidRuntime(13731):    at com.mongodb.CommandResult.throwOnError(CommandResult.java:110)
E/AndroidRuntime(13731):    at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:102)
E/AndroidRuntime(13731):    at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:142)
E/AndroidRuntime(13731):    at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:115)
E/AndroidRuntime(13731):    at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:248)
E/AndroidRuntime(13731):    at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:204)
E/AndroidRuntime(13731):    at com.mongodb.DBCollection.insert(DBCollection.java:76)
E/AndroidRuntime(13731):    at com.mongodb.DBCollection.insert(DBCollection.java:60)
E/AndroidRuntime(13731):    at com.mongodb.DBCollection.insert(DBCollection.java:105)
E/AndroidRuntime(13731):    at com.myapplication.examplemongodb.ActivityMain$1.run(ActivityMain.java:83)
E/AndroidRuntime(13731):    at java.lang.Thread.run(Thread.java:841)

Notes:

• I am using mongo-java-driver v2.11.3.
• I used the mongo-azure library to help create the MongoDB replica set with two worker roles.
  • (If you have any more resources for this, then I'd love to read it. I've already read the GitHub readme, this, this, and a few other things not related to MongoDB/Azure. But, those resources are not updated, nor detailed.)

Possible solutions:

• Has something to do with having a replica set, I think.
• I'm not if sure this is happening because I have just a two instance replica set (one primary and one secondary) and they are fighting (read: voting) over who wants to be primary. Perhaps, an arbiter is required? But, I do not know how to do that currently.

Update:

• Thanks to the help of @David Makogon, I'm pretty sure the issue lies with how I have set up the connection to Azure, and how I access the worker roles. So, here's my updated notes on how the system is configured:
  • Two worker roles (MongoDB.WindowsAzure.MongoDBRole), which I connect directly to via TCP Input Endpoint port 27017 from the Android app. As @David said, I currently have no control over which which instance is connected to.
  • One web role (MongoDB.WindowsAzure.Manager), which I don't do anything with, has one HTTP Input Endpoint on port 80. This was just made by default with the mongo-azure library I mentioned above. I'm not sure if there is anything I should be doing with this.

Answer 1

If the instances are all sitting behind a single load-balanced Input endpoint (e.g. 27017), then each time your client machine connects to the endpoint, it will be connected to potentially different nodes in the replicaset cluster (and you'd have no control over which instance you went to). This might explain why you sometimes attempt to write to the non-master and get an error, yet all your reads work (since you probably set up the MongoDB cluster to allow reads on secondary nodes).

Worker roles also support InstanceInput endpoints, which allow you to set an externally-facing port range (say, 27017-27019), mapping to a single port on the worker instances themselves (e.g. 27017). If you do this, your client app can now connect to all three instances directly (27017, 27018, 27019). Many drivers support replicaset connections, so it would be able to figure out which node is the master, directing all writes to it. I don't know if the driver you're using on Android supports replicasets. If the driver doesn't support replicasets, then you'll probably want to consider putting up an API tier which then does all communication to the database (a good practice to follow in general, anyway, and you can look at Azure's Mobile Services for a quick way to implement this).

So... if your replicaset cluster's endpoint is configured as Input, this likely explains the issue you're seeing, which should be resolvable by switching the endpoint type to InstanceInput.