How to resolve unmodifiable replica set state

Question

We are using mongoDB 3.0, and our setup uses six instances with one PRIMARY and the rest of them as SECONDARY.

We were trying to remove three of these using the following command sequence -

rs.remove("host1:port") //same for host2, host3

Now that we wanted to add those back, there is a weird state that the rs.conf() results in. i.e

The replica set before removal was at the _id:1,2 and 3 Now that we have added them back, they are at the _id:7,8, and 9

Why does this happen so, and how do I correct the rs.conf()?

Also if I again try and remove host1, host2 or host3, they are not getting removed.

rs.remove("host1:port")

displays the conf() value back again, so we are unable to remove them now. Is there a way to overcome from such state?

---

rs.conf()

{
    "_id" : "shard1",
    "version" : 15,
    "members" : [
        ...
        {
            "_id" : 4,
            "host" : "host4:port",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {

            },
            "slaveDelay" : 0,
            "votes" : 1
        },
        {
            "_id" : 5,
            "host" : "host5:port",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {

            },
            "slaveDelay" : 0,
            "votes" : 1
        },
        {
            "_id" : 8,
            "host" : "host1:port",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {

            },
            "slaveDelay" : 0,
            "votes" : 1
        }
    ],
    "settings" : {
        "chainingAllowed" : true,
        "heartbeatTimeoutSecs" : 10,
        "getLastErrorModes" : {

        },
        "getLastErrorDefaults" : {
            "w" : 1,
            "wtimeout" : 0
        }
    }
}

Answer 1

When you add hosts, _id is set always one bigger than what is current max _id. If you really want to "fix" that, you can always do:

var cfg=rs.conf()
cfg.member[3]._id=0
cfg.member[4]._id=1
rs.reconfig(cfg)

Same way, you can change anything or everything in your rs configuration. Remove or add new hosts, change an order of hosts in the list.