How to remove a node from BigCouch cluster?

Question

I created BigCouch cluster with 4 nodes, simply by sending PUT to nodes databases:

curl -X PUT http://bigcouch-a.example.com:5986/nodes/bigcouch@bigcouch-a.example.com -d {}
curl -X PUT http://bigcouch-a.example.com:5986/nodes/bigcouch@bigcouch-b.example.com -d {}
curl -X PUT http://bigcouch-a.example.com:5986/nodes/bigcouch@bigcouch-c.example.com -d {}
curl -X PUT http://bigcouch-a.example.com:5986/nodes/bigcouch@bigcouch-d.example.com -d {}

All nodes were registered properly, sending GET to membership returns all 4 nodes:

{"all_nodes":[
    "bigcouch@bigcouch-a.example.com",
    "bigcouch@bigcouch-b.example.com",
    "bigcouch@bigcouch-c.example.com",
    "bigcouch@bigcouch-d.example.com"]
 "cluster_nodes":[
    "bigcouch@bigcouch-a.example.com",
    "bigcouch@bigcouch-b.example.com",
    "bigcouch@bigcouch-c.example.com",
    "bigcouch@bigcouch-d.example.com"]
}

The question is: how to remove a node?

I tried to send DELETE to the nodes database

curl -X DELETE http://bigcouch-a.example.com:5986/nodes/bigcouch@bigcouch-d.example.com

but i got an error:

{"error":"conflict", "reason":"Document update conflict."}

Once I simply stopped node d then my membership returned:

{"all_nodes":[
    "bigcouch@bigcouch-a.example.com",
    "bigcouch@bigcouch-b.example.com",
    "bigcouch@bigcouch-c.example.com"]
 "cluster_nodes":[
    "bigcouch@bigcouch-a.example.com",
    "bigcouch@bigcouch-b.example.com",
    "bigcouch@bigcouch-c.example.com",
    "bigcouch@bigcouch-d.example.com"]
}

So node d disappeared from "all_nodes" but still exists as a "cluster_nodes member.

I cannot create any new database at this state. Each time I receive a message:

could not load validation funs {function_clause,
    [{lists, flatmap,
    [#Fun<couch_db.36.90395586>,
    {nodedown, <<"progress not possible">>}]},
    {couch_db, '-load_validation_funs/1-fun-1-', 1}]}

Any ideas how to fix it?

Answer 1

The nodes db is a couchdb database so you need to pass the current revision to update or delete a document.

See my previous answer on how to move shards: moving a shard from one bigcouch server to another (for balancing)