CouchDB load spike (even with low traffic)?

Question

We're been running CouchDB v1.5.0 on AWS and its been working fine. Recently AWS came out with new prices for their new m3 instances so we switched our CouchDB instance to use an m3.large. We have a relatively small database with < 10GB of data in it.

Our steady state metrics for it are system loads of 0.2 and memory usages of 5% or so. However, we noticed that every few hours (3-4 times per day) we get a huge spike that floors our load to 1.5 or so and memory usage to close to 100%.

We don't run any cronjobs that involve the database and our traffic flow about the same over the day. We do run a continuous replication from one database on the west coast to another on the east coast.

This has been stumping me for a bit - any ideas?

Answer 1

Just wanted to follow up on this question in case it helps anyone.

While I didn't figure out the direct answer to my load spike question, I did discover another bug from inspecting the logs that I was able to solve.

In my case, running "sudo service couchdb stop" was not actually stopping couchdb. On top of that, every couple of seconds a new process of couch would try and spawn only to be blocked by the existing couchdb process.

Ultimately, removing the respawn flag /etc/init.d/couchdb fixed this error.