Node.js(&MongoDB) server crashes , Database operations halfway?

Question

I have a node.js app with mongodb backend going to production in a week and i have few doubts on how to handle app crashes and restart .

Say i have a simple route /followUser in which i have 2 database operations

/followUser
----->Update User1 Document.followers = User2
----->Update User2 Document.followers = User1
----->Some other mongodb(via mongoose)operation

What happens if there is a server crash(due to power failure or maybe the remote mongodb server is down ) like this scenario :

----->Update User1 Document.followers = User2
 SERVER CRASHED , FOREVER RESTARTS NODE

What happens to these operations below ? The system is now in inconsistent state and i may have error everytime i ask for User2 followers

----->Update User2 Document.followers = User1
----->Some other mongodb(via mongoose)operation

Also please recommend good logging and restart/monitor modules for apps running in linux.

Right now im using domains for to catch exceptions , doing server.close , but before process.exit() i want to make sure all database transactions are done , can i check this by testing if the event loop is empty or not (how?) and then process.exit(1) ?

Answer 1

You need transactions for this, and since MongoDB doesn't have them here is a workaround http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/

Answer 2

One way to address this problem is to add cleanup code to your application that runs whenever the application starts. You write the cleanup code to perform sanity checks on any portions of your data that can be updated in multiple steps like your example and then repairs that data in whatever way make sense for your application.

Depending on the complexity of your application/data, this may also require that you keep a log of actions the app was trying to perform, but that gets complicated real fast. Ideally it's more a matter of refreshing denormalized data and deleting partial data.

You want to do this during startup rather than shutdown as there's no guarantee your shutdown code will fully run and if you're shutting down because of an exception you don't know what the state of your system is at that point.

Answer 3

the solution given by vkurchatkin in this link is a workaround in case your appserver crashes, because you will be able of knowing which transactions were pending at that moment. If you implement this in your code you can create cleanup code when your system restart as suggested by JohnnyHK. The code you mention (catching exceptions, test when closing, etc) will not work because...well.. your server crashed! ;-)

That said, this is done using the database, so you will have to warrantee to a certain point that your database does not crash. I would suggest your use replication for that. It is basically a cluster of servers that recovers itself if one node fails, and also you can make some check to make sure that the data reached the servers and is safe.

Hope this helps.