Periodic checks in node.js and mongodb (searching for missing record)

Question

I'm receiving some periodic reports from a bunch of devices and storing them into a MongoDB database. They are incoming at about 20-30 seconds. However I would like to check when a device does not send the report for some time (for example the last report is more then 3 minutes old) and I would like to send an email or trigger some other mechanism.

So, the issue is how to check for the missing event in the most correct manner. I considered a cron job and a bunch of timers each related to each device record.

A cron job looks ok but I am fearing that starting a full scan query will overload the server/db and cause performance issues. Is there any kind of database structure that could aid this (some kind of index, maybe?).

Timers are probably the simpler solution but I fear how many timers can I create because I can get quite a number of devices.

Can anybody give me an advice what is the best approach to this? Thanks in advance.

Answer 1

Do you use Redis or something similar on this server? Set device ID as key with any value, e.g. 1. Expire key in 2-3 min and update expiration every time the device connects. Then fire cron jobs to check if ID is missing. This should be super fast.

Also you may use MongoDB's timed collections instead of Redis but in this case you will have to do a bunch of round trips to the DB server. http://docs.mongodb.org/manual/tutorial/expire-data/

Update:

As you do not know what IDs you will be looking for, this rather complicates the matter. Another option is to keep a log in a separate MongoDB collection with a timestamp of last ping you've got from the device.

Index timestamps and query .find({timestamp: {$lt: Date.now() - 60 * 1000}}) to get a list of stale devices.

It's very important that you update existing document rather than create a new one on each ping. So that if you have 10 devices connected you should have 10 documents in this collection. That's why you need a separate collection for this log. There's a great article on time series data. I hope you find it useful http://blog.mongodb.org/post/65517193370/schema-design-for-time-series-data-in-mongodb

Answer 2

An index on deviceid+timestamp handles this neatly.

1. Use distinct() to get your list of devices

2. For each device d,

   db.events.find({ deviceid: d }).sort({ timestamp : -1 }).limit(1)

   gives you the most recent event, whose timestamp you can now compare with the current time.