Store data for x hours using NodeJS

Question

I need to collect data every 10 minutes and store that data for 32 hours. This means, that I will have 192 entries of this data and I need to access those 192 entires often.

At first I wanted to use an array and just push the elements into it. If my counter is higher tan 192, I would remove the first element (the one that was collected 32 hours ago). However, I think that could be pretty slow.

Another approach would be to write the data into the /tmp directory on linux and actually don't really care about deleting it, but just reading the latest 192 entries.

I haven't found much about this googling, so could anyone point me into the right direction?

Also, it's no problem if the data is lost on restarts/reboots of the machine.

Answer 1

Storing this in an array will be orders of magnitute faster than saving it to disk and having to re-read whatever file each time you want to access it.

Speed should actually be completely irrelevant for this type of operation, so you can just do somthing like the following:

var updateArray = function(arr, newData) {
    if (arr.length > 192) {
        arr = arr.slice(1); // Remove first entry
    }
    arr[arr.length] = newData;
    return arr;
};

Answer 2

For this kind of requirement I would suggest storing the data in a fast cache like redis or memcache. You can specify the time when the keys should get expired on their own. This way it will be very very fast and you do not have to manage it explicitly. So I would suggest cache to store the data for a longer time and cache server can be separated from your application server this way data will not get lost on restarts.

Also I want to point out that /tmp directory can not be trusted to store anything as it will be cleaned up from time to time by the operating system.

If you want to store data then use db instead.