Node.js global variable property is purged

Question

my problem is not about "memory leakage", but about "memory purge" of node.js (expressjs) app.

My app should maintain some objects in memory for the fast look-up's during the service. For the time being (one or two days) after starting the app, everthing seemed fine, until suddenly my web client failed to look-up the object bacause it has been purged (undefined). I suspect Javascript GC (garbage collection). However, as you can see in the psedu-code, I assigned the objects to the node.js "global" variable properties to prevent GC from purging them. Please give me some clue what caused this problem.

Thanks much in advance for your kind advices~

My node.js environments are node.js 0.6.12, expressjs 2.5.8, and VMWare cloudfoundry node hosting.

Here is my app.js pseudo-code :

var express = require("express");
var app = module.exports = express.createServer();

// myMethods holds a set of methods to be used for handling raw data.
var myMethods = require("myMethods");

// creates node.js global properties referencing objects to prevent GC from purging them
global.myMethods = myMethods();
global.myObjects = {};

// omited the express configurations

// creates objects (data1, data2) inside the global.myObjects for the user by id.
app.post("/createData/:id", function(req, res) {

    // creates an empty object for the user.
    var myObject = global.myObjects[req.prams.id] = {};

    // gets json data.
    var data1 = JSON.parse(req.body.data1);
    var data2 = JSON.parse(req.body.data2);

    // buildData1 & buildData2 functions transform data1 & data2 into the usable objects.
    // these functions return the references to the transformed objects.
    myObject.data1 = global.myMethods.buildData1(data1);
    myObject.data2 = global.myMethods.buildData2(data2);

    res.send("Created new data", 200);
    res.redirect("/");
});

// returns the data1 of the user.
// Problem occurs here : myObject becomes "undefined" after one or two days running the service.
app.get("/getData1/:id", function(req, res) {

    var myObject = global.myObjects[req.params.id];
    if (myObject !== undefined) {
        res.json(myObject.data1);
    } else {
        res.send(500); 
    }
});

// omited other service callback functions.

// VMWare cloudfoundry node.js hosting.
app.listen(process.env.VCAP_APP_PORT || 3000);

Answer 1

Any kind of cache system (whether is roll-your-own or a third party product) should account for this scenario. You should not rely on the data always being available on an in-memory cache. There are way too many things that can cause in-memory data to be gone (machine restart, process restart, et cetera.)

In your case, you might need to update your code to see if the data is in cache. If it is not in cache then fetch it from a persistent storage (a database, a file), cache it, and continue.

Answer 2

Exactly like Haesung I wanted to keep my program simple, without database. And like Haesung my first experience with Node.js (and express) was to observe this weird purging. Although I was confused, I really didn't accept that I needed a storage solution to manage a json file with a couple of hundred lines. The light bulb moment for me was when I read this

If you want to have a module execute code multiple times, then export a function, and call that function.

which is taken from http://nodejs.org/api/modules.html#modules_caching. So my code inside the required file changed from this

var foo = [{"some":"stuff"}];
export.foo;

to that

export.foo = function (bar) {
var foo = [{"some":"stuff"}];
return foo.bar;
}

And then it worked fine :-)

Answer 3

Then I suggest to use file system, I think 4KB overhead is not a big deal for your goals and hardware. If you familiar with front-end javascript, this could be helpful https://github.com/coolaj86/node-localStorage