Memory Issue while making MSSQL queries using Tedious

Question

I was trying Tedious module which is used to connect to MSSQL databases.

Let me show you my code...

var connection = new Connection(config);
connection.on('connect', function(err) {
// If no error, then good to go...
    console.log("Connected");
    executeStatement();
});

connection.on('error', function(err) {
    console.log("Error");
});




function executeStatement() {   
    var stmt = new Request("Select id from customers", function (err,rowCount) {
        if (err) console.log(err);      
        console.log("RowCount: " + rowCount);       
        process.exit(1);
    });

    stmt.on( 'row', function (columns) {            
        console.log("\t\tRow");
        columns.forEach(function(column) {
            console.log(column.value);
        });
    });
    stmt.on('done', function(rowCount, more) {
        console.log(rowCount + ' rows returned');
    }); 
    connection.execSql(stmt);       
}

My code makes a query to get all the records from the Customers table. And I am listening to "row" event and then I print the column value. The customers table has 120 Million records. The problem I face is, when I run my code, memory footprint of node application starts building up and after some time, node application exits with out of memory error.

As a work around, I started querying customers table in chunks. I order by on a field and then read all the records that corresponds to a value in that field. After all the records are read, I drop the connection and reconnect to the database and read all the records for the next value.

This way, I realized that memory footprint of my code is in check and there is no out of memory error.

I just want to know if some one has faced a similar problem and how to resolve it? Is it possible that we fix the module to handle it or some other solution.

Thank you for the help

Answer 1

The answer lies in the settings. Make sure that rowCollectiononRequestCompletion is set to false and you will see that there is no memory issue. It is also documented on the Tedious webpage.

   var connection = new tds.Connection({
        userName: this.userId,
        password: this.password,
        server: this.server,
        options: {
            database: this.dbName,
            rowCollectionOnRequestCompletion: false
        }
    });