Show realtime data to users with node.js and socket.io

Question

I'm looking to optimize/upgrade our web application from the traditional PHP/MySQL, to a more efficient technology. I've been researching node.js, socket.io and CouchDB, which look promising, but am looking for recommendations and some help in choosing the best technologies for what we want.

So, to outline the what is currently done...

We have devices that capture data and sends it to an FTP server (via .txt file). A PHP script runs every 5 minutes, scanning the FTP for .txt files that the devices send. The contents of the files are inserted into a MySQL database, and the .txt file deleted. Another PHP script that runs every 15 minutes will collate data from the database (eg, sum values for last hour) and produce an XML document, which are read by Flash dials.

Obviously, there are plenty of issues here, and it is far from a real-time solution, which is what we are aiming for. The solution we'd like is, that somehow, it could detect when the txt file is uploaded to the FTP, and only process the data then (rather than having a set interval that runs). The data would still need to be added to the database, and be read by the front-end as soon as it is added to the database. Flash would be removed entirely, as its a little...ick...

With exciting new technologies like node.js and Web Sockets (socket.io), i'm sure we could improve this process greatly! I'm aware of Ajax being able to do something like this, but hear it has rather high overheads compared to Web Sockets. I'm also a little hazy with how a database would work with node.js, not to mention the best option for what we're after...

Thanks!

Answer 1

Node works well with NoSQL, (especially JSON based NoSQL) but since their are modules for handling most databases in node that are async and return javascript objects I wouldn't dismiss using MySQL if you're more comfortable using it.

here's an database agnostic outline.

var fs = require('fs');  //to watch the FTP
var db = require('db');  //your choice of db
var express = require('express'); //http server
var io = require('socket.io'); //for realtime data push
var app = express.createServer(); //create http server
/*...usual express implementation...*/
app.listen(80);
var socket = io.listen(app); 
db.connect( ..., start );
function checkForFiles () {
  fs.readdir( FTPpath, sendFilesToDB );
};
function sendFilesToDB ( err, files ) {
  if( files.length === 0 ) {
    return checkForFiles();
  }
  db.insert( fileToQuery( files.pop() ), function () {
    sendFilesToDB( err, files );
  });
}
function fileToQuery ( file ) {
  ...
  return query;
}
function start () {
  checkForFiles();
  setTimeout( checkData, 1000 );
}
function checkData () {
  db.query( '....', function ( err, data ) {
    socket.broadcast( processData( data ) );
    setTimeout( checkData, 1000 );
  }
}
function processData (data) {
  ...
  return data;
}