Connect to a child process after navigating away from the page

Question

I am using SailsJS for a web application which basically lets the user download and process any video from the internet(say youtube). The user enters a link to the video and my sails app downloads the video if available and then starts processing the downloaded video using a shell script(Come OpenCV processing to find different frames).

This process takes a very long time to complete, and the user can navigate away from the page and do whatever he wants. Now, to check on the progress by visiting this page later I need to be able to connect with the child process that was created earlier for this video file.

I have come up with two possible solutions:

1) Using gearman to implement a job server and connect to it every time the user navigates to the page and start getting the callback events and show the progress based on them. This is the first time I'll be using gearman.

2) Somehow storing the processID of the child process in the session/db and then using it to find the process using ps-node.

Which of these is the better approach(if you think they'll work fine)? Or is there any other solution I don't know about? Any pointers in the right direction will be appreciated.

Answer 1

Let's start with the second option. Don't use it. Simply because this way your site users will have sort of more control over the number of processes running on your server then you will.

Number one is way better, but using a separate job server seems like a bit of overkill to me (I have to admit though that I'm not fully informed the scale of your plans).

Bottom line, I would use a message/job queue (kue seems like a perfect fit to me) and store the progress in DB or (preferably) Redis (or whatever cache you are using).