fetching file from job server to my frontend server

Question

I need some suggestions on how to fetch dynamically generated files via jobs running on my remote job server to my frontend server running a user panel to manage jobs.

I do not want my job server's IP to be exposed so allowing user to directly download from the job server is undesired.

I would also need some Jquery library to show download speed, a progress bar of some sort.

Answer 1

I think it would help to know the answers to a few questions.

Off the top of my head:

• How large can the files be?
• How long is it likely to take to download a file?
• What format are the files?
• Are the files you're fetching from the job server static? (or, are they being regenerated each time?)

Edit:

jQuery UI has a progress bar. (docs)

From your user panel you would have a link like this:

<a href="remoteTxtFile.php?file_id=100">File 100</a>

This link should start a jQuery.getJSON() request to remoteTxtFile.php on your frontend server, which:

• Checks to see if the requested file has already been downloaded.
• If it has, return a JSON object like this: {'location': '/cache/file_100.txt'}
  • The Ajax success callback can then show a link to the file.
• If not, start downloading the file using something (ideas further down) which can output progress to a text file. Save the file to a 'cache' folder. Return a JSON object like this: {'progress':0}
• Periodically (every second or so) fire another getJSON() to a PHP script which reads the download progress txt file and returns a JSON object like this: {'progress': 52} (percent)
• When the download is complete, from the PHP return JSON like this: {'location': '/cache/file_100.txt'}
  • When the Ajax success callback sees 'location', the file is done, so show a link to it.

I'm not sure what you should use for the actual downloading of files. The problem with using PHP and stream_notification_callback to do the download (and output progress) is that the entire file would be downloaded into memory, which is no good for a 1 GB file.

Perhaps using wget to do the actual download would work better.