Uploading large files to a server (non blocking)

Question

So I'm working on a php application that needs to communicate with a 4d application. The 4d application uses a SOAP web service for external communication.

The SOAP web service has a function to enable file uploads to the 4d application. Because of the broken nature of the file upload function of the 4d application's web service, we have implemented a progress bar to upload the file to the web server and then after that is successful try to send the file to the 4d application server. But for the client this would not be nice since the person would need to wait for the initial file upload to the server and then again when executing the file upload function to transfer the file from the web server to the 4d application's server.

My question is, Are there any ways to process a file upload but in a non-blocking way? Maybe even bypassing the SOAP function all together.

UPDATE: I was thinking maybe to use something like RabbitMQ to queue each file upload item and then process later using a script or something which is decoupled from the application so that it does not block the application.

Answer 1

Your suggested solution would be the proper way of solving it; accept the upload through the regular web application using either HTML5 file uploads or swfupload to allow a progress bar to be shown for the upload, then store it locally and queue up a Gearman or RabbitMQ task which a worker handles later.

The worker would fetch the file locally and upload it to the remote service (4d / SOAP), and then set the proper status in the database when finished, so that the user knows when the file has been processed and added to the actual server.