Uploading big Files

Question

I am an amateur ASP.net developer working on my first job (a friends website). ASP.net v4.0, using VS2010.

His company makes 3D models (using a 3D printer). The website is currently in development but can be found here. I would be the first to admit that the code has been a bit rushed and hacked in, but my friend is quite happy with what he has so far.

One of the requirements is that his customers need to be able to upload their model design files, which could be up to 100 MB each (or maybe more). I am struggling to get this to work properly.

I started by using the built in <asp:FileUpload ID="FileUpload1" runat="server" /> tag and an animated gif image, similar to the idea described in Joe Stagner's tutorial - thanks Joe, I like your presentations a lot.

This worked ok for a small test file but doesn't give any indication of upload progress. So I tried to improve my solution using ideas developed from Sunasara Imdadhusen in his code project article. My upload code looks like this:

Task t = Task.Factory.StartNew(() =>
{
    byte[] buffer = new byte[UPLOAD_BUFFER_BYTE_SIZE];

    // Upload the file in chunks so that we can measure how long it is taking.
    using (FileStream fs = new FileStream(Path.Combine(newQuotePath, filename), FileMode.Create))
    {
        DateTime stopwatch = DateTime.Now;

        while (stats.Uploaded < stats.TotalSize)
        {
            int bytecount = postedFile.InputStream.Read(buffer, 0, UPLOAD_BUFFER_BYTE_SIZE);
            fs.Write(buffer, 0, bytecount);

            stats.Uploaded += bytecount;

            double dRate = UPLOAD_BUFFER_BYTE_SIZE / Math.Abs((DateTime.Now - stopwatch).TotalSeconds);
            stats.Rate = (int)(Math.Min(dRate, int.MaxValue));

            // Sleep is for debugging only!
            //System.Threading.Thread.Sleep(2000);

            stopwatch = DateTime.Now;
        }
    }
}, TaskCreationOptions.LongRunning);

Where stats is a reference to a class that is stored as a session variable, and is accessed from a javascript function running in a setInterval(...) (while the upload is taking place) by the PageMethod:

[System.Web.Services.WebMethod]
[System.Web.Script.Services.ScriptMethod]
public static UploadStatus GetFileUploadStatus()
{
    UploadStatus stats = (UploadStatus)HttpContext.Current.Session["UploadFileStatus"];

    if ((stats != null) && (stats.IsReady))
    {
        return stats;
    }
    else
    {
        return null;
    }
}

This worked on my local machine (using the thread sleep to slow down the upload). So I published it to our host 123-Reg and it didn't work as expected. The animated gif comes on when the upload starts, but the progress bar doesn't start moving. The file input control and the submit button (in an IFrame) that should get disabled as soon as the upload starts take ages to become disabled. Then the web page just hangs there. After waiting for a while I clicked the page refresh button. When the page refreshed it showed that my test file had been uploaded successfully. I tried a 16 KB file and a 1.6 MB file.

Since this worked on my local machine, I suspect that this is happening because our website host (123-Reg) is using a web farm.

Anyway I thought this was going to be easy but it is not, and so I had a look for some open-source upload progress bars. I had a look at NeatUpload but it says "By default, NeatUpload won't work properly on a web garden or web farm", and it also says to get it to work on a web farm "Specify the same random 32-hex-digit decryptionKey attribute in the section of each server's Web.config". But I don't think I have access to 123-Reg's server config files(?).

My friend has suggested that perhaps we could use a service such as dropbox, but I had a look and I have no idea how to add this to our website. This might be a good option since they might be optimized for uploads.

Any advice or suggestions would be very much appreciated - thanks.

---

EDIT: The story so far ... still struggling with this.

I looked into using dropbox in detail (and other cloud storage providers such as SkyDrive etc). Seems that these services are designed around providing applications that interact with individual user's storage. I wanted all users to be able to upload files to MY dropbox folder but without sharing (customers should not have access to other customer's design files). Anyway I carried on, setup a dropbox account, installed the Sharpbox SDK and reprogrammed my upload code (the bit inside the task action). This seemed to work ok on my local machine, it was a bit slower because it was uploading to the dropbox server (I didn't need the Thread.Sleep). I published the website to the 123-Reg server and got a similar, unusable experience as before.

So far I had been testing on IE9, and I just happend to try it out with Chrome and I noticed a funny thing. Just after clicking the upload button but before the page refreshed Chrome showed an upload % complete dialog in the bottom left. This ran to 100%, then my controls started to show some action before the whole page started to hang again.

According to MSDN the HttpPostedFile: "By default, all requests, including form fields and uploaded files, larger than 256 KB are buffered to disk"

So does this mean that my Task is being started after the painful part of waiting for the upload (which is actually happening between the client and a buffer)? If this is the case then it would not make sense to make it more painful by then sending the file off to dropbox right? And my progress bar is tracking the progress of the wrong bit?

(Today I am going to check the error handling of my code since I have a feeling that the reason it is hanging is because it is not recovering from an exception properly). It certainly feels like I am learning a lot by doing this.

Answer 1

This might be a bit overkill but you could look into using BluImp's jQuery File Upload tool (Demo site here: http://blueimp.github.com/jQuery-File-Upload/ ) - a developer called Max Pavlov has modified the original version (originally for non-dotnet technologies) for use in MVC 3.

It can be found here on git hub https://github.com/maxpavlov/jQuery-File-Upload.MVC3 . I have successfully implemented this in both MVC 3 and MVC 4 Beta. The only thing I had to do to make it work effectively in MVC 4 is remove some of the ClientDependancy (This is a DLL that handles bundling and minification of JS and CSS files) code as this replicates functionality already in MVC 4 but not in MVC 3. Additionally I have added some pages to the wiki over on GitHub describing what I did although its not complete. If you decide to go down this route I could update some details there with my more recent findings. My notes so far on MVC 4 integration can be found here https://github.com/maxpavlov/jQuery-File-Upload.MVC3/wiki/MVC-4---EnableDefaultBundles .

Incidentally I have managed to upload files up to about 2Gb using this tool and have found it quite flexible!