Get actual upload progress on Azure Blob

Question

i know that this has been already asked, but the marked solution is not correct. Usually this article is marked as solution: https://docs.microsoft.com/en-us/archive/blogs/kwill/asynchronous-parallel-blob-transfers-with-progress-change-notification-2-0

It works and give an actual progress, but not the real time progress (and in some cases it gives a totally wrong value). Let me explain:

It gives the progress on the local read buffer, so when i upload something my first "uploaded value" is the read buffer total size. In my case this buffer is 4mb so every file smaller than 4mb results completed in 0 seconds for the progress bar, but it takes the real upload time to complete for real.

Also, if you try to kill your connection just before the upload start it gives as actual progress the first buffer size, so for my 1mb file i get 100% progress while disconnect.

I found another article with another solution, it reads the http response from azure everytime it complete a single block upload, but i need my blocks to be 4mb (since max block count for a single file is 50.000) and its not a perfect solution even with low block size.

The first article overrides the stream class and create a ProgressStream class with an ProgressChanged event that is triggered every time a read is done, there is some way to know the actual uploaded bytes when that ProgressChanged is triggered?

Answer 1

You can do this by using code similar to https://docs.microsoft.com/en-us/archive/blogs/kwill/asynchronous-parallel-block-blob-transfers-with-progress-change-notification (version 1.0 of the blog post you referenced), but instead of calling m_Blob.PutBlock you would instead upload the block with an HTTPWebRequest object and use the progress events from the HTTPWebRequest class. This introduces a lot more code complexity and you would have to add some additional error handling.

The alternative would be to download the Storage Client Library source code from GitHub and modify the block upload methods to track and report progress. The challenge you will face here is that you will have to make these same changes to every new version of the SCL if you plan on staying up to date with the latest fixes.

Answer 2

I must admit I didn't check if everything is as you desired, but here are my 2 cents on uploading with progress indication.

public async Task UploadVideoFilesToBlobStorage(List<VideoUploadModel> videos, CancellationToken cancellationToken)
{
    var blobTransferClient = new BlobTransferClient();
    //register events
    blobTransferClient.TransferProgressChanged += BlobTransferClient_TransferProgressChanged;
    //files
    _videoCount = _videoCountLeft = videos.Count;
    foreach (var video in videos)
    {
        var blobUri = new Uri(video.SasLocator);
        //create the sasCredentials
        var sasCredentials = new StorageCredentials(blobUri.Query);
        //get the URL without sasCredentials, so only path and filename.
        var blobUriBaseFile = new Uri(blobUri.GetComponents(UriComponents.SchemeAndServer | UriComponents.Path,
            UriFormat.UriEscaped));
        //get the URL without filename (needed for BlobTransferClient (seems to me like a issue)
        var blobUriBase = new Uri(blobUriBaseFile.AbsoluteUri.Replace("/"+video.Filename, ""));

        var blobClient = new CloudBlobClient(blobUriBaseFile, sasCredentials);
        //upload using stream, other overload of UploadBlob forces to put online filename of local filename
        using (FileStream fs = new FileStream(video.FilePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
        {
            await blobTransferClient.UploadBlob(blobUriBase, video.Filename, fs, null, cancellationToken, blobClient, 
                new NoRetry(), "video/x-msvideo");
        }
        _videoCountLeft -= 1;
    }

    blobTransferClient.TransferProgressChanged -= BlobTransferClient_TransferProgressChanged;
}

private void BlobTransferClient_TransferProgressChanged(object sender, BlobTransferProgressChangedEventArgs e)
{
    Console.WriteLine("progress, seconds remaining:" + e.TimeRemaining.Seconds);
    double bytesTransfered = e.BytesTransferred;
    double bytesTotal = e.TotalBytesToTransfer;
    double thisProcent = bytesTransfered / bytesTotal;
    double procent = thisProcent;
    //devide by video amount
    int videosUploaded = _videoCount - _videoCountLeft;
    if (_videoCountLeft > 0)
    {
        procent = (thisProcent + videosUploaded) / _videoCount;
    }

    procent = procent * 100;//to real %
    UploadProgressChangedEvent?.Invoke((int)procent, videosUploaded, _videoCount);
}

Actually Microsoft.WindowsAzure.MediaServices.Client.BlobTransferClient should be able to do concurrent uploads but there is no Method for uploading multiple yet it has properties for NumberOfConcurrentTransfers and ParallelTransferThreadCount, not sure how to use this.

There is a bug in this BlobTransferClient, when uploading using the localFile parameter, it will use the filename of that file, while I gave permissions on a specific file name in the SaSLocator.

This example shows how to upload from a client (not on server), so we don't need any CloudMediaContext which is usually the case. Everything about SasLocators can be found here.