upload huge numbers of files to blob

Question

Do you know best way to upload too many files to Azure Blob container?

I am currently do something to upload multiple files to Azure blob storage. The number of files may be huge, like 30,000 or more(each file could be sized of 10KB~1MB). Firstly, I have a list of files locations, then I would use Parallel.Foreach to upload the files. code snippet like this:

`List locations=...

Parallel.Foreach(locations, location=>
{
...
UploadFromStream(...);
...
});`
The codes run to inconsistence results.

Sometimes it runs well, I can see all files uploaded to the Azure blob container.

Sometimes, I will got exceptions like this:

Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature., Inner Exception: The remote server returned an error: (403) Forbidden

Sometimes, I got a timeout exception.

I have worked against the issue for several days, unfortunatly, I havn't got a perfect solution yet. So I want to know how do you do when you handling similar scenario, how do you do when upload too many files to Azure blob storage?

Answer 1

Finally, I have not found what's wrong with my code.

However, I have found solution for this issue. I expire Parallel.Foreach, just use common foreach. Then I use BeginuploadFromStream method instead of UploadFromStream, it actually upload files asynchronously.

So far, it runs prefectly, more stable, without any exception happens.