Add Cache-Control and Expires headers to Azure Storage Blobs

StackOverflow https://stackoverflow.com/questions/4507657

Question

I'm using Azure Storage to serve up static file blobs but I'd like to add a Cache-Control and Expires header to the files/blobs when served up to reduce bandwidth costs.

Application like CloudXplorer and Cerebrata's Cloud Storage Studio give options to set metadata properties on containers and blobs but get upset when trying to add Cache-Control.

Anyone know if it's possible to set these headers for files?

Was it helpful?

Solution

I had to run a batch job on about 600k blobs and found 2 things that really helped:

  1. Running the operation from a worker role in the same data center. The speed between Azure services is great as long as they are in the same affinity group. Plus there are no data transfer costs.

  2. Running the operation in parallel. The Task Parallel Library (TPL) in .net v4 makes this really easy. Here is the code to set the cache-control header for every blob in a container in parallel:

    // get the info for every blob in the container
var blobInfos = cloudBlobContainer.ListBlobs(
    new BlobRequestOptions() { UseFlatBlobListing = true });
Parallel.ForEach(blobInfos, (blobInfo) =>
{
    // get the blob properties
    CloudBlob blob = container.GetBlobReference(blobInfo.Uri.ToString());
    blob.FetchAttributes();

    // set cache-control header if necessary
    if (blob.Properties.CacheControl != YOUR_CACHE_CONTROL_HEADER)
    {
        blob.Properties.CacheControl = YOUR_CACHE_CONTROL_HEADER;
        blob.SetProperties();
    }
});

OTHER TIPS

Here's an updated version of Joel Fillmore's answer:

Instead of creating a website and using a WorkerRole, Azure now has the ability to run "WebJobs". You can run any executable on demand on a website at the same datacenter where your storage account is located to set cache headers or any other header field.

  1. Create a throw-away, temporary website in the same datacenter as your storage account. Don't worry about affinity groups; create an empty ASP.NET site or any other simple site. The content is unimportant.
  2. Create a console program using the code below which works with the updated Azure Storage APIs. Compile it for release, and then zip the executable and all required DLLs into a .zip file.
  3. Create a WebJob and upload the .zip file from step #2. enter image description here
  4. Run the WebJob. Everything written to the console is available to view in the log file created and accessible from the WebJob control page.
  5. Note the UpdateAzureServiceVersion method. Apparently, by default, Azure storage serves improperly formatted ETags so you may wish to run this code once, for details see: this

The code below runs a separate task for each container, and I'm getting about 70 headers updated per second per container. No egress charges.

using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;

namespace AzureHeaders
{
    class Program
    {
        static StorageCredentials storageCredentials =
            new StorageCredentials("azureaccountname", @"azzureaccountkey");
        private static string newCacheSettings = "public, max-age=7776000"; // 3 months
        private static string[] containersToProcess = { "container1", "container2" };

        static void Main(string[] args)
        {
            var account = new CloudStorageAccount(
                storageCredentials,
                false /* useHttps */);

            CloudBlobClient blobClient = account.CreateCloudBlobClient();

            var tasks = new List<Task>();
            foreach (var container in blobClient.ListContainers())
            {
                if (containersToProcess.Contains(container.Name))
                {
                    var c = container;
                    tasks.Add(Task.Run(() => FixHeaders(c)));
                }
            }
            Task.WaitAll(tasks.ToArray());
        }

        private static async Task FixHeaders(CloudBlobContainer cloudBlobContainer)
        {
            int totalCount = 0, updateCount = 0, errorCount = 0;

            Console.WriteLine("Starting container: " + cloudBlobContainer.Name);
            IEnumerable<IListBlobItem> blobInfos = cloudBlobContainer.ListBlobs(useFlatBlobListing: true);

            foreach (var blobInfo in blobInfos)
            {
                try
                {
                    CloudBlockBlob blockBlob = (CloudBlockBlob)blobInfo;
                    var blob = await cloudBlobContainer.GetBlobReferenceFromServerAsync(blockBlob.Name);
                    blob.FetchAttributes();

                    // set cache-control header if necessary
                    if (blob.Properties.CacheControl != newCacheSettings)
                    {
                        blob.Properties.CacheControl = newCacheSettings;
                        blob.SetProperties();
                        updateCount++;
                    }
                }
                catch (Exception ex)
                {
                    // Console.WriteLine(ex.Message);
                    errorCount++;
                }
                totalCount++;
            }
            Console.WriteLine("Finished container: " + cloudBlobContainer.Name + 
                ", TotalCount = " + totalCount + 
                ", Updated = " + updateCount + 
                ", Errors = " + errorCount);
        }

        // http://geekswithblogs.net/EltonStoneman/archive/2014/10/09/configure-azure-storage-to-return-proper-response-headers-for-blob.aspx
        private static void UpdateAzureServiceVersion(CloudBlobClient blobClient)
        {
            var props = blobClient.GetServiceProperties();
            props.DefaultServiceVersion = "2014-02-14";
            blobClient.SetServiceProperties(props);
        }
    }
}

The latest version of Cerebrata Cloud Storage Studio, v2011.04.23.00, supports setting cache-control on individual blob objects. Right click on the blob object, choose "View/Edit Blob Properties" then set the value for the Cache-Control attribute. (e.g. public, max-age=2592000).

If you check the HTTP headers of the blob object using curl, you'll see the cache-control header returned with the value you set.

Sometimes, the simplest answer is the best one. If you just want to manage a small amount of blobs, you can use Azure Management to change the headers/metadata for your blobs.

  1. Click on Storage, then click on the storage account name.
  2. Click the Containers tab, then click on a container.
  3. Click on a blob, then click on Edit at the bottom of the screen.

In that edit window, you can customize the Cache Control, Content Encoding, Content Language, and more.

Note: you cannot currently edit this data from the Azure Portal

This might be too late to answer, but recently I wanted to do the same in different manner, where I have list of images and needed to apply using powershell script (of course with the help of Azure storage assembly) Hope someone will find this useful in future.

Complete explanation given in Set Azure blob cache-control using powershell script

Add-Type -Path "C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\v2.3\ref\Microsoft.WindowsAzure.StorageClient.dll"

$accountName = "[azureaccountname]"
$accountKey = "[azureaccountkey]"
$blobContainerName = "images"

$storageCredentials = New-Object Microsoft.WindowsAzure.StorageCredentialsAccountAndKey -ArgumentList $accountName,$accountKey
$storageAccount = New-Object Microsoft.WindowsAzure.CloudStorageAccount -ArgumentList $storageCredentials,$true
#$blobClient = $storageAccount.CreateCloudBlobClient()
$blobClient =  [Microsoft.WindowsAzure.StorageClient.CloudStorageAccountStorageClientExtensions]::CreateCloudBlobClient($storageAccount)

$cacheControlValue = "public, max-age=604800"

echo "Setting cache control: $cacheControlValue"

Get-Content "imagelist.txt" | foreach {     
    $blobName = "$blobContainerName/$_".Trim()
    echo $blobName
    $blob = $blobClient.GetBlobReference($blobName)
    $blob.Properties.CacheControl = $cacheControlValue
    $blob.SetProperties()
}

Set storage blob cache-control Properties by PowerShell script

https://gallery.technet.microsoft.com/How-to-set-storage-blob-4774aca5

#creat CloudBlobClient 
Add-Type -Path "C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\v2.3\ref\Microsoft.WindowsAzure.StorageClient.dll" 
$storageCredentials = New-Object Microsoft.WindowsAzure.StorageCredentialsAccountAndKey -ArgumentList $StorageName,$StorageKey 
$blobClient =   New-Object Microsoft.WindowsAzure.StorageClient.CloudBlobClient($BlobUri,$storageCredentials) 
#set Properties and Metadata 
$cacheControlValue = "public, max-age=60480" 
foreach ($blob in $blobs) 
{ 
  #set Metadata 
  $blobRef = $blobClient.GetBlobReference($blob.Name) 
  $blobRef.Metadata.Add("abcd","abcd") 
  $blobRef.SetMetadata() 

  #set Properties 
  $blobRef.Properties.CacheControl = $cacheControlValue 
  $blobRef.SetProperties() 
}

Here's an updated version of Joel Fillmore's answer consuming WindowsAzure.Storage v9.3.3. Note that ListBlobsSegmentedAsync returns a page size of 5,000 which is why the BlobContinuationToken is used.

    public async Task BackfillCacheControlAsync()
    {
        var container = await GetCloudBlobContainerAsync();
        BlobContinuationToken continuationToken = null;

        do
        {
            var blobInfos = await container.ListBlobsSegmentedAsync(string.Empty, true, BlobListingDetails.None, null, continuationToken, null, null);
            continuationToken = blobInfos.ContinuationToken;
            foreach (var blobInfo in blobInfos.Results)
            {
                var blockBlob = (CloudBlockBlob)blobInfo;
                var blob = await container.GetBlobReferenceFromServerAsync(blockBlob.Name);
                if (blob.Properties.CacheControl != "public, max-age=31536000")
                {
                    blob.Properties.CacheControl = "public, max-age=31536000";
                    await blob.SetPropertiesAsync();
                }
            }               
        }
        while (continuationToken != null);
    }

    private async Task<CloudBlobContainer> GetCloudBlobContainerAsync()
    {
        var storageAccount = CloudStorageAccount.Parse(_appSettings.AzureStorageConnectionString);
        var blobClient = storageAccount.CreateCloudBlobClient();
        var container = blobClient.GetContainerReference("uploads");
        return container;
    }
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top