"Temporary failure" on Azure caching service

Question 1

Wrap your calling code in appropriate protection (try / catch) and then cope with the failure at the app tier. The commodity platform offered in any cloud can (and does) have these sorts of issues from time-to-time. You need to bake in logging and log somewhere like Azure Diagnostics (http://msdn.microsoft.com/en-us/library/gg433048.aspx) for later troubleshooting.

Question 2

I still haven't figured out what the problem was, and just ended up following Simon W's advice about wrapping everything in try/catches up the wazoo. But because it's not 100% intuitive, and took me several tries to get the code for cache retrieval right, I thought I'd post it here for anybody else who's interested.

public TValue Get(string key, Func<TValue> missingFunc)
{
    // We need to ensure that two processes don't try to calculate the same value at the same time. That just wastes resources.
    // So we pull out a value from the _cacheLocks dictionary, and lock on that before trying to retrieve the object.
    // This does add a bit more locking, and hence the chance for one process to lock up everything else.
    // We may need to add some timeouts here at some point in time. It also doesn't prevent two processes on different
    // machines from trying the same bit o' nonsense. Oh well. It's probably still a worthwhile optimization.
    key = _keyPrefix + "." + key;
    var value = default(TValue);
    object cacheLock;
    lock (_cacheLocks)
    {
        if (!_cacheLocks.TryGetValue(key, out cacheLock))
        {
            cacheLock = new object();
            _cacheLocks[key] = cacheLock;
        }
    }
    lock (cacheLock)
    {
        // Try to get the value from the cache.
        try
        {
            value = _cache.Get(key) as TValue;
        }
        catch (SerializationException ex)
        {
            // This can happen when the app restarts, and we discover that the dynamic entity names have changed, and the desired type 
            // is no longer around, e.g., "Organization_6BA9E1E1184D9B7BDCC50D94471D7A730423456A15BBAFB6A2C6AC0FF94C0D41"
            // If that's the error, we should probably warn about it, but no point in logging it as an error, since it's more-or-less expected.
            _logger.Warn("Error retrieving item '" + key + "' from Azure cache; falling back to missingFunc(). Error = " + ex);
        }
        catch (Exception ex)
        {
            _logger.Error("Error retrieving item '" + key + "' from Azure cache; falling back to missingFunc(). Error = " + ex);
        }

        // If we didn't get anything interesting, then call the function that should be able to retrieve it for us.
        if (value == default(TValue))
        {
            // If that function throws an exception, don't swallow it.
            value = missingFunc();

            // If we try to put it into the cache, and *that* throws an exception, 
            // log it, and then swallow it.
            try
            {
                _cache.Put(key, value);
            }
            catch (Exception ex)
            {
                _logger.Error("Error putting item '" + key + "' into Azure cache. Error = " + ex);
            }
        }
    }
    return value;
}

You can use it like so:

var user = UserCache.Get(email, () =>
    _db.Users
        .FirstOrDefault(u => u.Email == email)
        .ShallowClone());