Multiple instances use a co-located caching but fail to access, good named caching implementation required

Question

We have been transferring our services and MVC4 website to the cloud, overall this process went fine. Except for caching, since we have moved to Azure it would also be wise to use some kind of caching which azure provides. We choose for co-located / dedicated caching role which has the advantage that the cache is used over all the instances.

Setting up the caching worked fine, I've got a named caching client which I only initialize when its required. It is set up in a inherited layer of the controllers. As soon as one of the functions is called, it checks if the connection to the data-cache is still there or its created. This all seems to work fine, but I'm building a module do retrieve prices. And multiple ajax inserts (views which get inserted into the page with use of javascript) use these functions, some of them are called at the same time, by multiple ajax views. Some of these views then return either a 404 or 500 error, and I cant explain where these are coming from except a non working caching, or something alike.

Can someone help me with a good implementation of the named caching (co-located or dedicated), since all I can find is many examples illustrating the initializing of the DataCacheFactory, but not of the data insertion and retrieval.

Below is the code as I have it now, I've tried more ways with use of locking etc but this one so far worked best.

private static object magicStick = new object();

    private static DataCacheFactory dcf = null;
    private static DataCache priceCache = null;

    protected void CreateCacheFactory()
    {
        dcf = new DataCacheFactory();

    }
    protected void CreatePricesCache()
    {
        if (dcf == null)
        {
            CreateCacheFactory();
        }
        priceCache = dcf.GetCache("Prices");

    }
protected PriceData GetPrices(int productID)
{
    if (priceCache == null)
    {
        CreatePricesCache();
    }
    string cacheKey = "something";
    lock (magicStick)
    {
        PriceData datas = priceCache.Get(cacheKey) as PriceData;
        if (datas == null)
        {
            lock (magicStick)
            {
                Services svc = new Services();
                PriceData pData = svc.PriceService.GetPrices(productID);
                if (pData != null && pData.Offers != null && pData.Offers.Count() > 0)
                {
                    datas = pData;
                    datas.Offers = datas.Offers.OrderBy(pr => (pr.BasePrice + pr.ShippingCosts)).ToArray();
                    priceCache.Add(cacheKey, datas, new TimeSpan(0, cachingTimePricesKK, 0));
                }
            }
        } 
        return datas;
    }
}

As soon as I get to a page where there are pricelists and the function above is called multiple times with the same arguments, there is a 5-10% chance that it returns an error rather then returning the results. Can anybody help me, im totally stuck with this for a week now and its eating me up inside.

Answer 1

First I'd move your cache and cacheFactory instantiation out of your getPrices method. Also, evaluate your need for the lock - this may be causing timeouts. Another VERY important observation - you are using a constant cache key and saving/retrieving data for every productId with the same cache key. You should be using a cache key like: var cacheKey = string.format("priceDatabyProductId-{0}", productId);. You need to set some breakpoints and examine exactly what you are caching and retrieving from the cache. The code as written will save the first productId to the cache and then keep returning that data regardless of the productId.

Here is a full working example we use in production using the "default" named cache in dedicated cache roles:

public static class MyCache
{
    private static DataCacheFactory _cacheFactory = null;
    private static DataCache ACache
    {
        get
        {
            if (_cacheFactory == null)
            {
                try
                {
                    _retryPolicy.ExecuteAction(() => { _cacheFactory = new DataCacheFactory(); });
                    return _cacheFactory == null ? null : _cacheFactory.GetDefaultCache();
                }
                catch (Exception ex)
                {
                    ErrorSignal.FromCurrentContext().Raise(ex);
                    return null;
                }
            }

            return _cacheFactory.GetDefaultCache();
        }
    }

    public static void FlushCache()
    {
        ACache.Clear();
    }

    // Define your retry strategy: retry 3 times, 1 second apart.
    private static readonly FixedInterval  _retryStrategy = new FixedInterval(3, TimeSpan.FromSeconds(1));

    // Define your retry policy using the retry strategy and the Windows Azure storage
    // transient fault detection strategy.
    private static RetryPolicy _retryPolicy = new RetryPolicy<StorageTransientErrorDetectionStrategy>(_retryStrategy);

    // Private constructor to prevent instantiation
    // and force consumers to use the Instance property
    static MyCache()
    { }

    /// <summary>
    /// Add an item to the cache with a key and set a absolute expiration on it
    /// </summary>
    public static void Add(string key, object value, int minutes = 90)
    {
        try
        {
            _retryPolicy.ExecuteAction(() => { ACache.Put(key, value, TimeSpan.FromMinutes(minutes)); });
        }
        catch (Exception ex)
        {
            ErrorSignal.FromCurrentContext().Raise(ex);
        }
    }

    /// <summary>
    /// Add the object with the specified key to the cache if it does not exist, or replace the object if it does exist and set a absolute expiration on it
    /// only valid for Azure caching
    /// </summary>
    public static void Put(string key, object value, int minutes = 90)
    {
        try
        {  
            _retryPolicy.ExecuteAction(() => { ACache.Put(key, value, TimeSpan.FromMinutes(minutes)); });
        }
        catch (Exception ex)
        {
            ErrorSignal.FromCurrentContext().Raise(ex);
        }
    }

    /// <summary>
    /// Get a strongly typed item out of cache
    /// </summary>
    public static T Get<T>(string key) where T : class
    {
        try
        {
            object value = null;

            _retryPolicy.ExecuteAction(() => { value = ACache.Get(key); });

            if (value != null) return (T) value;
            return null;
        }
        catch (DataCacheException ex)
        {
            ErrorSignal.FromCurrentContext().Raise(ex);
            return null;
        }
        catch (Exception ex)
        {
            ErrorSignal.FromCurrentContext().Raise(ex);
            return null;
        }
    }
    /// <summary>
    /// Microsoft's suggested method for cleaning up resources such as this in a static class
    /// to ensure connections and other consumed resources are returned to the resource pool
    /// as quickly as possible.
    /// </summary>
    public static void Uninitialize()
    {
        if (_cacheFactory == null) return;
        _cacheFactory.Dispose();
        _cacheFactory = null;
    }
}

Note: this is also using the Transient Fault Handling block from the Enterprise Library for transient exception fault handling.