AppFabric CreateRoutingClient Error

Question

We have a problem with AppFabric that is causing the below error to occur:

Exception type: ArgumentException 
Exception message: An item with the same key has already been added.
    at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
    at Microsoft.ApplicationServer.Caching.DataCacheFactory.CreateRoutingClient(String cacheName, NamedCacheConfiguration config)
    at Microsoft.ApplicationServer.Caching.DataCacheFactory.CreateNewCacheClient(DataCacheDeploymentMode mode, String cacheName, NamedCacheConfiguration config, IClientChannel channel)
    at Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName)
    at Microsoft.ApplicationServer.Caching.DataCacheFactory.GetDefaultCache()
    ... our code where we are attempting to retrieve a cache item from the default cache by its key

This error happens infrequently in our test environment (we have not found a scenario to reproduce the issue on demand), but seems to always happen in our production environment just after each deployment. Our deployments are automated and we have verified that the steps to deploy to our various environments are the same.

Our server setup for a given environment is as follows:

• Server1 - Hosts our .net website, and is our AppFabric 1.1 server
• Server2 - Hosts our .net website

These servers are load balanced. AppFabric local caching has been enabled on both applications. Our test and production servers are setup the same. We are aware that it would be better to have AppFabric on a dedicated server, but don't think that would cause/resolve this issue.

We are stumped by this issue as we have not found any mention of it elsewhere online and because the stack trace seems to indicate that it is a problem with AppFabric itself. The exception mentions inserting something, but all we are doing when this happens is trying to get the default cache from the DataCacheFactory so that we can retrieve an item from it. So, what does this error mean and how might we resolve it?

Update

Here is the code I am using to create the DataCacheFactory and pull data from the cache:

private static readonly Lazy<DataCacheFactory> _DATA_CACHE_FACTORY = new Lazy<DataCacheFactory>(() => new DataCacheFactory());
private static readonly Lazy<DataCache> _CACHE = new Lazy<DataCache>(() => _DATA_CACHE_FACTORY.Value.GetDefaultCache());

public object Get(string key)
{
    return _CACHE.Value.Get(key);
}

Answer 1

I am 100% certain that the duplicate key error is generated by bad access into the private _myCache property of the DataCacheFactory. That property is a Hashtable. Repeated calls to Hashtable.Add("mykey","myvalue"); will generate this same expection that your'e seeing.

I ran multiple tests and calling GetCache("default") and GetDefaultCache() back to back doesn't produce the error. It is definitely something odd about how App Fabric is trying to populate that. Here is my code that has never generated this error. I wanted to post for reference incase you can see something that is obviously different from what your code is doing

if (cache == null)
{
    if(factory == null)
        factory = new DataCacheFactory();

    if(string.IsNullOrWhiteSpace(cacheName))
        cacheName = ConfigurationManager.AppSettings["APP_FABRIC_CACHE_NAME"];

    cache = factory.GetCache(cacheName);
    return cache;
}

In the above example, cache and factory are private static versions of their respective types, inside a static class called Cache.

Hope this helps

Answer 2

We use a lock around our factory creation and have never had this issue. We are also using AppFab 1.1 for our cluster. I would recommend you do something like this:

lock (cacheLock)
{
    if(cacheFactory == null)
    {
        DataCacheFactoryConfiguration config = new DataCacheFactoryConfiguration(...); 
        cacheFactory = new DataCacheFactory(config);
    }
    return cacheFactory;
}