You added AppFabric Session State Provider in web.config. All sessions will be persisted in the cluster : that's why you have multiple cache items.
Also, you will need to add an extra error management in your system, as said here :
Cache hosts can hold only a certain amount of cache operations in memory. It is possible, depending on system load, that some cache clients may not receive notifications before they are truncated in the cache host queues. Cache clients may also miss notifications when data is lost due to a cache server failure while the rest of the cluster remains running. In these cases, your cache client can discover that it has missed some cache notifications by using a failure notification. Your application can add a callback to receive failure notifications by using the AddFailureNotificationCallback method. For more information, see Add a Failure Notification Callback
EDIT :
CacheItemRemovedCallback
provides the key that is removed from the cache. Check if it's the always the same.I also recommand to not limit the distributed cached to a single item : if someone else add something in the cache your system will fails.