Integrating persistent caching into Apache Httpclient with EhCache

Question

Today I wanted to integrate a persistent cache into apache's httpclient library. A quick look into the documentation for Http-Caching tells you:

The current release includes support for storing cache entries using EhCache and memcached implementations, which allow for spilling cache entries to disk or storing them in an external process.

Fair enough, a quick google-search later I had exactly no idea what to do and how to do that ;)

But there was one more hint:

If none of those options are suitable for your application, it is possible to provide your own storage backend by implementing the HttpCacheStorage interface and then supplying that to caching HttpClient at construction time.

So at least we know that the HttpCacheStorage needs to be initialized using some class that implements HttpCacheStorage.

The question in a one liner:

• How do you integrate caching into any http-request?
• How do you initialiase an EhCache?
• How do your integrate an EhCache into the http-caching thingy.

Answer 1

Beaware: Ehcache's LOCALRESTARTABLE strategy requires Enterprise license. Well this, would work perfectly, if I had a commercial license of EhCache, which I don't. Had I known that, I wouldn't have gone through the trouble.

Digging further using google and the documentation of Httpclient, the first thing you need to do is to add httpclient-cache-<version>.jar to your classpath.

After that, use the org.apache.http.impl.client.cache.CachingHttpClients instead of the org.apache.http.impl.client.HttpClients

Now you can build the configuration from the documentation.

 CloseableHttpClient httpclient = CachingHttpClients.custom()
   .setCacheConfig(cacheConfig)
   .setHttpCacheStorage(ehcacheHttpCacheStorage)
   .build();

Which leaves you trying to figure out how to configure a cacheconfig, which is fairly easy.

//build the cacheconfig. this part belongs to httpclient
CacheConfig cacheConfig = CacheConfig
     .custom()
     .setMaxCacheEntries(1000)
     .setMaxObjectSize(8192)
     .build();

The next thing to worry about is the EhCacheHttpCacheStorage, that is fairly problematic, since EhcacheHttpCacheStorage constructor takes an EhCache, but there is no direct way to initialize one. What you need to do is construct a CacheManager, add a cache to it, get it out of the CacheManager, and finally wrap it into EhcacheHttpCacheStorage

Step 1: Create the manager:

mgr = CacheManager.create();

Step 2: Create the cache:

Cache testCache = new Cache(new CacheConfiguration("http", 100000)
.memoryStoreEvictionPolicy(MemoryStoreEvictionPolicy.LFU)
.eternal(false)
.timeToLiveSeconds(60)
.timeToIdleSeconds(30)
.diskExpiryThreadIntervalSeconds(0)
//Persist everything and keep it on restart
.persistence(new PersistenceConfiguration().strategy(Strategy.LOCALRESTARTABLE)));

Step 3: add it to the manager:

mgr.addCache(testCache);

Step 4: get the cache out of the manager:

Cache cache = mgr.getCache("http");

Step 5: wrap it into EhcacheHttpCacheStorage which implements HttpCacheStorage

EhcacheHttpCacheStorage ehcacheHttpCacheStorage = new EhcacheHttpCacheStorage(cache);

And to make sure you can try this at home, here is everything in the right order, with the necessary imports

import net.sf.ehcache.Cache;
import net.sf.ehcache.CacheManager;
import net.sf.ehcache.config.CacheConfiguration;
import net.sf.ehcache.config.Configuration;
import net.sf.ehcache.config.DiskStoreConfiguration;
import net.sf.ehcache.config.PersistenceConfiguration;
import net.sf.ehcache.config.PersistenceConfiguration.Strategy;
import net.sf.ehcache.store.MemoryStoreEvictionPolicy;

import org.apache.http.impl.client.cache.CacheConfig;
import org.apache.http.impl.client.cache.CachingHttpClients;
import org.apache.http.impl.client.cache.ehcache.EhcacheHttpCacheStorage;


//build the cacheconfig. this part belongs to httpclient
CacheConfig cacheConfig = CacheConfig
     .custom()
     .setMaxCacheEntries(1000)
     .setMaxObjectSize(8192)
     .build();
//this part creates the ehcache 
//Create a singleton CacheManager using defaults
mgr = CacheManager.create();

//Create a Cache named http specifying its configuration.
Cache testCache = new Cache(new CacheConfiguration("http", 100000)
.memoryStoreEvictionPolicy(MemoryStoreEvictionPolicy.LFU)
.eternal(false)
.timeToLiveSeconds(60)
.timeToIdleSeconds(30)
.diskExpiryThreadIntervalSeconds(0)
//Persist everything and keep it on restart
.persistence(new PersistenceConfiguration().strategy(Strategy.LOCALRESTARTABLE)));
mgr.addCache(testCache);
//get the cache back out the manager
Cache cache = mgr.getCache("http");

// and we are back to http-client
//Wrap it around EhcacheHttpCacheStorage which implements HttpCacheStorage
EhcacheHttpCacheStorage ehcacheHttpCacheStorage = new EhcacheHttpCacheStorage(cache);
//And Configure the httpclient
CloseableHttpClient httpclient = CachingHttpClients.custom()
   .setCacheConfig(cacheConfig)
   .setHttpCacheStorage(ehcacheHttpCacheStorage)
   .build();