Any way to cache an object using an outside process (i.e. not through a page's code-behind)?

Question

I've got my first batch of really massive data that I need for my ASP.NET web application. Up until now (for two years in fact!) nothing has come up that was so costly that I couldn't just do it on Page_Load or store it in ViewState or Session. But now, I need a list of objects that remains fairly static (gets updated once a day at most) and contains about 5000 objects, and building it every time on Page_Load causes the load time to take an obvious hit.

Some Googling regarding caching shows some very elementary examples using stuff like Cache["myStuff"] = myObject; and that's all well and good. But I'd like to know if there is some way I can set my Web Application to run an update at a specific time, regardless of whether or not the application is being accessed or in use.

For example, I want this process to update the list to run once a day at like 4:00am when no one is using anything. So let's say on day 1 in the afternoon, the Page_Load checks to grab the cached object, but it comes back null, so in that case, bite the bullet and request/build the list for the rest of day 1, store it in cache. Throughout the rest of day 1, no one has to build the object because it's in cache and it's still valid.

Day 1 ends, and it's now 4:00am on day 2. No one is using any pages. I want the object that was stored in cache from day 1 to be deleted and re-built, so that when people get to the office in the morning, the first person who accesses the page doesn't need to build the list for today, it's already done.

The only idea I have would be to look into creating a console application that runs on a Windows scheduled task or something. But even then, I'm not sure how I would be able to interact with the ASP.NET web application from an external application like that.

Hopefully I'm making sense. Is what I'd like to accomplish even possible?

Answer 1

If you use a distributed cache like MemCached, you could create a Windows Service or Console application for populating the cache. Since both applications (The service and the web app) will use the same cache, it will solve your problem.

Another solution is to have a page just for that (i.e. /startcache.aspx) that will do the cachin on page_load. And you can schedule a request to that page. You could also hide the page from outside the web-app using some kind of firewall (so nobody accesses that page!). Finally, the good part of this solution is that you can reuse your current code, as it will work in the same project.

Hope it helps.

Answer 2

Redis is another option for an out of proc 3rd party cache/key value store.

Answer 3

To reduce load on the server and backend and serving all daily data (as I understand from the question)

Why not create a data file (CSV,XML,JSON) and offer that as a data-point.

From your question it seems that the client(application) is capable of running their own queries against that dataset.

Creating a file like that can be done on a fixed time set, saved and as such just one time. No need to include thirt-party solutions (besides their respectable qualities)