ASP.net MVC static class and background worker

Question

I have an MVC 5 ASP.net app that is working perfectly. A colleague of mine has made a project that reads in data from files and churns some of it to output some calculations for me to display in the web application. I call his methods and I get the data back and display on screen.

As part of his project he has implemented a kind of caching system, it's not using any of the caching tools in. NET, it's simply holding some of the files required for calculating the numbers in memory as these take a long time to load in and are definitely the bottle neck.

So I need to do two things

1) Have one instance of the class within the project so it makes use of the in memory files.

2) Setup a worker process that runs at 7am everyday to load those files into memo.

For number one I have instantiated my colleague's class as a static class and this seems to work some of the time but it's unreliable for some reason I'm guessing due to app pool timeouts, I'm also worried about the effectiveness of this and whether or not I will have issues with multiple users sending concurrent requests. Is there a better way to achieve this? I've looked at various .NET caching libraries so I could get my colleague to implement a proper cache but for now I'm assuming I can't ask him. I'm a little confused as to why in MVC every time a user sends a new request a new controller is instantiated hence instantiating my colleague's class again and losing the in-memory files?

For number 2 I've read about webbackgrounder a library for ASP.net that can run background tasks. All I need to do is whilst the app is running on IIS then run a method everyday at 7am.

Update:

I can't save the output because I need to calculate these vlaues on the fly, there's about 10 different inputs into the calculation that can change to be any number of different values.

I can indeed put the input files into a database but that doesn't solve my problem, it wouldn't be any faster than loading them in from files, which takes too long as it is, I want to hold these items in memory hence the cache approach.

Answer 1

Deploy your colleague's library as a cached microservice

Assuming you can't change your friend's library at all, you don't want to use a database, and you want to maintain a reasonable level of process isolation, here's the right way to do it:

1. Write a bit of glue to expose your colleague's library as a RESTful microservice. If you've never done this before, start here. It shouldn't take more than 100 lines of code to wrap one function call.

2. Use the uniform interface to manage cache and its duration.

   • Set up ASP.NET output caching. You can do it programmatically or use the OutputCacheAttribute, whichever is easier for you. Set the cache to an absolute expiration of 7:01 am.

   • Set IIS to enable output caching per these instructions.

3. Set the application pool for this service to recycle at 6:59 am.

4. Modify your web site to utilize the new service.

5. If your colleague's code takes a long time to run, the microservice may not be responsive when handling its very first request. To deal with this, you need to "warm up" the service. Set up warmup using IIS auto-start; details can be found here.

With this approach, your colleague's code is completely isolated from yours, and the caching logic simply leverages standard features in IIS. With the caching and recycle scheme set up properly, you should have seamless transitions when the program is recycled, since IIS will continue to serve requests in flight from the old cache while building up the new cache. We set the cache expiration to exactly 7:01 am (as opposed to letting the app pool recycle take care of it) just in case there is some other network node looking at your cache headers, e.g. a proxy server that might cache the response if it isn't encrypted.

Answer 2

ASP.NET MVC (on IIS) cannot be used reliably for any sort of background work. The problem is that IIS processes are recycled regularly, and you can never be certain that a background task fired off will finish correctly (this is due to various reasons inherent to the process and the environment in which it operates). There's no such "fire and forget" functionality that can work well with ASP.NET MVC (or anything else running on IIS).

What I've done in the past for very simple timed tasks is to simply create a little console application that taps into the main project and place it on the server to run via Windows Task Scheduler.

For (slightly) more complex scenarios, writing a "command" to a database, then having a server-based (console, typically) application run that polls for new tasks to execute on a regular basis.

For robust management, look into a queue system like RabbitMQ, which can effectively decouple the background process from the MVC process.

(at least part of the reason for the recycling of IIS processes/AppPools and the fact that a controller is instantiated for each new request is because it helps ensure that if one request "goes bad", it doesn't bring down the entire system)

Here is the first reply on a search for "ASP.NET fire and forget": https://stackoverflow.com/questions/18502745/fire-and-forget-async-method-in-asp-net-mvc

Answer 3

ASP.NET comes with a variety of caching mechanisms. The best fit for this situation is probably HttpApplicationCache. It has a specific class for monitoring flat files and keeping the cache up to date.

Basically you would set up a static property that wraps the caching logic, and use the static property whenever you want to access the file.

static public string CachedFileContents
{
    get
    {
        ObjectCache cache = MemoryCache.Default;
        string fileContents = cache["filecontents"] as string;

        if (fileContents == null)
        {
            CacheItemPolicy policy = new CacheItemPolicy();
            policy.AbsoluteExpiration = DateTimeOffset.Now.AddHours(24);  //Set to taste

            List<string> filePaths = new List<string>();
            string cachedFilePath = Server.MapPath("~") + "\\cacheText.txt";
            filePaths.Add(cachedFilePath);
            policy.ChangeMonitors.Add(new HostFileChangeMonitor(filePaths));

            // Fetch the file contents.
            fileContents = File.ReadAllText(cachedFilePath);

            cache.Set("filecontents", fileContents, policy);    
        }
        return fileContents;
    }
}