Tracking last image request using nginx and redis

Question

I use nginx to serve static files. For each file I would like to save the timestamp when this file was retreived by a browser request last. Each file has a "unique ID" consisting of 1. servername, 2. path and 3. filename. The filename itself is not unique.

I would like to use a key value store like redis to store this information and a cron job afterwards which pushes this timestamp information to a mySQL database. I need to put redis in between since the system needs to handle a lot of concurrent requests.

Ultimate goal is to automatically delete all files which have not been requested in the last 6 months or so.

How would you configure/set up nginx/redis to make this happen?

Best Kilian

Answer 1

There are two components to this: 1) how to structure the data in Redis and 2) How to configure Nginx to update it.

Unless you have an external requirement for MySQL I don't see a reason to use it in this chain.

First: Redis Structure I am assuming you will be running your cleanup job on a frequent basis such as daily. If you are doing it based in a fixed time such as "every month" you might structure your data differently.

I think your best structure may be to use a sorted set. The key name would be "SERVER:PATH", the membername would be the file name, and the score would be a UNIX timestamp.

With this setup you can pull members without needing to know their filename and do so based on their score. This would allow you to pull "any member with SCORE <= TIMESTAMP" where timestamp is also a UNIX timestamp in your example for "six months ago" using zrangebyscore or zrevrangebyscore.

The job you run to clean unused files would use these commands to pull the list. When they are removed you can use the zremrange command to clean them from Redis. If your writes are frequent enough you could run a read-only slave to do the clean-up pulls from.

If you are expecting to have a large amount of such entries you may see a long period of time lead to a larger database. If so you'd likely need to reduce the amount of time you keep the file cache from six months to something more manageable. Six months is a long time to keep a cache.

Second: Configuring Nginx to update the sorted sets This depends very heavily on how comfortable you are with mucking about with nginx modules. It doesn't do it natively, but you could use the lua-resty-redis module to add the ability directly into Nginx. I've used it for similar tasks.

Hopefully this will get you started. The key portion is really the data structure in Redis, as the rest is simply configuring and testing the Nginx portion in and for your setup.