Instead of rendering a relative timestamp, render the actual, non-relative timestamp in your template, and then use something like Moment.js to convert it to a relative timestamp client-side.
This also has the advantage that the timestamp will update itself if the user has the page open for a long enough period that the relative timestamp would change.
You should also consider not caching anything until you actually understand what on your site is slow. Caching has a lot of gotchas like this, and prematurely optimization isn’t always worth it.