I believe Thinktecture Identity Server has an endpoint for this at (baseurl + "/core/accessTokenValidation?token=" + access_token).
example:
GET request to: http://localhost:3333/core/accessTokenValidation?token=aEdhoi23hlv2khdf2lkhfv4pv....
If the access_token is valid, it should return a 200 response, otherwise it'll return a JSON error message {"error":"invalid_token"} and a 4XX response.
You should wire up a call to this service to validate tokens, then cache the response.
Take a look at the source code: https://github.com/IdentityServer/IdentityServer3/blob/master/source/Core/Endpoints/Connect/AccessTokenValidationController.cs