So, I can't explain why but I'm unable to cache the documentset.
Instead I was forced to cache the documentset->to('array'). Then, while reading from cache create a new DocumentSet object by passing the models schema and the cached array, returning that.
Seems to be working fairly well, if not as elegant as I had hoped.