My real question is what happens when the cache grows so large
This is where you would implement a sort of Garbage Collection. Items could be removed from the cache following a Cache Algorithm.
So for example following Least Recently Used you would record how many times a specific object was used when it was last accessed and remove those from the cache that were not used recently.
Edit
Soundcloud use an object store and a very interesting read is this article on how they built their webapp.