To answer your question, yes the entire sheet is loaded into memory.
However, only once.
Each CSS displays only a small portion of the sprite.
In theory: As you move from page to page that one large image is already in the cache and not requested from the server again. Making the next page load that much faster.
There is much more on caching v/s the number of requests v/s the combined sizes of images that would go into a more complete answer.