Don't understand the need for $cacheFactory if Factories are singletons

Question

CacheFactory is a factory:

The purpose of the Factory Method pattern is to define an interface for creating an object but deferring instantiation of the object to subclasses.

which creates new instances each time an object of its type is created, unlike a Singleton:

The intent of the Singleton pattern is to ensure that only one instance of a class is loaded, and that the instance provides a global point of access.

In Angular, the CacheFactory offers the following benefits over other solutions:

Caches can be injected into a filter, unlike controller scopes
Caches can be enumerated
Caches can be easily destroyed or redefined
Caches are not tied to digest data binding, unlike controller scopes
Caches are not tied to browser permissions, unlike cookies or localStorage/sessionStorage

I'm not sure what the benefit is if my data remains in the factory

CacheFactory offers the following features:

'should create cache with defined capacity'
'should blow away the entire cache via removeAll and start evicting when full'
'should correctly refresh and evict items if operations are chained'

But CacheFactory also has the following unexpected behaviors:

'should purge the next entry if the stalest one was removed'
'should only decrement size when an element is actually removed via remove'

References