The factories will be instantiated when you request them via the service manager's get
method. This method calls the create method, which in turn calls the createFromFactory method in the case of requesting a factory. As per the highlighted line on the last link, you can see that it does indeed instantiate the factory at the time it is requested. What you can also see is that it stores the factory. This is because services fetched from the service manager are shared by default. Shared means that the same instance will be returned on subsequent requests. You can think of it like a cache. You can disable this if you so desire, such that an instance of a given service will be created on every request.
In a configuration file:
// Some configuration omitted
'service_manager' => array(
/* ... */
'shared' => array(
'MyFactory' => false,
)
);
By accessing the service manager object:
$service_manager->setShared('MyFactory', false);
Some time ago I wrote an article about the service manager, which includes a section about the concept of shared services (near the bottom), which may be useful for you.