In addition to what Phil said, you can only convert an atomic subsystem into a referenced model. An atomic subsystem means that the subsystem executes as a whole rather than the hierarchy of the model being flattened during compilation, as with virtual subsystems. For more details, see the documentation. You can have library components within referenced models, but there are a number of limitations, see Simulink Model Referencing Requirements and Model Referencing Limitations.
Despite the constraints, model referencing is generally advised as the way to approach large-scale modelling and it should give you performance improvements, as the referenced models run in accelerated mode (unless otherwise configured), and are rebuilt only when structural changes are made to the model. See Componentization Guidelines in the documentation for more details.