If you store a vector of entities and then just store their iterators to access them: yes, a reallocation might invalidate all your data.
The suggested way is to store a vector of pointers (if you need memory collection capabilities you might want to go for a vector of smart pointers). This way you will be sure that the pointers are valid (assuming nothing else touched the objects) at every insertion/deletion regardless of the reallocation of the container's space.
From the question isn't clear but a word of advice if you're just storing objects in your containers instead of pointers: when inserting elements into a container like with
std::vector<T>::push_back()
you're storing a copy of the object. This is usually undesirable since brings additional copy overhead and might create problems if things aren't properly set up. See "shallow copies" and "deep copies" to learn more about this problem.