It's not a good idea to replace all members with pointers, as it would add an extra layer of dereferencing. A direct member is actually allocated within the containing object, so accessing it is only requires a fixed offset from the containing object's location.
If you hide everything behind pointers, you would need to access the pointer member, then dereference the pointer to access the data (which will be somewhere else in memory). It's a trivial overhead on an individual basis, but it would soon add up if the principle was expanded across an entire program.
Another issue to bear in mind is maintainability. If you store everything via pointers then you need to manually including appropriate construction and copying. That can lead to various problems on a large project, as it increases the likelihood of programmer error. By contrast, the compiler does a lot of that automatically if you're using direct members.
An alternative that's worth looking-up is the PIMPL design pattern (Private Implementation, or Pointer to Implementation). It's essentially a way of hiding the private details of a class inside its source file, with a view to reducing header dependencies.