UserCollectionType
is for when you want to rely on Hibernate's collection handling when handling persistence, but want to expose a different class to clients of your persistent objects. For example, if you have an Auction and a one-to-many of Bids, you might want Hibernate to treat this a set when persisting, but you might want getBids() to return a FastSet (as in your link).
CustomCollectionType
is for when you want to persist a collection in a completely different way with different semantics using a new implementation of PersistentCollection. I'm not sure when you'd really want to do this except to change some Hibernate framework behavior.