I think it always comes down to personal reference.
Usually, i use formatters with types such as string and datetim where i just want them to be printed on the UI in a pretty way.
With ValueConverters i usually take a totally different approach. A common ValueCnverter i often see if BooleanToVisibillityConverter where you have some bool which decides whether a control will be displayed or not. generally when you want have some property which decides whether some actions will execute or not but you cannot use that property in a straight forward way with Binding.
I hope that made sense :)