What you are arguing over here is a micro-optimisation. Unless your value converter is quite massive, the memory it consumes will be quite small.
Do not optimize prematurely. The 'standard' pattern for value converters is to create a new instance within each binding, i.e. avoiding singletons. If you do find yourself encountering memory issues, optimize at that point. Use profiling tools to determine where the issue is and target them specifically. I am willing to bet that your value converter will not be teh root cause of a memory issue.