VS Team is aware of the problem :
Allowing users to disable the automatic provider switching is something we're considering. When we added Git support, we thought about Git-TF users and optimized for using the developers wanting to use the Git tools. The side effect is that you can't easily work with the TFS team project.