I actually found my answer so I will post it here in case anyone else runs into the same issue!
You have to add your old url to the list of legacy-urls like so:
git config tfs-remote.default.legacy-urls http://your.old.url/
I hope that helps for anyone that has or will be pulling their hair out in the future!