The issue seems to have been that I was trying to debug by overwriting my dlls in the packages directory. This doesn't work because VS copies your package directory to something like:
%TEMP%\VisualStudioTestExplorerExtensions\My.TestAdapter.1.0.0.23
essentially caching it with the key being the version number. If you copy the file into the project packages directory VS won't copy it because the folder number didn't change. So, to update your extension you have to keep publishing NuGet packages.
At least that's how I understand it.
Edit: I've also noticed that when you upgrade your test adapter through NuGet, it won't start using the new version until you restart VS, or kill vstest.*