Found the answer, finally.
git rm --cached path/to/submodule
(no /
or *
at the end)
I haven't added these repos as submodules myself, but clearly they have been cached as such somehow.
After clearing the cache you still need to add the files to the outer repo, as Peter suggested.