Removing it from the project structure just modifies the solution file (.sln). After removing a project it's not anymore listed in there and the reference is gone.
So removing a project from the solution just removes the reference to the project. If you'd like to remove it entirely, delete it manually from its old location on the disk, as well.