OK. Again, I have to answer my own question. I'll try to describe the process of fixing it, too. First thing I thought of is to google a solution, but with no success. Then I tried Revo Uninstaller, again to a deadend. Then I noticed that when uninstall AnkhSVN, a devenv.exe process showed up, I suddenly realize that this was OUR HITLER. So after killing the process with Task Manager, the uninstall proceeded and succeeded.
Bravo.
But this indeed took sometime to do. I spent about 2 to 3 hours on this.
Also this method fixed the issue that installing of AnkhSVN 2.4 (the latest) hangs up when registering with Visual Studio 2010. Just kill devenv.exe and everything works fine.