Linux and installed as regular user ? Than all you need to do is:
rm -Rf ./.rvm
rm -Rf ./.gem
rm -f ./.gemrc
You can also check ~/.bashrc and ~/.bash_profile files for lines like this:
PATH=$PATH:$HOME/.rvm/bin # Add RVM to PATH for scripting
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*
This should be enough. If you've installed it in some custom way then try this:
$ rvm list # to see what version you have installed
$ rvm gemdir # to see where are gemfiles located
There is something like rvm remove
as well but, if you want to just remove all, than it's not very interesting I think.