Yes, you can remove the tmp directory.
If you don't have a new-enough version of libxml2 or libxslt nokogiri downloads and builds it for you to link against. Unfortunately RubyGems doesn't ask the nokogiri to clean up after itself at this time, so these build artifacts are left around.