I fixed it, but I'm still not sure what the issue was. As root, I did
gem update --system
gem install --no-user-install mysql2
For some reason the permissions ended up wrong so I had to do
chmod -R o+rX /usr/local/share/gems1.9
Then everything worked again. Also require 'rubygems'
ended up being unnecessary in the CGI script.