I solved my problem by removing all my homebrew packages and installing them with macports. Especially the gem that needed glib2 (causing the error) was rsvg2, so I solved the problem by reinstalling:
sudo port install ImageMagick
sudo port install librsvg
And then I could install my gems