You probably need to install libsndfile as a universal library - by default it doesn't compile a 32-bit slice, which is required for compatibility with the system ruby.
brew uninstall libsndfile
brew install libsndfile --universal
sudo gem install ruby-audio