I had trouble with this as well on OS 10.9. I tried many things and this is what worked for me (after installing Xcode 5):
brew install libtool --universal
brew link libtool
Quick research just found the following with other users who also had success:
Stack Overflow: rmagick-error-after-installing-os-x-mavericks
TheKUnit Blog: RMagick errors running rake tasks os x maverick
Edit: Another possibility is that you have a different version of RMagick than the gem is looking for (possibly from the reinstall). Your log shows your gem is looking for /usr/local/lib/libMagickCore.5.dylib which I am guessing it cannot find. Do
ls /usr/local/lib/
and see if you can find that file. If not, a
gem pristine rmagick
may help if you haven't already tried it. In my case I have Core 6 in that folder and not 5.