The --enabled-shared flag fixed it in the end. Here was the whole line I used to re-install ruby:
CC=/usr/local/opt/apple-gcc42/bin/gcc-4.2 CONFIGURE_OPTS="-C --enable-shared --with-readline-dir=$(brew --prefix readline) --with-opt-dir=/usr/local/opt/openssl:/usr/local/opt/readline:/usr/local/opt/libyaml:/usr/local/opt/gdbm:/usr/local/opt/libffi --without-tcl --without-tk CC=/usr/local/opt/apple-gcc42/bin/gcc-4.2" rbenv install 1.9.3-p484