This turned out to be a bug in RVM with the version of RubyGems installed by RVM. You need to make sure that the RubyGems that is installed is 2.x. This wasn't the case for me, RVM had either not installed RubyGems at all, or, not installed the correct version; we ultimately got some help from the RVM channel on IRC.
There's a bit more detail on twitter off the back of the conversation we had.