Okay, the issue ended up being, of course, autoload paths.
Since the models have dependices that are hard to satisfy by requiring them one at a time, I had the models.rb file that looked like:
autoload :User, 'my_gem/models/user'
autoload :Role, 'my_gem_models/role'
Which meant it was manually autoloaded and not part of the Rails autoload_path paradigm. Simply pulling in ActiveSupport::Dependencies and appending the models directory to the autoload paths worked.
ActiveSupport::Dependencies.autoload_paths += Dir.glob "#{File.expand_path("../my_gem/models/", __FILE__)}/**/"