So, the solution turned out to be a bit hacky, but it seems to work:
I do need it to be an association, rather than methods, but it turned out I could scope my associations.
Agency.rb
has_many :bridge_roles, -> {where(resource_type: 'Agency')}, class_name: 'Role', foreign_key: :resource_id
has_many :agents, -> {where('roles.name=?', 'agent')}, class_name: 'User', through: :bridge_roles, source: :users
The key being that I created the bridge explicitly to make the association aware of the rolify roles. Through that, I can access the set of Users I care about.