I think what you are looking for is support for mapping into a multimap, which is a map with one key and the value is a list of values all associated to the same key.
This is not supported by Hibernate, this is a thread in their forum with an attempt to implement it.
An alternative is to still map to a Set at the property level, and then provide a getter that does the grouping on the fly. This would be transparent for consumers and only become a problem for very large datasets:
@OneToMany(fetch = FetchType.EAGER)
@JoinColumn(name = "X", referencedColumnName = "X", insertable = false, updatable = false)
@NotFound(action = NotFoundAction.IGNORE)
@Fetch(FetchMode.SELECT)
private Set<MyType> siblings;
public Map<String,List<MyType>> getSiblings() {
... do the grouping manually in Java ...
}