I would say the existing approach is more readable than your bitwise approach. It says exactly what you mean: if the set contains X, or the set contains Y... Keep it as it is. It's already clean.
If the set becomes larger, you could use:
EnumSet<Foo> valid = EnumSet.of(Foo.X, Foo.Y, Foo.A, Foo.B);
valid.retainAll(enumSet);
if (valid.isEmpty()) {
...
}
But I'd only keep that for larger cases. For two or three options I'd use the longhand form.