A simpler approach would be:
return scores.stream()
.mapToDouble(MatchScore::getScore)
.sorted()
.skip(1)
.limit(scores.size() - 2)
.average().getAsDouble();
Note: that works because elements in a set are unique - with a list there could be more than one element equal to the min or max score.
Performance wise*, the skip/limit is significantly faster on a small set of 10 elements (the Mean column shows the average time taken by a sample call, in nanoseconds):
Benchmark Mode Samples Mean Mean error Units
c.a.p.SO22923505.maxMin avgt 5 6996.190 284.287 ns/op
c.a.p.SO22923505.skipLimit avgt 5 479.935 4.547 ns/op
*using jmh - and here is the source code for the tests.