“Shop by” / “layered navigation” configurable products not filtering correctly

Question

For example. You have t-shirts with size and color. You have a configurable product t-shirt with a number of variations on these attributes as simple products.

When filtering using "shop by" and you have say a "red small" and a "green medium" shirt but no "red medium", if try filter by "red" and "medium" the configurable product will be displayed even though no such combination exists within the configurable product.

Is there a option or fix to get the correct behavior?

Using 1.8.1.0 Community Edition and SCP (Simple Configurable Products).