Explainable AI and unsupervised algorithms

Question

There are several packages that allow explaining ML algorithms (Lime, Shap and so on). However, it is not clear how we can explain unsupervised algorithms for example, if we use PCA for dimensionality reductions first and random forest after, how it is possible to explain the whole process?

Answer 1

PCA isn't a great example, as it's inherently explainable, in the same way that linear regression is inherently explainable.

In supervised learning, at a high level, the Shapley value for each feature tells you the extent to which including that feature in an ML model allows you to make better predictions about the target variable than if you did not have access to that feature. One way of implementing this is to calculate the difference in test cross entropy between a model which includes that feature and one which does not (and it's much more complicated than that in reality, as there are many models which do not include that feature, you have to average over all coalitions of other features, but that's another story).

So in short, you need some sort of cost function, which including features tends to improve.

Now let's turn our example to the simplest of all clustering algorithms, KMeans. In KMeans, the most sensible cost function is probably to split your data into train and test, fit on a training set, and then take the sum over your test set, of each point's euclidean distance from its nearest centroid.

Now, what happens if you remove a feature? Well, in general, post fitting, the sum described above will run over fewer features, so the cost will be lower. This in turn will mean that every feature has a negative Shapley value...not good

More generally, in unsupervised learning, the cost function tends to be related to the probability of seeing a datapoint, according to the model, i.e. the likelihood. The key however, is that this is not just the likelihood of the target, it's the entire, multidimensional likelihood of all features (indeed, there is no target in unsupervised learning).

What you would need, in order to calculate Shapley values for unsupervised learning problems, is some way to describe "how good the clustering is" with one number. The "goodness of clustering" would need to be comparable when you changed the dimensionality of the problem...and standard cost functions used in inferential clustering (which KMeans is kind of a simplified form), do not satisfy this.

So I've only answered a subset of your problem, I've said that Shapley values don't really make sense for inferential clustering problems, rather than "explainability doesn't make sense for unsupervised algorithms". I don't know enough about all unsupervised algorithms to claim Shapley values don't make sense for any unsupervised algorithm, but it certainly seems to me to be the case, that you'll need an algorithm which lends itself to determining for any coalition of features included in the clustering, "how good the clustering is", in a way which somehow "objectively measures whether similar instances are grouped", rather than which calculates full data likelihood in a way which is inversely proportional to the number of features in the model. This seems to go against the idea of unsupervised learning, namely that there's no objective truth.