How regularization helps to get rid of outliers?

Question

I have heard regularization helps to get rid of outliers, how so? 'My intuition is, regularization shrinks parameter or even make it zero, and hence large value will have less effect on overall result'. Could you shed some more light on it?

Answer 1

You don't get rid of the actual outliers (no data reduction). But statistical methods can be robust against outliers.

Robust statistics are statistics with good performance for data drawn from a wide range of probability distributions, especially for distributions that are not normal. Robust statistical methods have been developed for many common problems, such as estimating location, scale, and regression parameters. One motivation is to produce statistical methods that are not unduly affected by outliers.

Source: wikipedia.

So, L-1 regularization is robust against outliers as it uses the absolute value between the estimated outlier and the penalization term.

Whereas, L2-regularization is not robust against outliers as the squared terms blow up the differences between estimation and penalization.