Compare Coefficients of Different Regression Models

Question

in my project, I am using asuite of shallow and deep learning models in order to see which has the best performance on my data. However, in the pool of shallow machine learning models, I want to be able to compare the coefficients of each regression model between each other. Example: I have

• Lasso
• Ridge
• ElasticNet
• AdaBoostRegressor
• GradientBoostRegressor

I am aware that I could get the coefficients of Lasso, Ridge, and ElasticNet from model.coef_ and model.intercept_ from sklearn. However, AdaBoostRegressor does not have this, but rather, it has weights assigned to each of the estimators. Similarly, GradientBoostingRegressor of sklearn does not even have weights at all. How am I to compare the coefficients between each of the regressive models I am using??

Answer 1

Weights across different types of models are not always comparable, so I think that it would make more sense to do this kind of comparison not across different types of model but within a single type of model varying:

• the hyper-parameters (if any),
• the set of instances (e.g. selecting different subsets randomly),
• the set of features.

I'd recommend in particular varying the instances: if the weight of a feature tends to change a lot depending on the training set it's a sign of overfitting.

If the goal is to compare the importance of some specific features for different models, it's probably more reliable to directly evaluate how the model performs with/without the feature(s).

Answer 2

It's not actually possible to directly compare model coefficients. What you might do that would make more sense is to compare similar metrics.

A good start would be to learn about explainability metrics that are comparable across models : LIME, SHAP... etc. (see here : https://christophm.github.io/interpretable-ml-book/) to see how models reacts on different features.