When should we start using stacking of models?

Question

I am solving a Kaggle contest and my single model has reached score of 0.121, I'd like to know when to start using ensembling/stacking to improve the score.

I used lasso and xgboost and there obviously must be variance associated with those two algorithms. So stacking should theoretically give me better output than my individual algorithms.

But how to idenfity if stacking is worth it and that we've reached dead end to accuracy of a particular model?

Answer 1

Stacking is going to help most when individual models capture unique characteristics of the data. It is often the case that different architectures perform similarly, if somewhat differently, on the same data. In those cases, ensembling/stacking will only offer slight incremental benefits. In the limit, of you only care about prediction, you can wire up as many different approaches as you can think of. However if interpretability is key, each additional component model will further complicate things.

Your specific question of when to know if it’s worth it or if you’ve reached the limit can be treated like anything else - is your incremental r-square/error/classification accuracy significantly better versus a simpler approach?