Are “Gradient Boosting Machines (GBM)” and GBDT exactly the same thing?

Question

In the category of Gradient Boosting, I find some terms confusing.

I'm aware that XGBoost includes some optimization in comparison to conventional Gradient Boosting.

• But are Gradient Boosting Machines (GBM) and GBDT the same thing? Are they just different names?

• Apart from GBM/GBDT and XGBoost, are there any other models fall into the category of Gradient Boosting?

Answer 1

Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the optimization technique called gradient descent (Boosting Algorithms as Gradient Descent.

Given this, you can boost any kind of model that you want (as far as I know). Moreover in the scikit learn library, gradient boosting, its under the ensemble folder. You can boost any kind of model (linear, svm) its just that decision trees achieve normally great results with this kind of ensemble. Same way that you can do a bagging with any kind of estimator, but if you do it with a decision tree and add a couple technichalities more, you can call it Random Forest.

From scikit learn documentation: GB builds an additive model in a forward stage-wise fashion; it allows for the optimization of arbitrary differentiable loss functions. In each stage a regression tree is fit on the negative gradient of the given loss function.

But are Gradient Boosting Machines (GBM) and GBDT the same thing? Are they just different names?

Gradient boosting machines are a kind of ensemble and Gradient Boosting Decision Tree the particular case when a tree is used as the estimator.

Apart from GBM/GBDT and XGBoost, are there any other models fall into the category of Gradient Boosting?

You can use any model that you like, but decision trees are experimentally the best.

"Boosting has been shown to improve the predictive performance of unstable learners such as decision trees, but not of stable learners like support vector machines (SVM)." Kai Ming TingLian Zhu, Springer