Gradient Boosting Intuition
22 min
Scikit-learn's estimators-api module trained a single decision tree.
Gradient boosting (what XGBoost implements) builds an ENSEMBLE of
many small trees, but not independently — each new tree is trained
specifically to predict the RESIDUAL ERRORS of everything built so far,
and its output gets added on top to correct those errors. Round by
round, the combined prediction gets closer to the true values.
from xgboost import XGBRegressor
def train_and_score(X_train, y_train, X_test, y_test, n_estimators):
model = XGBRegressor(n_estimators=n_estimators, max_depth=2, random_state=0)
model.fit(X_train, y_train)
preds = model.predict(X_test)
mse = sum((float(p) - y) ** 2 for p, y in zip(preds, y_test)) / len(y_test)
return round(mse, 4)
That's roughly a 15,000x reduction in error just from adding more boosting rounds -- a dramatic, concrete illustration of what 'boosting' actually buys you: each of those 50 tiny trees only needs to fix what's STILL wrong after the previous 49, and those small corrections compound fast.
Write `train_and_score(X_train, y_train, X_test, y_test, n_estimators)`: train an `XGBRegressor(n_estimators=n_estimators, max_depth=2, random_state=0)`, fit it, predict on `X_test`, and return the mean squared error against `y_test`, rounded to 4 decimal places.
Unlike a single decision tree, XGBoost builds MANY small trees in sequence. What does each new tree actually learn to predict?
You can train an XGBoost model and measure how error drops as boosting rounds increase, and understand that each new tree specifically targets the ensemble's current residual error.