Feature Importance
18 min
After training an ensemble of trees, XGBoost can tell you which
features it actually relied on: .feature_importances_ scores each
feature by how much it was used across every tree's splits, weighted
by how much those splits actually reduced error. A feature the model
barely ever splits on — or splits on without much benefit — scores low;
one the model consistently leans on to make accurate predictions scores
high.
from xgboost import XGBRegressor
def most_important_feature(X_train, y_train, feature_names):
model = XGBRegressor(n_estimators=20, max_depth=2, random_state=0)
model.fit(X_train, y_train)
importances = model.feature_importances_
best_idx = max(range(len(importances)), key=lambda i: importances[i])
return feature_names[best_idx]
This is exactly why feature importance is such a practically useful debugging tool -- if a feature you EXPECTED to matter scores near zero, or one you thought was irrelevant scores surprisingly high, that's a real signal to double-check your data (a leak, a bug, or a genuinely surprising real pattern).
Write `most_important_feature(X_train, y_train, feature_names)`: train an `XGBRegressor(n_estimators=20, max_depth=2, random_state=0)`, then use its `.feature_importances_` array to return the name (from `feature_names`) of the single most important feature.
Where does XGBoost's feature_importances_ score for each feature actually come from?
You can extract and interpret feature importance from a trained XGBoost model, and understand it's derived from actual split usage across the ensemble, not a precomputed correlation.