Tuning num_leaves
18 min
Just as XGBoost's learning_rate and n_estimators trade off (small
steps need more rounds), LightGBM's num_leaves trades off against
n_estimators too: a larger leaf budget lets each individual tree
capture more complexity per round — often reaching a good fit in fewer
total rounds — but risks overfitting faster, especially on small
datasets, since bigger, leaf-wise-grown trees can carve out very
specific (and potentially noise-fitting) regions of the data.
from lightgbm import LGBMRegressor
def evaluate_num_leaves(X_train, y_train, X_test, y_test, num_leaves):
model = LGBMRegressor(n_estimators=20, num_leaves=num_leaves, random_state=0, verbosity=-1, min_child_samples=1)
model.fit(X_train, y_train)
preds = model.predict(X_test)
mse = sum((float(p) - y) ** 2 for p, y in zip(preds, y_test)) / len(y_test)
return round(mse, 4)
Notice the improvement here (37.76 -> 35.50) is much smaller than XGBoost's learning_rate demo (168.77 -> 1.62) -- num_leaves=2 is an EXTREME restriction (barely more than a single split per tree), so this specific comparison mostly illustrates the floor of what's achievable with a near-stump-sized tree budget, not a dramatic tuning win.
Write `evaluate_num_leaves(X_train, y_train, X_test, y_test, num_leaves)`: train an `LGBMRegressor(n_estimators=20, num_leaves=num_leaves, random_state=0, verbosity=-1, min_child_samples=1)`, and return the mean squared error on `X_test`/`y_test`, rounded to 4 decimal places.
XGBoost's tuning module showed n_estimators and learning_rate need to be tuned TOGETHER. What's the equivalent coupled relationship for LightGBM's num_leaves?
You can tune LightGBM's num_leaves and understand its tradeoff against n_estimators, the leaf-wise-growth analog of XGBoost's learning_rate/n_estimators relationship.