Learning Rate & Rounds
20 min
The previous module showed more boosting rounds reduce error — but
n_estimators isn't the only knob. learning_rate scales down how
much of each new tree's correction actually gets applied (a
learning_rate of 0.1 only applies 10% of what a tree "wants" to
correct). A HIGH learning rate corrects aggressively in few rounds; a
LOW learning rate takes smaller, more cautious steps — more stable and
often better-generalizing, but it needs many more rounds to accumulate
the same total correction.
from xgboost import XGBRegressor
def evaluate_learning_rate(X_train, y_train, X_test, y_test, n_estimators, learning_rate):
model = XGBRegressor(n_estimators=n_estimators, learning_rate=learning_rate, max_depth=2, random_state=0)
model.fit(X_train, y_train)
preds = model.predict(X_test)
mse = sum((float(p) - y) ** 2 for p, y in zip(preds, y_test)) / len(y_test)
return round(mse, 4)
This ISN'T evidence that a high learning_rate is just better -- it's evidence that low learning_rate + few rounds is UNDER-trained. Given enough additional rounds, a low learning rate typically catches up to (and sometimes surpasses) a high learning rate's final fit quality, more stably.
Write `evaluate_learning_rate(X_train, y_train, X_test, y_test, n_estimators, learning_rate)`: train an `XGBRegressor(n_estimators=n_estimators, learning_rate=learning_rate, max_depth=2, random_state=0)`, and return the mean squared error on `X_test`/`y_test`, rounded to 4 decimal places.
Why does a LOW learning_rate need MORE boosting rounds to reach the same fit quality as a HIGH learning_rate?
You can evaluate how learning_rate and n_estimators interact, and understand why they must be tuned together, not independently.