Pipelines
20 min
Real models almost always need preprocessing before training — scaling
features, encoding categories. Doing that by hand invites a subtle but
serious bug: forgetting to apply the EXACT SAME transformation to new
data at prediction time, or accidentally fitting a scaler using
statistics from the test set (leaking information the model shouldn't
have access to). A Pipeline bundles preprocessing and the model into
one estimator that shares scikit-learn's usual fit/predict
interface — call .fit() once, and every step (fit correctly, in
order) happens automatically forever after.
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
def build_and_run_pipeline(X_train, y_train, X_test):
pipe = Pipeline([("scaler", StandardScaler()), ("clf", DecisionTreeClassifier(random_state=0))])
pipe.fit(X_train, y_train)
return [int(p) for p in pipe.predict(X_test)]
Behind the scenes, pipe.fit(X_train, y_train) fits the scaler on X_train ONLY, then fits the classifier on the SCALED training data -- pipe.predict(X_test) reuses that already-fitted scaler (never re-fitting it on X_test), which is exactly the leak-proof behavior manual step-by-step code has to get right by hand every time.
Write `build_and_run_pipeline(X_train, y_train, X_test)`: build a `Pipeline` chaining a `StandardScaler()` step named `'scaler'` and a `DecisionTreeClassifier(random_state=0)` step named `'clf'`, fit it on the training data, and return its predictions on `X_test` as a list of ints.
Why bundle preprocessing (like scaling) and the model together in a single Pipeline, instead of just scaling the data manually before calling fit()?
You can chain preprocessing and modeling steps into a single Pipeline, and understand why that prevents the data-leakage and inconsistent-preprocessing bugs manual step-by-step code is prone to.