Experiment Tracking
18 min
A serious ML project runs dozens or hundreds of experiments — different
hyperparameters, architectures, data splits. Experiment tracking
means logging every run's configuration and results somewhere queryable,
so questions like "what was my best run?" or "which runs used
learning_rate=0.001?" have an actual answer instead of relying on
memory or scattered notebook cells.
def best_run(runs, metric_key="metric"):
return max(runs, key=lambda r: r[metric_key])["run_id"]
def runs_with_param(runs, param_name, param_value):
return [r["run_id"] for r in runs if r["params"].get(param_name) == param_value]
This is a miniature version of exactly what a real experiment tracker's query interface does -- 'find the run with the max metric' and 'filter runs by a param value' are the two most common queries against any experiment log, real tool or not.
Write `best_run(runs, metric_key='metric')`: `runs` is a list of dicts, each with a `run_id` and a metric value under `metric_key`. Return the `run_id` of the run with the highest metric.
Write `runs_with_param(runs, param_name, param_value)`: each run has a `'params'` dict of hyperparameters. Return the `run_id`s of every run whose `params[param_name]` equals `param_value`, in the given order.
Why do ML teams track every experiment run's hyperparameters and metrics (e.g. via MLflow, Weights & Biases) instead of just remembering 'the last good result'?
You can query a logged set of experiment runs to find the best one and filter by hyperparameter value — the core operations any experiment tracking tool provides.