Orchestration Basics
20 min
CI/CD's pipelines module computed one FULL execution order upfront —
useful when everything runs predictably start to finish. A real data
pipeline orchestrator (Airflow, Dagster, and similar tools) instead runs
a live loop: given which tasks have already completed, figure out which
tasks are ready to run RIGHT NOW (every prerequisite satisfied), launch
them, wait for more completions, and repeat. This handles retries,
variable task durations, and partial failures far better than a single
fixed schedule computed in advance.
def ready_tasks(dependencies, completed):
ready = []
for task, deps in dependencies.items():
if task not in completed and all(d in completed for d in deps):
ready.append(task)
return ready
An orchestrator calls this exact function again every time a task finishes -- 'extract done' unlocks 'transform', 'transform done' unlocks 'load', and so on -- rather than committing to one static order the way CI/CD's topological_order did.
Write `ready_tasks(dependencies, completed)`: `dependencies` maps each task to a list of prerequisite tasks. Return the list of tasks that are NOT already in `completed` but whose every prerequisite IS in `completed` — i.e. tasks ready to run right now.
CI/CD's `pipelines` module computed a FULL static execution order upfront (topological sort). Why does a live orchestrator (like Airflow) instead repeatedly ask 'what's ready to run RIGHT NOW'?
You can compute which tasks are ready to run given a dependency graph and a set of completed tasks — the core readiness check behind live workflow orchestrators.