Latticework — Master the skills engineers actually use

Explanation

Real pipelines fail partway through and get retried — a network hiccup, a worker crash, an orchestrator restart. If the LOAD step simply appends every record it sees, a retried batch creates duplicate rows. The fix is to make loading idempotent: upsert by a stable key (insert if new, overwrite if it already exists) so running the exact same batch twice leaves the target in the identical final state either way — this is the same guarantee Kafka's delivery-guarantees module achieved for message processing, applied here to the load step of a pipeline.

def load_upsert(target, records, key_field):
    for record in records:
        target[record[key_field]] = record
    return target

Try it

Contrast this with Kafka's delivery-guarantees dedup_process, which detects and skips duplicate MESSAGES by ID -- here, idempotency comes from the STORAGE OPERATION itself (upsert-by-key naturally overwrites rather than duplicates), so there's no need to track which records were 'already seen' at all.

Loading editor…

Exercise

Write `load_upsert(target, records, key_field)`: for each record in `records`, set `target[record[key_field]] = record` (insert if new, overwrite if the key already exists). Return `target`.

Quiz

Why must an ETL pipeline's LOAD step be idempotent (safe to re-run with the same input, producing the same final result) rather than simply appending every batch?

Checkpoint

You can implement an idempotent upsert-based load step, so a pipeline retry never produces duplicate data.

Idempotent Loads