Transformations vs. Actions
24 min
Apache Spark's execution model has one central idea: transformations
(map, filter, join, and similar) are LAZY — calling one doesn't
touch any data, it just records "here's another step in the plan."
Only an action (collect(), count(), save()) actually triggers
execution, running every recorded transformation in one pass. This
module builds a small pure-Python simulation of that exact model — real
Spark needs a JVM and isn't runnable in this browser sandbox, but the
laziness behavior itself is real and directly testable.
class LazyRDD:
def __init__(self, data, ops=None):
self.data = data
self.ops = ops or []
def map(self, fn):
return LazyRDD(self.data, self.ops + [("map", fn)])
def filter(self, fn):
return LazyRDD(self.data, self.ops + [("filter", fn)])
def collect(self):
result = list(self.data)
for kind, fn in self.ops:
if kind == "map":
result = [fn(x) for x in result]
elif kind == "filter":
result = [x for x in result if fn(x)]
return result
This is exactly why Spark code that builds a long transformation chain but never calls an action does NOTHING at all when run -- a genuinely common real beginner confusion, and precisely the behavior len(log) == 0 before collect() proves here.
Given the provided `LazyRDD` class (with `map` and `filter` already implemented as lazy — they just record the operation and return a new `LazyRDD`), write `collect(self)`: actually execute every recorded operation, in order, against `self.data`, and return the final list of results.
This module builds a small Python simulation of Spark's core execution model (real Spark isn't runnable in this browser sandbox). What real Spark concept does LazyRDD.map()/.filter() returning a NEW LazyRDD without running anything demonstrate?
You can implement lazy transformation chaining and eager action execution, the core distinction that makes Spark's execution model different from an ordinary eagerly-evaluated Python pipeline.