Attention Intuition
24 min
Attention is how a transformer decides, for each position, how much to weight information from every OTHER position. The core computation has three ingredients per token: a query (what am I looking for?), a key (what do I offer, for others to match against?), and a value (what do I actually contribute, once matched?). A query's similarity to each key (via dot product) becomes that key's raw attention score.
import math
def softmax(scores):
max_score = max(scores)
exps = [math.exp(s - max_score) for s in scores]
total = sum(exps)
return [round(e / total, 4) for e in exps]
Subtracting max_score before exponentiating (max_score - max_score = 0 for the largest score) doesn't change the final result at all -- softmax is shift-invariant -- but it keeps every exponent <= 0, which avoids overflow for large input scores. A real implementation detail, not just a style choice.
Once every key has a softmax weight (summing to 1 across all keys), the attention output for that query is simply the weighted average of every position's VALUE — positions with high-scoring keys contribute more, low-scoring ones contribute almost nothing. This is the mechanism that lets a model dynamically pull in context from anywhere in the sequence, rather than only ever looking at a fixed nearby window.
Using the provided `softmax(scores)`, write `attention_output(query, keys, values)`: compute each key's dot-product similarity to `query`, softmax those scores into weights, then return the WEIGHTED SUM of `values` using those weights (each component rounded to 4 decimal places).
In scaled dot-product attention, what does the softmax step accomplish?
You can implement scaled dot-product attention from scratch — query-key similarity scores, softmax normalization, and the resulting weighted sum over values.