Evaluating Retrieval
18 min
Once you have a retrieval pipeline, you need a way to measure whether it's actually any good. Two standard metrics, borrowed straight from Model Evaluation's broader metrics module but applied here to ranked retrieval: precision@k (of what you retrieved, how much was actually relevant?) and recall@k (of everything relevant, how much did you actually retrieve?). They trade off against each other — you could get perfect recall by just retrieving everything, tanking precision, or perfect precision by retrieving only one thing you're certain about, tanking recall.
def precision_at_k(retrieved, relevant, k):
top_k = retrieved[:k]
hits = sum(1 for r in top_k if r in relevant)
return round(hits / k, 4)
def recall_at_k(retrieved, relevant, k):
top_k = retrieved[:k]
hits = sum(1 for r in top_k if r in relevant)
return round(hits / len(relevant), 4)
c6 never shows up in `retrieved` at ANY k here -- no amount of increasing k will ever recover it, since it just isn't in the candidate list at all. That's a genuinely different failure mode from 'it's relevant but ranked too low' -- it means the retrieval step itself never even considered it a candidate.
Write `precision_at_k(retrieved, relevant, k)`: of the first `k` items in `retrieved`, what fraction are in the `relevant` set? Round to 4 decimal places.
Write `recall_at_k(retrieved, relevant, k)`: of everything in the `relevant` set, what fraction appear somewhere in the first `k` items of `retrieved`? Round to 4 decimal places.
What's the practical difference between optimizing for precision@k vs. recall@k in a retrieval system?
You can compute precision@k and recall@k for a ranked retrieval result, and reason about the tradeoff between them.