Selecting Few-Shot Examples
18 min
Few-shot prompting shows a model a handful of example input/output
pairs before asking it to handle a new case. A FIXED set of examples,
reused for every query, often doesn't match the specific pattern the
current query actually needs. Dynamic example selection fixes this:
embed a library of candidate examples once, then for each new query,
retrieve the k examples most similar to it — the exact same retrieval
idea from Embeddings and RAG Fundamentals, applied here to prompt
construction instead of document search.
def cosine_similarity(a, b):
dot = sum(x * y for x, y in zip(a, b))
mag_a = sum(x ** 2 for x in a) ** 0.5
mag_b = sum(x ** 2 for x in b) ** 0.5
return dot / (mag_a * mag_b)
def select_few_shot_examples(query_vector, examples, k):
scored = sorted(examples, key=lambda e: -cosine_similarity(query_vector, e[1]))
return [example_id for example_id, _ in scored[:k]]
At real scale, this is exactly how tools like DSPy and LangChain's example selectors work -- a small library of curated, embedded examples, with the k most relevant ones pulled dynamically into each prompt rather than hand-picking a static few-shot set once and reusing it everywhere.
Using the provided `cosine_similarity`, write `select_few_shot_examples(query_vector, examples, k)`: `examples` is a list of `(example_id, vector)`. Return the `k` example IDs whose vectors are most similar to `query_vector`, most similar first.
Why select few-shot examples by SIMILARITY to the current query, rather than using the same fixed set of examples for every prompt?
You can dynamically select the most relevant few-shot examples for a given query via similarity search, instead of relying on one fixed example set for every prompt.