Tokenization
18 min
A language model's input layer works entirely with numbers — vectors looked up from a fixed vocabulary — not raw text. Tokenization is the bridge: split text into pieces (words, or in real LLMs, smaller sub-word chunks via algorithms like BPE), then map each piece to an integer ID via a vocabulary built ahead of time from training data. Anything not seen during vocabulary-building becomes an "unknown" token.
def build_vocab(corpus_words):
vocab = {}
for w in corpus_words:
if w not in vocab:
vocab[w] = len(vocab)
return vocab
def encode(text, vocab, unk_id=-1):
words = text.split()
return [vocab.get(w, unk_id) for w in words]
Real tokenizers (BPE, WordPiece, SentencePiece) split on SUB-WORD units rather than whole words specifically so an unseen word like 'dog' never has to become a single opaque <unk> token -- it can instead be broken into familiar smaller pieces the model has actually seen before.
Write `build_vocab(corpus_words)`: given a list of words, return a dict mapping each UNIQUE word to an integer ID, assigned in first-seen order starting from 0.
Write `encode(text, vocab, unk_id=-1)`: split `text` on whitespace, and return the list of each word's ID from `vocab` — or `unk_id` for any word not in the vocabulary.
Why can't a language model work directly on raw text characters — why does it need tokenization first?
You can build a word-level vocabulary and encode text into integer token IDs, the numeric bridge every language model's input layer relies on.