Latticework — Master the skills engineers actually use

Explanation

Every compiler (and interpreter) starts the same way: turn raw source text into structure. The first stage is lexing (tokenizing) — scan character by character and group them into meaningful chunks called tokens: "12 + 3" becomes [12, "+", 3]. The lexer doesn't know or care whether the tokens form a valid expression — that's the parser's job, next.

def tokenize(expr):
    tokens = []
    i = 0
    while i < len(expr):
        c = expr[i]
        if c == " ":
            i += 1
            continue
        if c.isdigit():
            j = i
            while j < len(expr) and expr[j].isdigit():
                j += 1
            tokens.append(int(expr[i:j]))
            i = j
        elif c in "+-*/()":
            tokens.append(c)
            i += 1
    return tokens

Try it

The inner while loop that scans forward while expr[j].isdigit() is exactly how a real lexer handles multi-character tokens -- a single digit could be the start of a much longer number, so the lexer has to keep consuming until it finds a character that clearly doesn't belong.

Loading editor…

Exercise

Write `tokenize(expr)`: split an arithmetic expression string (digits, `+ - * / ( )`, and spaces) into a list of tokens — integers as `int`, operators/parens as single-character strings. Skip spaces.

Quiz

What is a 'lexer' (tokenizer) responsible for, that a parser is NOT?

Checkpoint

You can write a lexer that turns an arithmetic expression string into a token stream, the first stage of every compiler and interpreter.

Lexing & Parsing