Tokenization Strategies

A token is the smallest chunk of text a model reads. Tokenization is the process of cutting text into those chunks. There are three families of approach, and each trades vocabulary size against sequence length.

The example below builds a tiny tokenizer by hand so you can see exactly what "splitting text" means, then runs the single most important step of subword learning.

BPE is the tokenizer behind GPT and LLaMA. The idea is simple enough to do on paper. You just repeat one rule until your vocabulary is the size you want:

1. Start with every word as a list of single characters.
2. Count every adjacent pair of symbols across all words.
3. Merge the most frequent pair into one new symbol everywhere.
4. Record that merge, then go back to step 2.

Each merge adds one entry to the vocabulary. After thousands of merges the frequent words and fragments ("the", "ing", "tion") have become single tokens, while anything rare still decomposes into smaller learned pieces — so the model is never stuck on a new word. You already ran step 1 through 3 once in the example above; the only thing a real trainer adds is a loop.

BPE is one subword algorithm; two close cousins power most other models. They differ in how they pick merges and how they mark where words begin.

Below, a real Hugging Face tokenizer shows the ## continuation prefix and the special tokens BERT wraps around your text. Read the # Expected output comments to see exactly what each call returns.

The output of training is a vocabulary: a fixed list of every token the model knows, each paired with a unique integer token ID. The model never works with text — it works with these IDs. Encoding turns text into a list of IDs; decoding turns IDs back into text.

A handful of IDs are reserved for special tokens that carry structure rather than meaning. They never appear in ordinary text:

• [CLS] — start-of-input marker BERT uses to summarise the whole sequence for classification.
• [SEP] — separates segments (and marks the end of a BERT input).
• <eos> / <|endoftext|> — "end of sequence" for GPT-style models, telling them to stop generating.

The example below uses OpenAI's tiktoken — the actual tokenizer GPT-4 uses — so you can see real token IDs and where a leading space lives.

# The real thing: OpenAI's tokenizer (a fast byte-level BPE).
# pip install tiktoken
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")   # used by GPT-4 / GPT-3.5

text = "Tokenization isn't magic."
ids = enc.encode(text)                       # text  -> token IDs (integers)
print("Token IDs:", ids)
# Expected output: Token IDs: [3404, 2065, 4267, 956, 11204, 13]

# Every ID maps back to a piece of text (a "token"):
pieces = [enc.decode([i]) for i in ids]
print("Tokens:", pieces)
# Expected output: Tokens: ['Token', 'ization', ' isn', "'t", ' magic', '.']

print("Token count:", len(ids))
# Expected output: Token count: 6

# Note the leading SPACE inside ' isn' and ' magic'.
# Byte-level BPE attaches the space to the FOLLOWING word — that is normal.

Token count is not an academic detail — it is what you pay for and what limits how much the model can read at once. Two hard constraints flow directly from it:

• Cost: APIs bill per token (for example, per 1,000 tokens). A wordier prompt is a more expensive call.
• Context length: every model has a maximum number of tokens it can hold. Your prompt plus the model's answer must fit inside that window.

A useful rule of thumb for English: 1 token ≈ 4 characters ≈ 0.75 words. Code, long numbers, and non-English scripts use far more tokens per word, so always measure rather than guess. The exercise below turns a token count into a dollar cost and a remaining-context budget.