Before a model can reason about a dispute note or a fee schedule, it chops the text into tokens — the units it actually reads and generates. Tokens drive cost and context limits, so it pays to see them.
A model doesn't see letters or words — it sees tokens: common chunks of text, often word-pieces. "Representment" might be one token; a rare code like "DSP-4821" splits into several. Everything the model reads and writes is counted in tokens.
Common words are one token; rare words, IDs, and numbers split into several. Roughly 4 characters ≈ 1 token in English.
Pricing is per million tokens, both input and output. A long policy pasted into every prompt gets expensive fast.
The context window is measured in tokens. Big documents can overflow it — a key reason to use RAG and send only relevant chunks.
••••4921 and ACP-DIS-014 use more tokens than they look. The playground on the next tab lets you feel this with real payments text.Watch a sub-word tokenizer actually build itself. Byte-Pair Encoding starts from single characters and repeatedly merges the most frequent adjacent pair into a new token — press ▶ Play, or step through, to see common payments sub-words emerge.
No merges yet — press ▶ to start
Split text into characters, with ▁ marking word ends
Count every adjacent token pair
Merge the most frequent pair into one new token
Repeat — common sub-words emerge as single tokens
dispute / chargeback / representment collapse into single tokens after a few merges — which is why familiar payments words are cheap in tokens, while a rare code like DSP-4821 stays fragmented.Modern tokenizers use Byte-Pair Encoding (BPE) or similar. The idea: start from characters and repeatedly merge the most frequent pairs into bigger pieces, until you have a vocabulary of common sub-words. Frequent words become single tokens; rare ones stay in pieces.
Every word begins as individual characters the tokenizer already knows.
Pairs that appear often ("in", "ing", "ment") get merged into single tokens over training.
The result is a fixed vocabulary of common sub-words — enough to represent any text, common or rare.
New text is greedily matched to the longest known pieces. "representment" → "represent" + "ment"; "DSP" stays split.
| Approach | Problem it has | Why BPE wins |
|---|---|---|
| Whole words | Vocabulary explodes; can't handle new words, codes, or typos | BPE composes any word from known pieces |
| Single characters | Sequences get very long and slow | BPE keeps common words compact (one token) |
| Sub-words (BPE) | — | Balance: compact for common text, flexible for rare |
Because everything is counted in tokens, they shape the two numbers that matter most in production: what a request costs, and how much the model can consider at once.
| Rule | Detail |
|---|---|
| ~4 characters ≈ 1 token | For typical English; codes, numbers, and masked values run higher |
| ~0.75 words ≈ 1 token | So ~100 words ≈ ~130 tokens |
| Input + output both count | A long grounded answer costs tokens too — constrain the format |
| Whole doc vs RAG | A full policy can be thousands of tokens per call; a few retrieved chunks are a fraction of that |
There are four levels at which text can be split, and a few competing sub-word algorithms. Understanding the trade-offs explains why modern models all landed on sub-word tokenization.
| Level | How it splits | Example: "representment" | Trade-off |
|---|---|---|---|
| Word | Each word = 1 token | representment | Simple, but a huge vocabulary and can't handle unseen words, codes, or typos |
| Sub-word ⭐ | Meaningful pieces | represent + ment | Best balance — what Nova, Claude, Llama all use |
| Character | Each character = 1 token | r·e·p·r·e·s·… | Tiny vocabulary, but very long sequences (slow, costly) |
| Byte | Raw byte encoding | 114·101·112·… | Handles any language/symbol, but extremely long |
DSP-4821 stay in pieces.| Algorithm | How it decides merges | Used by (examples) |
|---|---|---|
| BPE (Byte-Pair Encoding) | Repeatedly merge the most frequent adjacent pair (what the playground shows) | GPT family, many open models |
| WordPiece | Merge the pair that most improves the model's likelihood, not just raw frequency | BERT family |
| SentencePiece / Unigram | Language-agnostic; starts from a large vocabulary and prunes to the most useful pieces | Llama, T5, multilingual models |
Tokenization is step one of every LLM pipeline — everything downstream depends on how text was split. Here's the whole path, from your text to a generated answer.
Tokenization is step 1 — the tokenizer turns your text into tokens, embeddings give those tokens meaning, the transformer weaves them into context, and the model generates an answer token by token.