Foundations
Tokenization & Transformers
FundamentalHow text becomes tokens and how self-attention relates them.
What a token actually is
A token is the atomic unit a language model reads and writes - not a word, not a character, but a subword chunk produced by a learned compression algorithm called Byte-Pair Encoding (BPE).
Why not whole words?
A word-level vocabulary would need millions of entries to handle every inflection, compound, and proper noun. Unknown words become “[UNK]” - useless for generation. Subwords let the model recombine a compact vocabulary into any word it has never seen.
Why not characters?
Character-level models work, but every sentence becomes hundreds of steps. Attention cost scales with sequence length squared, so character sequences blow up the compute budget fast and make long-range dependencies harder to learn.
How BPE works
BPE starts with individual bytes and repeatedly merges the most frequent adjacent pair into a new symbol. After ~50 000 merges you get a vocabulary of common stems, suffixes, and punctuation. The word “tokenization” typically splits into tokenization - two tokens the model knows well.
Modern tokenizers (GPT-4's cl100k_base, Llama's SentencePiece variants, Gemma's tokenizer) all descend from the BPE idea, with different merge rules, vocabulary sizes, and byte-fallback strategies for rare characters.
See it for yourself
Type any text below and watch a real tokenizer split it into tokens. Switch between four popular tokenizers to see how the same string produces different splits - then hit Compare token counts to see how many tokens each one charges you for. Try prose, code, numbers, emoji, or a non-English sentence.
BPE · o200k_base · ~200k vocab
Loading GPT-4o tokenizer…
· marks a leading space (word boundary); ## marks a WordPiece continuation; amber chips are special tokens. Hover a chip to see its raw form.
Same text, four tokenizers
Tokenizers run entirely in your browser via Transformers.js; vocabularies are fetched from the Hugging Face Hub on first use and cached. No text leaves your device.
From token to vector
Text tokens are just integers - an ID in a lookup table. The transformer never touches the raw text; it works entirely with dense floating-point vectors. The translation happens in one lookup:
Step 1 - Tokenization
The tokenizer converts the input string into a sequence of integer IDs. “Hello world” might become [9906, 1917] in one vocabulary, or completely different integers in another.
Step 2 - Embedding lookup
Each integer indexes a row in the embedding matrix - a learnable table of shape [vocab_size, d_model]. The result is a dense vector of dimension d_model (e.g. 4 096 for Llama-3-8B, 8 192 for Llama-3-70B).
Step 3 - Into the transformer
These vectors, plus positional encodings that tell the model where each token sits, enter the first attention layer. Every subsequent layer reads and writes in this same vector space. See Attention mechanisms for how that works.
The embedding matrix is a large chunk of model parameters: at vocab size 128 000 and d_model 4 096, it accounts for roughly 500 M parameters before you count a single attention layer. This is why large-vocabulary models cost more memory even at the same layer depth. For the full memory accounting, see Model sizing.
See embeddings for real
The lookup above is abstract until you watch it happen. Below, a real embedding model turns your text into actual vectors - first one per token, then a similarity map showing how embeddings place related words close together in space.
Every token becomes a vector
Type a phrase. Each token is mapped to a real 384-dimensional vector by all-MiniLM-L6-v2. Click a token to see its actual embedding - orange dimensions are positive, blue are negative, brighter means larger magnitude.
Loading embedding model (~25 MB, first time only)…
Embeddings capture meaning
A vector on its own is just numbers. The point is distance: words with related meaning land near each other. Below, each word is embedded and compared with every other word by cosine similarity (1.0 = identical direction, 0 = unrelated). Edit the list and recompute.
Up to 8 words.
This demo uses a single small model (all-MiniLM-L6-v2, 384 dimensions) running entirely in your browser. Every model has its own embedding matrix learned during training - a different model produces different vectors of a different dimension (BERT 768, Llama-3-8B 4 096), so embeddings are not portable across models. The structure you see here - meaning encoded as geometry - is what every model learns, even though the exact numbers differ.
The vocabulary, and why tokenizers differ across models
Every model family ships its own tokenizer trained on its own corpus. The same sentence can become 12 tokens for one model and 19 for another - with real consequences for cost, context window usage, and quality.
Vocabulary size tradeoffs
- ►Larger vocab (100k–200k): common words become single tokens, fewer tokens per sentence, smaller sequences. The trade-off is a bigger embedding matrix and more softmax overhead at the output layer.
- ►Smaller vocab (32k–50k): leaner parameter count, but more tokens per sentence and harder coverage of rare languages or technical notation.
- ►Byte fallback: modern tokenizers reserve slots for raw UTF-8 bytes so no character is ever truly unknown - it just costs more tokens.
Why the same text differs
- ►The BPE merge order is determined by the training corpus. A model trained heavily on code will merge programming symbols into fewer tokens than one trained on news.
- ►Whitespace handling varies: some tokenizers treat a leading space as part of a token (“ hello” vs “hello” are different token IDs).
- ►You cannot safely share token IDs across model families. A prompt crafted for GPT-4 will have a different ID sequence when fed to Llama or Gemma, even if the text is identical.
Model architecture choices like GQA and MLA affect how the transformer processes those vectors - not how they are tokenized. See Model architectures for that layer of the stack.
Why tokens are the unit of cost and compute
Tokens are not just an internal implementation detail - they are the fundamental unit along which every measurable resource scales: API billing, context window limits, memory for the KV cache, and inference latency.
API billing
Providers charge per input token and per output token. A 10 000-token context costs ~10× a 1 000-token context - regardless of how many words are in it.
Context window
The maximum sequence length (e.g. 128k, 1M) is a token count, not a word or character count. One dense paragraph of code can fill the window faster than the same paragraph of prose.
KV cache memory
Each token in the context occupies a slice of the KV cache for every attention layer. Longer token sequences mean more GPU memory. See the KV cache page for exact formulas.
Throughput & latency
Inference systems measure decode speed in tokens per second. A reasoning model that emits 8 000 output tokens per query costs 8× the memory bandwidth of one that emits 1 000.
Characters per token by content type
The rule of thumb for English prose is ~3–4 characters per token, or ~0.75 words per token. But that ratio shifts dramatically by content type. Accurate token estimation matters for cost forecasting and model sizing.
| Content type | Example | Chars / token | Words / token | Notes |
|---|---|---|---|---|
| Plain English prose | “The model generates a response.” | ~3.5–4 | ~0.75 | Dense but predictable subwords |
| Python source code | def forward(self, x): | ~2.5–3.5 | - | Indentation & operators split frequently |
| JSON / structured data | {"key": "value"} | ~2–3 | - | Punctuation-heavy; brackets each get a token |
| Non-English text (e.g. Chinese) | 模型生成响应 | ~1–2 | - | Many scripts lack subword coverage; each character may be one token |
| Numbers | 3.14159265 | ~1–2 | - | Long integers and decimals split digit by digit |
| URLs / file paths | /api/v2/models/list | ~2–3 | - | Slashes and dots each consume tokens |
Figures are approximate and vary by tokenizer. Always use the model's own tokenizer to get exact counts before committing to a cost estimate or context window design. The KV cache calculator on this site uses token counts as its primary input.
Tokenization gotchas
Tokenization surprises are responsible for a surprising share of production bugs and cost overruns. Here are the most common ones:
Leading-space tokens
Most BPE tokenizers merge a leading space into the following word: the token for “ hello” (space+hello) is different from the token for “hello”. This means inserting or removing a space at a boundary changes which IDs are produced - a common source of off-by-one bugs when concatenating prompt fragments programmatically.
Numbers split digit by digit
Large integers and long decimals are often tokenized as individual digit characters: 3141592 may become 7 tokens. Arithmetic on long numbers is hard precisely because the model must learn relationships across tokens that represent a single quantity. If your application passes numeric data, format it carefully or consider pre-processing.
Code and markup are token-expensive
Every bracket, brace, semicolon, and indentation byte in source code competes for vocabulary slots. A 500-character Python function may consume 200+ tokens - far more per character than the equivalent prose explanation. Budgeting context for code-heavy prompts requires explicit token counting, not character estimation.
Non-English text costs more
Vocabularies built primarily on English corpora have sparse coverage for other scripts. A Chinese, Arabic, or Thai sentence that takes 20 tokens in its native vocabulary may cost 60–100 tokens in a tokenizer trained on English text. This raises API costs, consumes context window, and can impair model quality on those languages if training coverage is also thin.
Images, audio, and video become tokens too
Multimodal models extend the token concept beyond text. Vision encoders (e.g. ViT-style patch embeddings) map image patches into the same d_model vector space as text tokens - a 336×336 image at 14-pixel patches yields ~576 image tokens before the text prompt even starts. Audio is similarly chunked into spectrogram frames. These “virtual tokens” fill the context window and the KV cache the same way text tokens do; a single image can cost more context than a long paragraph.
Where to go next
Now that you understand how text becomes tokens and tokens become vectors, the next step is understanding how those vectors interact - via attention - and how the KV cache avoids recomputing that work on every new token.