—
A byte-pair-encoding tokenizer trained from scratch, in your browser, on a sample of
the first two chapters of Alice's Adventures in Wonderland. Nothing on this page
is a lookup into a shipped vocabulary: every token boundary, every count and every
curve is produced by the merge loop in js/model.js running on that text
when the page loads, and again whenever you change the sample. The same file runs
under node, which is how the claims below were tested.
That loop knows nothing about words. Words fall out of it only because frequent things merge first, and because the pre-tokeniser stops every merge at a space.
| Bucket | What is in it |
|---|---|
| computed live | Every merge, its count, and the order they happen in. The vocabulary. The split of your text after any number of merges, its tokens per word and characters per token. The split of the training text and of the held-out paragraphs, and both curves. The pair leaderboard. The merge at which any named token was born, which is what several narration stops jump to. The stopping point itself: training ends when no pair repeats, and where that happens depends on the sample. |
| measured | The three reference points on the chart. On 2026-09-20, the full two chapters
(4,274 words, 22,119 characters) were encoded with tiktoken 0.14.0:
r50k_base (GPT-2, 50,257 tokens) → 5,805 tokens, 1.358 per word;
cl100k_base (GPT-4, 100,277) → 5,309 tokens, 1.242 per word;
o200k_base (GPT-4o, 200,019) → 5,305 tokens, 1.241 per word. Vocabulary
sizes include each encoding's special tokens. These were not produced by this
page; the page cannot run those tokenizers. |
| assumed | Characters, not bytes. Production tokenizers start from the 256 byte values so any string is encodable; this page starts from the characters that occur in the corpus so the first merges are readable. A non-Latin letter or an emoji is therefore one starting symbol here and two to four in GPT-2. The stop rule — never merge a pair seen only once — is mine; real training stops at a chosen vocabulary size instead. The sample: the tokenizer sees 75% of the corpus's words, whole paragraphs in a seeded shuffled order, so that "different sample" means something and so the remaining 25% can be a held-out test. Sample 0 is the whole text. Ties between equally frequent pairs go to the pair that appears first in the text; any deterministic rule would do. The pre-tokeniser is GPT-2's; GPT-4's and GPT-4o's differ in detail (they split digits into groups of at most three and match contractions case-insensitively). |
| faked for legibility | Leading spaces are drawn as ␣; they are ordinary spaces in the data. The corpus panel shows only the first few hundred characters of the sample, tokenised with the same merges as everything else. Chip colour encodes a token's length in characters, in five bands. The orange ring marks tokens made by the most recent merge and fades on the next. Playback speed is a slider; the training itself finishes in under a second. |
| unresolved | Why a production tokenizer has a single token for a word like ␣Alicia cannot be answered from here: it depends on a training corpus nobody outside the vendor has seen. The comparison on the chart is between the same loop at two scales, not a claim about how those vocabularies were chosen. Whether the early merge order would change under byte-level encoding: yes, slightly, because multi-byte characters would first have to merge back into themselves; the order of the common English pairs would not. |
| Claim | Source |
|---|---|
| Byte-pair encoding as a compression algorithm: replace the most frequent pair of bytes with an unused byte, repeat. | Gage, P. (1994). "A New Algorithm for Data Compression." C Users Journal 12(2). |
| BPE adapted to word segmentation for neural machine translation, merging characters instead of bytes, for open-vocabulary translation. | Sennrich, Haddow & Birch (2016). "Neural Machine Translation of Rare Words with Subword Units." ACL 2016, pp. 1715–1725. aclanthology.org/P16-1162 |
| Byte-level BPE and the pre-tokenisation regex used here. | Radford et al. (2019). "Language Models are Unsupervised Multitask Learners"
(GPT-2), §2.2; the regex is from openai/gpt-2 encoder.py, reproduced as
r50k_pat_str in openai/tiktoken openai_public.py. |
| Vocabulary sizes and pre-tokenisers of cl100k_base and o200k_base; the measured tokens-per-word figures. | openai/tiktoken openai_public.py; measurement run locally with
tiktoken 0.14.0 on 2026-09-20. |
| The corpus. | Carroll, L. (1865). Alice's Adventures in Wonderland, chapters I–II.
Project Gutenberg ebook #11 (public domain). Line-wrapping joined, Gutenberg's
_italic_ markers and scene-break asterisks removed; no word changed. |
That production tokenizers are trained this exact way — they use byte-level symbols, far larger corpora, deduplication and vendor-specific rules. That tokens per word is a good measure of anything but cost — it is the denominator readers most often assume is one. Or that a tokenizer should know words: the whole design point is that it does not need to.
Press A or Esc to close.