A Token Is Not A Word

I kept hearing that a language model “reads words”. It does not. It reads tokens, and I could not have told you where a token comes from.

So I built the thing that makes them. The page trains a byte-pair-encoding tokenizer from scratch, in the browser, on two chapters of Alice in Wonderland. It starts with single characters, counts every neighbouring pair, merges the most frequent one, and repeats until nothing repeats. That loop is the whole intelligence. “Alice” becomes one token because it is frequent. " ALICE" stays five pieces because the book never shouts. “tokenizer” never becomes one token, because no such word existed in 1865.

GPT-4’s tokenizer, measured on the same chapters, still needs 1.24 tokens per word. Closer to one, never one.

Type your own sentence and watch it split.