Get in touch
Open to principal-scope roles · Phoenix, Arizona · Remote or hybrid
Open to principal-scope roles · Phoenix, Arizona · Remote or hybrid
Learn AI · On the map · Language · idea 15 of 44
The small chunks of text an AI actually reads, often word pieces rather than whole words.
In one line
The small chunks of text an AI actually reads, often word pieces rather than whole words.
Picture it
Lego bricks of language. 'Unbelievable' might be three bricks: un, believ, able.
Under the hood
Produced by a tokenizer, and tokenizers differ between models. Context windows, limits and usage are all counted in tokens; for Claude a token is about 3.5 English characters on average, varying by language.
Builds on
On a guided path
Path 3: What a language model does, stop 1 of 10.
Marking "I get this" is self-reported and stays in this browser. The same mark shows on the map.