What Are Tokens and Why Do LLMs Use Them Instead of Words?

Every time you talk to an AI model, your text is quietly chopped into tokens before the model ever sees it. Tokens are the real units an LLM reads and writes, and they are usually pieces of words rather than whole words. This sounds like an implementation detail, but it quietly shapes cost, context limits, and some genuinely strange model behavior. Understanding tokens is one of the highest-leverage basics in all of AI coding.

A token is a chunk of text

A token is a small unit of text, often a common word, a word fragment, or a piece of punctuation. The word cat might be one token, while a longer or rarer word like tokenization could split into several. On average, across English text, a token works out to roughly four characters or about three-quarters of a word. So a paragraph you think of as fifty words might be sixty-five or seventy tokens to the model. The mapping is not one token per word, and that gap matters more than it first appears.

Why not just use words

Using whole words as the unit sounds natural but breaks down fast. A model would need a fixed vocabulary, and human language has an effectively unlimited number of words once you count names, typos, code identifiers, and every language on earth. Any word missing from that vocabulary would be impossible to represent at all. Tokens solve this by breaking rare or unseen words into familiar smaller pieces, so the model can always represent anything. It is the difference between needing a word for everything and being able to spell anything from a small alphabet of fragments.

Subword tokenization is the compromise

The standard approach sits between letters and words, and is usually called subword tokenization. Common words get their own single token for efficiency, while rare words are assembled from a handful of frequent fragments. This keeps the vocabulary at a manageable size, often around a hundred thousand tokens, while still covering any possible input. Techniques like byte-pair encoding build this vocabulary by repeatedly merging the most common pairs of characters. The result is a compact set of pieces that can spell out anything the model encounters. Reference material on tokenization covers the mechanics in depth.

From text to numbers

Models do not actually work with text, they work with numbers, and tokens are the bridge. Each token in the vocabulary maps to a unique number, so your sentence becomes a list of integers before anything else happens. Those numbers are what flow through the network, and the model predicts the next number, which is then translated back into a token and shown to you. The whole conversation, from your side and the model’s, is really a stream of token numbers. Text is just the human-facing surface of a numeric process.

Tokens explain the context window

The famous context window is measured in tokens, not words or characters. When a model advertises a limit, that ceiling is a number of tokens it can consider at once, covering your prompt, any history, and its own reply. This is why a long code file can eat into your available space faster than you expect, since code often tokenizes less efficiently than prose. It also ties directly to how the model uses everything in its context as working memory. Every token you add is a token spent against a hard budget.

Tokens explain the bill

Pricing for AI models is almost always measured per token, both for what you send and what you receive. That means the length of your prompts and the verbosity of the responses translate directly into cost. A chatty system prompt repeated on every call, or an agent that reads many files, can quietly run up a large token count. Thinking in tokens rather than words is what lets you reason about spend accurately. Once you see the meter running per token, you start writing leaner prompts by instinct.

Tokens explain weird behavior

A lot of odd model quirks make sense once you think in tokens. Models have historically struggled to count the letters in a word or reverse it, because they never see individual letters, only whole tokens. Splitting a number into unusual tokens can throw off arithmetic, and a rare identifier chopped into fragments can confuse the model. None of this is the model being dim, it is a direct consequence of the unit it reads in. When output looks strangely wrong at the character level, tokenization is often the culprit.

Different models tokenize differently

There is no single universal tokenizer, so the same text can be a different number of tokens across models. Two providers may split your prompt differently, which affects both cost and how much fits in context. This is one more reason that comparing models purely on advertised numbers is misleading, a theme running through any honest look at how models differ. The safe habit is to check token counts with the specific model you are using rather than assuming. What is cheap on one model can be pricier on another for identical text.

Why this matters for coding agents

For agentic coding the stakes are higher, because agents consume tokens fast. An agent that reads your files, runs tools, and iterates in a loop can burn through tokens at a rate that surprises people used to simple chat. Every file it opens and every step it takes is more tokens against both your budget and your context limit. This is part of why AI coding tool pricing leans so heavily on usage rather than flat fees. Efficient token use is not a nicety at this scale, it is a real cost lever.

Thinking in tokens is a superpower

Once tokens click, a dozen separate mysteries collapse into one idea. Context limits, pricing, character-level errors, and model differences all trace back to the same humble unit. You do not need to compute token counts by hand, you just need to feel their weight when you write a prompt or point an agent at a repository. That instinct, sensing how heavy an input is, is quietly one of the most practical skills in the whole field.

Common questions