How Large Language Models Work: Tokens, Inference, Memory and Reasoning

If you write code with AI tools, a working model of how an LLM produces text saves you hours of confusion. This guide covers the whole chain in one read: what a model is, how text becomes tokens, how answers are generated one token at a time, why the model seems to remember your conversation, what reasoning modes add, and why the same prompt can give different results.

Large language models power almost every AI tool you touch, yet most explanations of them jump straight to jargon. The core idea is simpler than the branding suggests. An LLM is a program that has learned, from an enormous amount of text, to predict what word is likely to come next. Everything it does, from writing code to answering questions, is built on that single ability scaled up to a staggering degree.

A prediction engine at heart

Strip away the mystique and an LLM is a next-token predictor. You give it some text, and it estimates the probability of every possible next piece, then picks one. That is genuinely the whole loop at the center of the machine. It looks like conversation and reasoning, but underneath it is always answering the same narrow question: given everything so far, what comes next? The magic is not the mechanism, it is how much capability emerges when that mechanism is trained on nearly the whole internet.

Trained on a mountain of text

The model gets its ability from training, not from being programmed with rules. During pretraining it is shown vast quantities of text and repeatedly asked to predict the next piece, adjusting billions of internal parameters each time it is wrong. Over trillions of examples it gradually absorbs grammar, facts, styles, and patterns of reasoning, not by memorizing them but by tuning itself to reproduce them. Nobody writes the rules for how a verb conjugates or how a function is structured. The model infers all of it from exposure, the way you would learn a language by reading endlessly.

Attention is the key mechanism

The architecture that made this work is called the transformer, and its central trick is attention. Instead of reading strictly left to right, the model can weigh how relevant every earlier word is to the one it is producing now. That lets it connect a pronoun to the noun it refers to, or a closing brace to the function that opened it. Visual explainers like the illustrated guide to the transformer show this mechanism far better than equations do. Attention is why these models handle long, structured context so much better than their predecessors.

It generates one token at a time

When an LLM writes, it does not compose a whole answer at once. It predicts the first token, appends it to the input, and then predicts the next token based on that slightly longer text, repeating until it decides to stop. This one-at-a-time process is called autoregressive generation, and it is why you often see responses stream out word by word. Each new token is a fresh prediction conditioned on everything written so far. The fluent paragraph you read is really thousands of tiny next-step guesses stitched together.

Randomness makes it creative

The model does not always pick the single most likely next token, and that is deliberate. A setting often called temperature controls how much randomness enters the choice, so the same prompt can yield different but plausible results. Turn it down and the output grows focused and repetitive, turn it up and it grows varied and occasionally strange. This controlled randomness is what makes a model feel creative rather than robotic. It is also why agent workflows built on these models rarely produce the exact same run twice.

Why it feels like understanding

When prediction gets good enough, it becomes almost indistinguishable from understanding. To predict the next token well across code, math, and prose, the model has to build internal representations that behave a lot like concepts. That is why it can follow instructions and reason through problems, even though its only training objective was guessing the next piece of text. Whether that counts as real understanding is a genuine debate, and worth holding loosely. For practical purposes, treat it as an extraordinarily capable pattern machine rather than a mind.

Not a database of facts

A crucial consequence is that an LLM is not looking anything up. It has no table of facts to consult, only patterns compressed into its parameters, so it generates the most plausible-sounding continuation rather than a retrieved truth. This is exactly why models sometimes state false things with total confidence, a behavior usually called hallucination. Reference material on large language models stresses this point, because it explains most of their failures. Plausible and correct are not the same thing, and the model optimizes for the first.

Training happens in stages

The model you actually use went through more than raw pretraining. After learning to predict text, it is fine-tuned on examples of helpful behavior and shaped further by human feedback, which teaches it to follow instructions and avoid obvious harms. This later stage is where much of its personality and manners come from. The base predictor supplies the raw capability, and the tuning supplies the willingness to be useful. Both are essential to the assistant you experience.

Context is its working memory

An LLM has no memory of past conversations on its own, so everything it uses for a task has to be present in the text you give it. That input, the prompt plus any history or documents, is its entire working memory for that moment. This is why the same model can seem brilliant or clueless depending on what you put in front of it. Managing that input well is a large part of what separates good results from bad, and it underpins the whole shift toward modern agent workflows. What the model sees is what the model knows.

The same engine behind every tool

Once you see the core clearly, the landscape of products stops being confusing. Every assistant, editor, and terminal agent is the same kind of next-token engine wrapped in a different interface. The differences you feel between tools come mostly from the surface and the tuning, not from some fundamentally different intelligence underneath. Understanding this is what lets you place any new tool on the map of surfaces within minutes. Learn the engine once and every product becomes a variation on a theme.

Its limits shape how to use it

Knowing how an LLM works tells you exactly where to be careful. Because it predicts rather than retrieves, it can be confidently wrong, and because its knowledge is frozen at training time, it can be out of date. These are not bugs to be annoyed by but properties to design around, which is why review and verification matter so much. Treating output as a strong draft to check, in the spirit of measuring value rather than trusting hype, is the healthy default. The model is a powerful collaborator precisely as long as you remember what it is.

Why the basics are worth it

This one concept, prediction trained at scale, unlocks nearly everything else in AI coding. It explains why outputs vary, why models hallucinate, why context matters, and why different tools feel different. You do not need the math to get the leverage, you just need the mental model. Hold onto it, and the rest of the field stops being a wall of jargon and starts being a set of sensible consequences.

What Are Tokens and Why Do LLMs Use Them Instead of Words?

Every time you talk to an AI model, your text is quietly chopped into tokens before the model ever sees it. Tokens are the real units an LLM reads and writes, and they are usually pieces of words rather than whole words. This sounds like an implementation detail, but it quietly shapes cost, context limits, and some genuinely strange model behavior. Understanding tokens is one of the highest-leverage basics in all of AI coding.

A token is a chunk of text

A token is a small unit of text, often a common word, a word fragment, or a piece of punctuation. The word cat might be one token, while a longer or rarer word like tokenization could split into several. On average, across English text, a token works out to roughly four characters or about three-quarters of a word. So a paragraph you think of as fifty words might be sixty-five or seventy tokens to the model. The mapping is not one token per word, and that gap matters more than it first appears.

Why not just use words

Using whole words as the unit sounds natural but breaks down fast. A model would need a fixed vocabulary, and human language has an effectively unlimited number of words once you count names, typos, code identifiers, and every language on earth. Any word missing from that vocabulary would be impossible to represent at all. Tokens solve this by breaking rare or unseen words into familiar smaller pieces, so the model can always represent anything. It is the difference between needing a word for everything and being able to spell anything from a small alphabet of fragments.

Subword tokenization is the compromise

The standard approach sits between letters and words, and is usually called subword tokenization. Common words get their own single token for efficiency, while rare words are assembled from a handful of frequent fragments. This keeps the vocabulary at a manageable size, often around a hundred thousand tokens, while still covering any possible input. Techniques like byte-pair encoding build this vocabulary by repeatedly merging the most common pairs of characters. The result is a compact set of pieces that can spell out anything the model encounters. Reference material on tokenization covers the mechanics in depth.

From text to numbers

Models do not actually work with text, they work with numbers, and tokens are the bridge. Each token in the vocabulary maps to a unique number, so your sentence becomes a list of integers before anything else happens. Those numbers are what flow through the network, and the model predicts the next number, which is then translated back into a token and shown to you. The whole conversation, from your side and the model’s, is really a stream of token numbers. Text is just the human-facing surface of a numeric process.

Tokens explain the context window

The famous context window is measured in tokens, not words or characters. When a model advertises a limit, that ceiling is a number of tokens it can consider at once, covering your prompt, any history, and its own reply. This is why a long code file can eat into your available space faster than you expect, since code often tokenizes less efficiently than prose. It also ties directly to how the model uses everything in its context as working memory. Every token you add is a token spent against a hard budget.

How Does the Token-by-Token Inference Process Work Under the Hood?

When you watch a model reply, the text streaming out one piece at a time is not a stylistic choice, it is the machinery showing through. Generating a response, a process called inference, happens strictly token by token in a loop. Understanding that loop explains streaming, cost, latency, and a lot of model behavior that otherwise seems arbitrary. It is one of the most clarifying things you can learn about how these systems actually run.

Inference means running the model

Training is how a model learns, and inference is what happens every time you actually use it. During inference the model’s parameters are frozen, and it simply runs your input forward to produce output. No learning happens while you chat, despite how it might feel, because the weights do not change. Inference is the model applying what it already learned, over and over, to each new request. Every reply you have ever gotten from an AI was the product of inference, not training.

It starts by reading the prompt

The first phase is digesting everything you sent. The model reads your entire prompt, which has already been split into tokens, and processes it in one pass to build up an internal understanding of the context. This initial step is often called the prefill, and it is where the model absorbs the whole input before writing anything. The longer your prompt, the more work this phase takes. It is the model getting up to speed on the situation before it opens its mouth.

Then it predicts one token

With the prompt absorbed, the model produces exactly one token. It computes a score for every possible next token in its vocabulary, turning them into probabilities, and then selects one according to those odds. This is the same next-token prediction that underlies everything an LLM generates, happening for real, one step at a time. That single chosen token, maybe a word, maybe a fragment, is the first piece of your answer. Everything the model says is built from this one operation repeated.

This is why responses stream

The token-by-token nature is exactly why replies appear gradually rather than all at once. Each token is ready the instant it is produced, so applications show it to you immediately instead of waiting for the whole answer. What looks like typing is really the generation loop surfacing each token as it finishes. This is not a cosmetic animation, it is the real pace of the underlying computation. You are watching the model think in real time, one prediction at a time.

When does it stop

The loop does not run forever, and knowing why it stops is useful. The model can produce a special end token that signals it is finished, and applications also set a maximum length to cap runaway output. When either condition is met, generation halts and the response is complete. This is why a reply can occasionally cut off mid-thought if it hits the length ceiling before reaching a natural end. Stopping is a decision, made either by the model or by the app around it.

How Does ChatGPT Remember Your Conversation if LLMs Are Stateless?

Here is a puzzle that trips up almost everyone new to AI. The language model behind ChatGPT is stateless, meaning it remembers nothing between one request and the next. Yet the chat clearly recalls your name, your last question, and the code you pasted three messages ago. Both of those things are true at once, and the way they fit together reveals how these systems really work.

The model itself has no memory

Each time an LLM is called, it starts completely fresh. It has no built-in record of previous requests, no notebook, and no sense that it has ever spoken to you before. A call goes in, a response comes out, and the moment it finishes the model retains nothing at all. This is what stateless means, and it is true even in the middle of what feels like a flowing conversation. The model you talk to at message ten is, from its own point of view, meeting you for the first time.

The trick is resending everything

So how does the conversation hold together? The application, not the model, keeps the history, and it resends the entire conversation with every single message. When you type your tenth message, the app quietly bundles up all nine previous exchanges plus your new one and sends the whole thing as the input. The model then predicts a reply as if it is reading the whole discussion for the first time, which, in a sense, it is. What looks like memory is really a very fast rerun of the full transcript on every turn.

Memory lives in the context, not the model

This means the conversation lives in the context window, the block of text the model reads on each call. Everything the assistant appears to remember is simply text that got included in that input again. The model is not recalling your name, it is reading your name, freshly, in the transcript it was just handed. This is the same principle by which an LLM uses its context as working memory for any task. Remove a line from that resent history and the model instantly forgets it, because for the model there was never anything else.

Why this design makes sense

Statelessness sounds like a limitation, but it is a deliberate and useful design. A stateless model is simple to scale, since any server can handle any request without needing to know its history. It also keeps each interaction clean and reproducible, with the full input visible rather than hidden in some internal state. The cost of carrying context is pushed to the application, which is exactly where it can be managed and inspected. Simplicity in the model buys enormous flexibility in the system around it.

Context windows set the limit

Because the whole conversation is resent each time, it cannot grow forever. Every message you exchange adds to the transcript, and that transcript is capped by the model’s context window, measured in tokens. Once a long conversation approaches that ceiling, something has to give, or the model would simply run out of room to read. This is why very long chats can start to feel like the assistant is losing track of the earliest details. The limit is not the model forgetting, it is the transcript no longer fitting.

What Is LLM Reasoning, and Why Does “Thinking Step by Step” Produce Better Outputs?

One of the strangest and most useful facts about language models is that they get better at hard problems when you simply ask them to think out loud. Give a model a tricky question and it may blurt a wrong answer, but ask it to work through the steps first and it often lands correctly. This is the heart of what people mean by LLM reasoning. It looks like a trick, but it reflects something real about how these models actually compute an answer.

Reasoning is thinking on the page

For a language model, reasoning is not a hidden faculty, it is text the model writes before its final answer. When it reasons, it produces a chain of intermediate steps, working through a problem in the open rather than leaping to a conclusion. Those steps are generated the same way as any other output, one piece at a time. The visible working is not a report of some separate thought process, it is the thinking itself. The model reasons by writing, not by writing up reasoning it did elsewhere.

Why writing the steps helps

The reason this works comes down to computation. A model does a fixed amount of work per token it generates, so forcing a hard answer into a single step gives it very little room to compute. Letting it produce many intermediate tokens gives it far more total computation to spend on the problem. Each step it writes also becomes part of the context it reads for the next step, so it can build on its own partial work. More visible steps means more thinking room and more chances to catch itself, which is exactly why the answers improve.

Thinking step by step in a prompt

The simplest way to trigger this is to ask for it directly. Adding a phrase like work through this step by step nudges the model to lay out its reasoning before committing to an answer. This technique, often called chain-of-thought prompting, reliably improves performance on math, logic, and multi-step coding problems. It costs almost nothing to try and frequently turns a wrong answer into a right one. When a model fumbles a hard question, asking it to slow down and show its work is the first thing worth reaching for.

Reasoning models bake it in

Newer models take this further by being trained to reason extensively on their own. So-called reasoning models will spend a long internal stretch working through a problem before they answer, without you having to ask. They have learned that generating a thorough chain of thought produces better results, so they do it by default on hard tasks. This is why some models pause noticeably before responding to a difficult prompt. The step-by-step habit that used to be a prompting trick has become a built-in capability.

It is still prediction underneath

It is worth staying grounded about what is happening here. The reasoning steps are still generated by the same next-token prediction that drives everything an LLM produces. There is no separate logic engine switching on, only the model conditioning each step on the ones before it. That is powerful, because good reasoning patterns learned from training genuinely help, but it also means the reasoning can look sound and still be wrong. The appearance of careful thought is not a guarantee of a correct conclusion.

What’s the Difference Between an LLM Like GPT and an AI Application Like ChatGPT?

GPT and ChatGPT get used interchangeably in everyday talk, but they are not the same kind of thing at all. One is a model, a raw engine that predicts text, and the other is a full product built around such a model. Blurring the two is behind a surprising amount of confusion about what AI can and cannot do. Once you separate the engine from the car, a lot of the field suddenly makes more sense.

The model is the raw engine

An LLM like GPT is, at its core, a function that takes text and returns likely next text. On its own it has no interface, no memory, no buttons, and no personality beyond what its training gave it, as covered in any explanation of what a large language model is. You typically reach a bare model through an API, sending it input and getting back output. It is enormously capable and completely inert until something wraps around it. The engine can generate power, but by itself it does not take you anywhere.

The application is the whole car

ChatGPT is the car built around that engine. It adds a chat interface, conversation history, a system prompt that shapes behavior, safety filters, tools, and often long-term memory. All of those parts are the application, not the model, and they are what make the raw predictor usable by ordinary people. When you chat with ChatGPT, only a fraction of the experience is the model itself. The rest is careful engineering wrapped around it to turn a capability into a product.

The system prompt sets the personality

Much of an assistant’s character comes from a hidden instruction the app sends with every request. This system prompt tells the model who to be, how to behave, and what rules to follow, all before you type a word. Two products can use the identical model and feel completely different because their system prompts differ. What reads as the model’s personality is often the application’s instructions doing the work. The engine is the same, but the driver has given it very different orders.

Tools turn talk into action

A bare model can only produce text, so on its own it cannot browse, run code, or touch a database. Applications add tools that let the model take real actions, turning a text generator into something that can do things in the world. When ChatGPT searches the web or runs Python, that is the application handing the model capabilities it does not have by itself. The model decides when to use a tool, but the app is what actually provides and executes it. Action is an application feature layered onto a talking engine.

Why the distinction matters

This is not pedantry, because the confusion has real consequences. People credit the model for things the app does, like memory or web access, and then get surprised when a different tool on the same model behaves differently. Understanding the split tells you where a capability actually lives and therefore who can change it. It also explains why the surface you use shapes your experience so strongly, since the surface is the application layer. Know which layer you are talking about and the whole picture clarifies.

Why Do Autonomous Agent Outputs Vary Between Models Like ChatGPT and Claude, and How Should You Embrace That?

Hand the exact same task to ChatGPT and to Claude and you will often get two noticeably different answers. The instinct is to ask which one is right, but that is usually the wrong question. The two models diverge because they were built differently, and learning to use that divergence is more useful than trying to erase it.

Same job, different minds

Two frontier models can be equally capable and still approach a problem in distinct ways. One may write terse, clever code while the other writes verbose, defensive code, and both can be correct. Thinking of them as two experienced engineers with different instincts is closer to the truth than thinking of them as two copies of one calculator. They agree on the basics and diverge on style, judgment, and taste. Ask each to solve a genuinely tricky problem and the shape of the answer, not just the wording, will differ. That is a sign of a real difference in approach, not a rounding error.

Different training, different instincts

The deepest reason is training. Each large language model learns from a different mixture of data, so the patterns it absorbs and prefers are not identical. Different examples lead to different habits, defaults, and blind spots. Before any tuning at all, the raw material each model learned from already pushes them toward different answers. No two providers curate their data identically, and those choices ripple all the way to the output. The model is, in a real sense, a compression of what it was shown.

Different tuning and defaults

On top of the base training sits a layer of tuning that shapes personality and behavior. The reinforcement and instruction tuning each provider applies encodes different values about verbosity, caution, formatting, and how the model handles uncertainty. System defaults then nudge behavior further, so one model asks clarifying questions while another charges ahead. Much of the felt difference between ChatGPT and Claude lives in this tuning layer rather than raw capability. This is why one assistant feels cautious and another feels eager, even on identical prompts. The base intelligence may be close, but the manners are trained separately.

Different tool-use styles

In agentic work the differences get sharper, because the models also decide when and how to use tools. One model may reach for a code execution tool early while another reasons further before acting, and their planning styles differ in ways that change the whole run. This is part of why the coding agents built on different models feel distinct even doing the same job. The model is not just writing code, it is choosing a strategy.

This shows up most in agents

A single answer hides these differences, but an autonomous run puts them on full display. Over dozens of steps, small divergences in judgment compound into very different solutions, which is exactly why comparisons like Cursor versus Claude Code feel like more than a surface swap. The longer the agent runs unattended, the more its underlying model shows through. Personality becomes trajectory. A small early preference, like reaching for a test first, steers everything that follows. By the end of a long run, two models can land on genuinely different architectures.

Related guides
What Is an AI Agent? Tools, Loops and Multi-Agent Systems Explained
Context Engineering for AI Coding: Context Windows, Compaction and Fresh Chats
Best LLMs for Agentic Coding: How to Choose, Compare and Switch Models

Common questions