How Do System Prompts, Tool Descriptions, Memory, and Conversation History Form the Input Context for AI Agents?

When an AI agent takes a step, the model behind it receives one big block of text and nothing else. That block, the input context, is not a single thing you wrote but an assembly of several distinct parts stitched together on every call. Knowing what those parts are, and how they combine, is the foundation of controlling how an agent behaves. It is the anatomy lesson behind all of context engineering.

The context is assembled every call

Because a model is stateless, the whole context has to be rebuilt and resent on every single step, exactly as conversation memory works under the hood. Nothing persists inside the model between calls, so the application gathers all the relevant pieces and concatenates them into one input each time. This assembly is invisible to you, but it is happening constantly. Understanding that the context is freshly built each call is the key to understanding everything else. The agent is only ever as good as the pile it is handed this turn.

The system prompt sets the stage

First in the assembly is usually the system prompt, a hidden instruction that defines who the model is and how it should behave. It sets the role, the tone, the rules, and the goals before the agent reads anything else. Because it comes first and frames everything after, it has outsized influence over the whole run. Two agents with identical everything else but different system prompts will behave very differently. This is the layer where much of an agent’s character and reliability is decided.

Tool descriptions advertise what it can do

Next come the descriptions of the tools the agent may use. Each tool is presented with a name, a purpose, and the inputs it expects, so the model knows what actions are available. These descriptions are not just documentation, they actively steer when and how the model reaches for each tool. Vague descriptions lead to misused tools, while precise ones lead to correct choices. In effect, the tool descriptions are part of the context that programs the agent’s behavior.

Memory carries facts forward

Memory is the part that lets an agent seem to remember things across time. Stored facts, whether about the user, the project, or earlier decisions, get pulled in and inserted into the context so the model can use them. This memory lives outside the model, in the application, and is added back into the input when relevant. It is how an agent recalls your preferences or a decision made yesterday. Without it, every session would start from nothing, since the model itself remembers nothing.

Conversation history provides continuity

The running history of the current session is another major component. Previous messages, actions, and their results are included so the model can follow the thread and build on what already happened. This is what makes an ongoing task coherent rather than a series of disconnected steps. As the session grows, this history is often the largest and fastest-growing part of the context. Managing it well is central to keeping a long agent run on track.

Retrieved documents add knowledge

Many agents also pull in outside information on demand. When a task needs specific knowledge, the application can retrieve relevant documents, code, or data and drop them into the context for that step. This lets an agent work with information far beyond its training, grounded in your actual files and systems. The retrieval is selective, fetching only what seems relevant so it does not overwhelm the budget. Well-chosen retrieved context is often what makes an agent accurate on your specific project.

The user request points the way

Somewhere in the assembly is the actual task, the thing you asked the agent to do. It might be your latest message or a goal handed to the agent at the start, and it gives the whole context its direction. Everything else in the context exists to help the model accomplish this request well. It is easy to forget that your instruction is just one part of a much larger input. The model weighs it alongside everything else it was given.

It all shares one budget

Every one of these parts competes for the same finite space, measured in tokens. A long system prompt, verbose tool descriptions, a big memory, and a growing history all draw from the same limited window, so they trade off against each other. Spend too much on one and there is less room for the rest, including the model’s own reasoning. This shared budget is exactly why context engineering is a balancing act. Every part has to earn its place in the window.

Assembly order and format matter

How the pieces are arranged and formatted is not neutral either. Models pay more attention to some positions than others, and clear structure helps the model tell the system prompt from the history from the retrieved data. Good assembly makes the important parts easy for the model to find and use. This is a subtle but real part of the craft, closely tied to the broader discipline of context engineering. The same information, arranged badly, produces worse results.

Why this matters for agents

For agents this assembly runs on every step of a long loop, so getting it right compounds. A well-built context keeps an agent focused and capable across dozens of turns, while a bloated or disorganized one makes it drift and fail. Because the model only ever sees this assembled pile, controlling the pile is controlling the agent. This is why serious agent work is really context work. The assembly is the steering wheel.

The takeaway

An agent’s input context is assembled fresh on every call from a system prompt, tool descriptions, memory, conversation history, retrieved documents, and your request, all sharing one token budget. Each part shapes the output, and how you build and balance them determines how the agent behaves. Learn the anatomy, and you learn where every lever for controlling an agent actually is.

Common questions