Context engineering for AI agents: why your agent fails in production

Most AI agents that fail in production do not fail because the model was not smart enough. They fail because nobody designed what the model was allowed to see. That work has a name, context engineering for AI agents, and it decides whether you get a demo or a system you can trust with real customers.

That distinction matters, because it changes what you fix. When an agent starts giving wrong answers, the usual response is to upgrade to a newer model, add a framework, or rewrite the system prompt for the fifth time. Those moves rarely help, and teams burn a quarter finding that out. The actual failure is almost always upstream: the agent was handed a policy document that contradicts another policy document, a database field that means three different things depending on which team filled it in, and no way to tell which one applies to the request in front of it.

Designing that input deliberately, as a system rather than a prompt, is now the largest and least understood part of building an agent that survives contact with real work.

This article is written for founders, CTOs, and operations leaders who already have an agent in production or a pilot that stalled, and who need to decide what to fix next. It assumes you have gone past the demo stage. If you are still deciding whether to try an LLM at all, this will be deeper than you need right now.

You will get a working definition of context engineering for AI agents, the 5 layers that make up an agent's context, the 6 ways context breaks once real users arrive, what this work actually costs and where the budget goes, how to specify it in a project so a vendor can be held to it, and who has to own it after launch.

‍

Key takeaways

• Context engineering is the practice of deciding what information, tools, and history an AI agent receives at each step, and building the systems that assemble that package reliably.

• Prompt engineering asks how to phrase a request. Context engineering asks what the model needs to know, where it comes from, and what to leave out.

• Model upgrades rarely fix a failing agent, because the constraint is usually retrieval quality, conflicting source data, or missing state, not reasoning ability.

• More context is not better. Agent accuracy degrades as the context window fills, so relevance beats volume.

• An agent's context has 5 layers: instructions, retrieved knowledge, tools, memory and state, and guardrails. Most production failures trace to one of them.

• In our builds, roughly 60 to 70 percent of the effort goes into context work, not model or interface work. Budget accordingly.

• Context decays. Policies change, schemas change, products launch. Without an owner and an evaluation suite, an accurate agent quietly becomes an inaccurate one.

‍

What is context engineering for AI agents?

Context engineering is the practice of deciding what information, tools, and conversation history an AI agent receives at each step of a task, and building the systems that assemble that package correctly every time. It covers retrieval, data modelling, tool design, memory, and the rules for what gets excluded.

The simplest way to see the difference from prompt engineering is to look at what each one asks. Prompt engineering asks how a request should be worded. Context engineering asks what the model needs in front of it to answer correctly, where that information comes from, how fresh it is, how it is formatted, and what should be kept out. One is a writing task. The other is an engineering and data problem, and it does not go away when a better model ships.

It also runs against an instinct most teams have, which is that more context produces better answers. It does not. Anthropic's engineering team describes context as a finite resource with diminishing returns, where model performance degrades as the token count grows, an effect often called context rot. Their guidance in effective context engineering for AI agents is to treat the context window as an attention budget and spend it on the smallest set of high-signal tokens that will do the job. Dumping an entire knowledge base into every call is not thoroughness. It is noise, and it costs you money on every request.

So the goal is not to give the agent everything. It is to give it the right thing, at the right moment, in a shape it can act on.

‍

Why a better model will not fix a failing agent

Take a refund agent, because it is the request we see most often and the failure pattern is always the same.

In the demo it looks excellent. Someone assembles 20 clean tickets, writes a tight prompt, connects the model, and the agent resolves 19 of them correctly. Everyone agrees it is ready. Then it goes live against the real queue.

Now it meets a 90 page terms document, a separate refund policy that marketing published last year, and a third set of rules the support lead keeps in a spreadsheet because the official policy was never updated for the new subscription tiers. All three are technically in force. Two of them contradict each other. The CRM has a status field that engineering uses to mean payment state and that support uses to mean case state. Nobody wrote that down, because everyone already knew.

The agent starts approving refunds it should decline, and declining ones it should approve. The team's first instinct is to switch to a stronger model. The stronger model produces the same wrong answers, more fluently, because no amount of reasoning ability tells it which of the three refund policies governs a customer who upgraded mid cycle in March. That is not a thinking problem. It is a missing input.

This is why data readiness, not model choice, is the thing that decides whether an agent project survives. Gartner has predicted that through 2026, organisations will abandon 60 percent of AI projects that are not supported by AI-ready data, in its analysis of how a lack of AI-ready data puts AI projects at risk. Read that number carefully. The projects do not fail at the model layer. They fail because the material the model was supposed to work from was never fit for the job.

The uncomfortable version of this, and the one worth acting on: an agent is a very fast reader of your internal information. If that information is contradictory, out of date, or ambiguous, the agent does not smooth it over. It amplifies the mess and applies it at machine speed to every customer.

‍

The 5 layers of an agent's context

An agent's context is not one thing. It is 5 layers assembled at runtime, and a production failure almost always traces back to exactly one of them. Knowing which layer broke is how you stop guessing.

1. Instructions

The system prompt sets the agent's role, its boundaries, and how it should behave when it is unsure. The common mistake is pitching this at the wrong altitude. Too vague and the agent invents its own policy. Too rigid, with a hardcoded rule for every case anyone has thought of, and it becomes brittle: the first situation nobody anticipated produces nonsense, and the prompt turns into a 4,000 word document nobody dares edit. Aim for clear principles plus escalation rules, and let retrieval supply the specifics.

2. Knowledge

This is the retrieved material: documents, records, policies, past cases. The design decision that matters is when it gets pulled. Loading everything up front is expensive and dilutes attention. Fetching just in time, at the moment the agent needs it, keeps the window clean and makes the agent's behaviour far easier to debug, because you can see exactly what it asked for. This layer is also where source quality shows up. Retrieval cannot resolve a contradiction that exists in your documents.

3. Tools

Tools are what the agent can call: a search over your knowledge base, a CRM lookup, a pricing calculation, a write action. Two things decide whether they help. First, are the tool descriptions unambiguous enough that the agent picks the right one under pressure. Second, and more often overlooked, is the shape of what comes back. A tool that returns a 12,000 token raw API response has just spent most of the attention budget on fields the agent will never use. Tool outputs need to be trimmed and structured deliberately, the same way you would design an internal API.

4. Memory and state

Multi step work needs something that persists: what has already been tried, what the user confirmed 3 steps ago, which records have been touched. Without it, agents repeat completed actions or lose a constraint the user gave at the start. The practical patterns here are compaction, where the agent summarises the run so far and continues in a fresh window, and structured notes kept outside the context window and read back on demand.

5. Guardrails

The last layer is what must never enter the context, and what the agent must never do with it. Permission scoping so the agent retrieves only what the requesting user is entitled to see, redaction of personal data before it reaches the model, and hard stops on irreversible actions. Under the EU AI Act this layer also carries your documentation burden, so treat it as a design requirement from day one rather than something to bolt on before launch.

‍

6 ways context breaks once real users arrive

These are the failure modes we see repeatedly in production agents. They are worth memorising, because each one has a different fix and they are easy to confuse with each other.

• Overload. The window is packed with marginally relevant material, and accuracy drops even though nothing is technically missing. Fix by tightening retrieval, not by adding instructions.

• Stale retrieval. The agent confidently cites a policy that was replaced 8 months ago, because the index was built once and never refreshed.

• Conflicting sources. Two documents disagree and nothing tells the agent which one wins. This is a content governance problem wearing an AI costume.

• Tool output bloat. A single verbose API response crowds out everything else in the window, including the user's actual question.

• Lost state. In a long task the agent forgets a constraint set early on, or redoes a step it already completed.

• Permission leakage. The agent surfaces information in an answer that the person asking was never cleared to see. This one is a security incident, not a quality issue.

Back to the refund agent. Its real problem was three at once: conflicting sources, because nobody had retired the old policy, stale retrieval, because the index predated the new subscription tiers, and an ambiguous schema that made the CRM status field unreadable to anything without tribal knowledge. Fixing it took no model change at all. It took retiring one policy document, adding an effective-date filter to retrieval so the agent could only see rules in force on the transaction date, and renaming two fields. Accuracy moved from roughly two thirds to the high nineties on the same model that had been blamed for the failure.

That is the shape of most of this work. It is unglamorous, it looks like data and documentation cleanup, and it is where the result comes from.

If you are scoping this kind of build, our SaaS founder's AI blueprint walks through how to structure an AI feature so it holds up in production, including the data questions worth answering before anyone writes a prompt. It is free and takes about 20 minutes to read.

‍

Context engineering vs prompt engineering

Prompt engineering is not obsolete, and anyone claiming it is has usually not shipped an agent. It is a genuine skill and it is still the fastest lever on a single, well-bounded task. What has changed is its share of the problem. Once an agent runs multi step work against live systems, wording is a small fraction of what determines the outcome, and the rest sits in the plumbing.

‍

‍

The practical test for which one you have: if the same request works when a person pastes the right document in by hand, and fails when the agent runs on its own, your prompt is fine and your context pipeline is not. We run that check early on almost every engagement, because it takes an hour and it settles an argument that teams otherwise have for weeks.

‍

Where the budget actually goes

The first surprise for most buyers is how little of an agent build is spent on the model. Inference is a real running cost, but as a share of the project it is usually minor next to the engineering around it. The work that consumes the budget is the context layer.

Across the agent projects we have delivered, the effort tends to distribute roughly like this. About 30 percent goes to discovery and data readiness: mapping where the truth actually lives, finding the contradictions, deciding what is authoritative. Another 30 percent goes to retrieval and tool design, which is the code that assembles context at runtime. Around 20 percent goes to evaluation and guardrails, meaning the test suite that proves the agent is right often enough to trust and the controls that stop it doing damage when it is wrong. The remaining 20 percent covers integration and the interface people actually use.

Those proportions shift with the project. An agent working over a clean, well governed data warehouse skews away from discovery. One working across 6 legacy systems and a shared drive skews hard toward it, and that is the case where a fixed quote given before anyone has looked at the data should worry you.

The useful consequence: if a proposal for an AI agent is mostly model selection, prompt design, and UI, with data work as a small line item near the bottom, it is priced for the demo and not for production. Ask what happens in week 6 when the first policy contradiction shows up, and see whether there is a plan or a change request.

‍

How to spec context engineering into an AI agent project

You do not need to design the retrieval architecture yourself. You do need to make the context layer explicit in the scope, so it is somebody's committed deliverable rather than a discovery nobody budgeted for. These are the 7 questions worth putting in writing before work starts.

• Which sources are authoritative for each decision the agent makes, and who signs off on that list?

• How does the agent know a document is current, and what happens when a policy is superseded?

• What is the retrieval strategy, and how will its accuracy be measured before the agent goes live?

• What does each tool return, and who is responsible for keeping those responses small and structured?

• What carries across steps and sessions, and what is deliberately forgotten?

• What must the agent never see, and how is that enforced per user rather than per system?

• What is the evaluation suite, what accuracy threshold gates the launch, and how often does it run after launch?

The last one carries more weight than it looks. An agent without an evaluation suite cannot be improved safely, because you have no way of knowing whether today's fix broke last month's behaviour. We treat the eval set as a deliverable in its own right, built from real cases including the ones that went wrong. If you want the fuller version of this, our guide on AI agent evaluation and knowing when it is ready for production covers how to build one and what threshold is defensible.

‍

Who owns context after launch

Context decays, and this is the part almost nobody budgets for. Your pricing changes. A policy gets rewritten. A product launches with terms the agent has never seen. Somebody renames a field in the CRM during an unrelated migration. None of these events touch the agent's code, and all of them change its answers.

An agent that was 95 percent accurate at launch and that nobody maintains does not fail loudly. It drifts, quietly, and you find out from a customer complaint 3 months later. That is worse than an outage, because there was no moment where anything obviously broke.

So name an owner. Someone has to be accountable for the agent's information, the way a data owner is accountable for a warehouse table. In practice that means a named person, a review cadence tied to the events that actually change context such as policy updates and product launches, and an evaluation suite that runs automatically whenever a source document or schema changes. This is a modest ongoing commitment, usually a few hours a month, and it is the difference between an agent that compounds in value and one that becomes a liability by its second year.

‍

How Codelevate approaches context engineering

We start every agent engagement with the data, not the model. Before anything is built, we map the decisions the agent has to make, trace each one back to the source that should govern it, and surface the contradictions. That first pass is often uncomfortable for the client, because it exposes process ambiguity that predates any interest in AI, and it is consistently the highest value week of the project.

From there we build the context pipeline as real software: versioned retrieval with effective-date awareness, tools that return tight structured responses, explicit state, and permission scoping enforced at retrieval rather than trusted to the prompt. Every build ships with an evaluation suite drawn from real cases, and an accuracy threshold agreed before launch, so the decision to go live is a measurement rather than a feeling.

We also say no to a fair number of agent requests, usually because the underlying process is not defined well enough for any system to automate it reliably. Telling a client that first is cheaper for them than finding out in month 4. If you want that assessment done properly on your own use case, that is what our AI development services are built around.

‍

‍

The takeaway

If your agent is failing, the model is probably not the problem. Look at what it was given: whether the sources agree, whether they are current, whether the tools return something usable, whether anything remembers what happened 3 steps ago. That is where the fix lives, and it is also where the durable advantage lives, because a competitor can switch to the same model tomorrow and cannot copy the years of clean, governed, well-modelled context sitting behind your agent.

The shift worth making is from treating an agent as a clever piece of software to treating it as a system that runs on your information. Once you see it that way, the roadmap gets obvious and the budget gets honest.

If you want a practical starting point, the SaaS founder's AI blueprint is a free download that covers how to scope, build, and validate an AI feature end to end, and it pairs directly with everything above.

And if you already have an agent that is not performing and you want a second opinion on why, book a free call with our team. Bring the failure cases. We will tell you honestly whether it is a context problem, a process problem, or something that should not have been an agent in the first place.