For decades, software architecture was something you could draw on a whiteboard, validate with unit tests, and ship. Design came first, implementation second, evaluation last. That linear workflow worked because the components were deterministic: a database behaves like a database, an API returns what its contract promises, and a function does exactly what its code says.
LLM-based agents break that contract. At the heart of every agent is a large language model — a component that is probabilistic, opaque, and unevenly capable. You cannot fully architect an agent in an offline room anymore, because you are not really designing around code; you are designing around a model whose strengths and weaknesses only reveal themselves when the agent is actually running.
1. The Offline Design Problem
When you design a traditional system, you can reason about behavior from specifications. With an LLM agent, the specification
is the model itself, and the model does not come with a reliable manual. A given LLM may be excellent at structured reasoning, terrible at following multi-step instructions, surprisingly good at creative writing, and dangerously wrong at arithmetic. These traits are not uniform across models, or even across different versions of the same model.
This makes offline agent design fundamentally uncertain. You can sketch the tool set, define the workflow, and write the prompts, but you cannot know whether the model will reliably execute the intended behavior until you test it against real tasks. Every LLM has a unique profile of capabilities and shortfalls, and those profiles change as models are updated. The architecture that looked elegant on paper can collapse the moment the model makes an unexpected decision.
A skeptic would say this is just TDD and SRE observability applied to a new component. The objection sounds reasonable: we already iterate, measure, and observe. But the difference is mechanistic, not metaphorical. A unit test asserts a deterministic output:
f(2) must equal
4, every time. An LLM does not have a single correct output; it has a distribution of plausible outputs. A test that passes on Tuesday may fail on Wednesday, not because the code changed, but because the model sampled a different path. You cannot write a boolean assertion against that behavior. Your “tests” must become statistical — pass rate over N trials, variance across prompts, sensitivity to phrasing — and that changes what it means to ship with confidence.
2. Design, Build, and Evaluate Through Feedback Loops
Because the model is the heart of the agent, design, implementation, and evaluation can no longer be sequential. They have to happen concurrently. You build a prototype, evaluate it against realistic tasks, discover where the model fails, redesign the prompts or tooling, and evaluate again. This iterative loop is not a refinement phase at the end — it is the primary way to discover what the system can actually do.
By mid-2026, the dominant metaphor has shifted from “prompt engineering” as a linguistic trick to
feedback-driven design: treating the LLM as one component within a larger, self-correcting state machine. Some recent writing calls this “loop engineering” — the architecture of autonomous iteration — where the system itself reflects, critiques, and regenerates its work rather than relying on a single shot of model output.
Recent work in the field reflects this shift. A 2024 paper on
Evaluation-Driven Development of LLM Agents argues for embedding explicit feedback pipelines into the agent lifecycle so that systems can evolve iteratively rather than being designed once and deployed (
arXiv:2411.13768). The authors describe how runtime logs, metrics, and traces from real operation should feed back into test cases and safety cases, keeping offline evaluations aligned with real-world demands.
Instead of trying to craft the perfect prompt, you build feedback loops: a Critic or Verifier reviews the agent’s output, the agent reflects on the feedback, and the loop regenerates the response. This
think-correct-execute pattern is especially important for complex tasks like code generation, mathematical reasoning, and multi-step planning.
Worked example: from hallucinated SQL to schema-aware generation
Consider a natural-language analytics task:
“What were the top 5 products by revenue in Q2 2024?”
Version 1: ask the model to write SQL directly. It produces:
SELECT product_name, SUM(revenue)
FROM sales
WHERE quarter='Q2 2024'
GROUP BY product_name
ORDER BY revenue DESC
LIMIT 5
The SQL is syntactically plausible, but the
sales table does not exist and the
revenue column does not exist. A deterministic validator catches it —
Unknown table: SALES — but a critic alone cannot recover, because the model has no source of truth about the database. This is the difference from TDD: the failure is not in the code; it is in the model’s hidden knowledge gap. Re-running the same prompt will not fix it.
Trace of the failed attempt:
{
"kind": "plan",
"content": {
"approach": "naive direct SQL generation",
"sql": "SELECT product_name, SUM(revenue) FROM sales WHERE quarter='Q2 2024' GROUP BY product_name ORDER BY revenue DESC LIMIT 5"
}
}
{
"kind": "critique",
"content": {
"passed": false,
"feedback": "Unknown table: SALES"
}
}
{
"kind": "final",
"content": {
"status": "failed_naive",
"lesson": "Model hallucinated schema it was never shown. Need explicit schema tools before generation."
}
}
Version 2: add explicit schema tools. Before generation, the agent calls
get_schema and
get_table_sample. It now sees that revenue must be computed from
orders.quantity * products.unit_price, and that the date filter belongs on
orders.order_date. It generates:
SELECT p.product_name, SUM(o.quantity * p.unit_price) AS total_revenue
FROM orders o
JOIN products p ON o.product_id = p.product_id
WHERE o.order_date BETWEEN '2024-04-01' AND '2024-06-30'
GROUP BY p.product_id, p.product_name
ORDER BY total_revenue DESC
LIMIT 5
The validator passes it, the query executes, and the trace logs every step. The cost went from $0.025 to $0.065 — a small, measurable premium for replacing a hallucination with a verifiable chain.
Trace of the successful run:
{
"task": "What were the top 5 products by revenue in Q2 2024?",
"cost_estimate": 0.065,
"steps": [
{
"kind": "tool_call",
"content": {
"tool": "get_schema",
"result": {
"orders": {
"columns": ["order_id", "product_id", "quantity", "order_date", "customer_id"],
"sample": [
[1, 101, 2, "2024-04-15", 5001],
[2, 102, 1, "2024-05-20", 5002],
[3, 101, 5, "2024-06-10", 5003],
[4, 103, 10, "2024-05-05", 5004],
[5, 101, 1, "2024-06-25", 5005]
]
},
"products": {
"columns": ["product_id", "product_name", "unit_price"],
"sample": [
[101, "Widget A", 25.0],
[102, "Widget B", 40.0],
[103, "Widget C", 15.0]
]
}
}
}
},
{
"kind": "tool_call",
"content": {
"tool": "get_table_sample",
"result": {
"orders": [
{"order_id": 1, "product_id": 101, "quantity": 2, "order_date": "2024-04-15", "customer_id": 5001},
{"order_id": 2, "product_id": 102, "quantity": 1, "order_date": "2024-05-20", "customer_id": 5002},
{"order_id": 3, "product_id": 101, "quantity": 5, "order_date": "2024-06-10", "customer_id": 5003}
],
"products": [
{"product_id": 101, "product_name": "Widget A", "unit_price": 25.0},
{"product_id": 102, "product_name": "Widget B", "unit_price": 40.0},
{"product_id": 103, "product_name": "Widget C", "unit_price": 15.0}
]
}
}
},
{
"kind": "plan",
"content": {
"approach": "schema-aware SQL generation",
"sql": "SELECT p.product_name, SUM(o.quantity * p.unit_price) AS total_revenue\nFROM orders o\nJOIN products p ON o.product_id = p.product_id\nWHERE o.order_date BETWEEN '2024-04-01' AND '2024-06-30'\nGROUP BY p.product_id, p.product_name\nORDER BY total_revenue DESC\nLIMIT 5"
}
},
{
"kind": "critique",
"content": {
"passed": true,
"feedback": "SQL passes read-only, schema, and shape checks."
}
},
{
"kind": "execute",
"content": {
"rows": [
{"product_name": "Widget A", "total_revenue": 175.0},
{"product_name": "Widget C", "total_revenue": 150.0},
{"product_name": "Widget B", "total_revenue": 40.0}
],
"row_count": 3
}
},
{
"kind": "judge",
"content": {
"helpfulness": 5,
"correctness_confidence": 5,
"justification": "Query uses validated schema and returns grouped revenue data."
}
},
{
"kind": "final",
"content": {
"status": "success",
"sql": "SELECT p.product_name, SUM(o.quantity * p.unit_price) AS total_revenue\nFROM orders o\nJOIN products p ON o.product_id = p.product_id\nWHERE o.order_date BETWEEN '2024-04-01' AND '2024-06-30'\nGROUP BY p.product_id, p.product_name\nORDER BY total_revenue DESC\nLIMIT 5",
"result": {
"rows": [
{"product_name": "Widget A", "total_revenue": 175.0},
{"product_name": "Widget C", "total_revenue": 150.0},
{"product_name": "Widget B", "total_revenue": 40.0}
],
"row_count": 3
}
}
}
]
}
Worked example: when feedback forces a model upgrade
Schema tools fix the knowledge-gap problem, but they do not fix every capability gap. Some queries require reasoning patterns the cheap model simply cannot produce. The feedback loop then has to include a routing decision.
Consider this follow-up task:
“Which customers bought Widget A in Q2 2024 and also bought Widget C within 30 days?”
The weak model tries the same approach: it retrieves the schema and sample rows, then generates SQL. But the query requires a self-join on the
orders table with a date window. The weak model cannot hold that structure in mind, so it invents a helper table that does not exist:
SELECT customer_id, product_name
FROM customer_purchases
WHERE product_name IN ('Widget A', 'Widget C')
AND purchase_date BETWEEN '2024-04-01' AND '2024-06-30'
GROUP BY customer_id
HAVING COUNT(DISTINCT product_name) = 2
The deterministic validator rejects it —
Unknown table: CUSTOMER_PURCHASES. At this point the feedback is not “try again with the same brain.” The feedback is “this task is too hard for the current model; escalate to a stronger one.” The router logs the decision:
{
"kind": "routing",
"content": {
"decision": "strong",
"reason": "Weak model hallucinated a helper table; query requires self-join and date-window reasoning."
}
}
The stronger model, given the same schema and sample rows, generates the correct self-join:
SELECT DISTINCT a.customer_id, a.order_date AS widget_a_date, c.order_date AS widget_c_date
FROM orders a
JOIN orders c ON a.customer_id = c.customer_id
JOIN products pa ON a.product_id = pa.product_id
JOIN products pc ON c.product_id = pc.product_id
WHERE pa.product_name = 'Widget A'
AND pc.product_name = 'Widget C'
AND a.order_date BETWEEN '2024-04-01' AND '2024-06-30'
AND c.order_date BETWEEN a.order_date AND DATE(a.order_date, '+30 days')
ORDER BY a.customer_id
The validator passes, the query executes, and the trace records the successful resolution:
{
"task": "Which customers bought Widget A in Q2 2024 and also bought Widget C within 30 days?",
"model_used": "strong",
"cost_estimate": 0.125,
"steps": [
{
"kind": "tool_call",
"content": {
"tool": "get_schema",
"result": {
"orders": {
"columns": ["order_id", "product_id", "quantity", "order_date", "customer_id"],
"sample": [
[1, 101, 2, "2024-04-15", 5001],
[2, 102, 1, "2024-05-20", 5001],
[3, 101, 5, "2024-06-10", 5002],
[4, 103, 10, "2024-05-05", 5004],
[5, 101, 1, "2024-06-25", 5003],
[6, 103, 1, "2024-06-28", 5003]
]
},
"products": {
"columns": ["product_id", "product_name", "unit_price"],
"sample": [
[101, "Widget A", 25.0],
[102, "Widget B", 40.0],
[103, "Widget C", 15.0]
]
}
}
}
},
{
"kind": "tool_call",
"content": {
"tool": "get_table_sample",
"result": {
"orders": [
{"order_id": 1, "product_id": 101, "quantity": 2, "order_date": "2024-04-15", "customer_id": 5001},
{"order_id": 2, "product_id": 102, "quantity": 1, "order_date": "2024-05-20", "customer_id": 5001},
{"order_id": 3, "product_id": 101, "quantity": 5, "order_date": "2024-06-10", "customer_id": 5002}
],
"products": [
{"product_id": 101, "product_name": "Widget A", "unit_price": 25.0},
{"product_id": 102, "product_name": "Widget B", "unit_price": 40.0},
{"product_id": 103, "product_name": "Widget C", "unit_price": 15.0}
]
}
}
},
{
"kind": "plan",
"content": {
"approach": "weak model first attempt",
"sql": "SELECT customer_id, product_name\nFROM customer_purchases\nWHERE product_name IN ('Widget A', 'Widget C')\nAND purchase_date BETWEEN '2024-04-01' AND '2024-06-30'\nGROUP BY customer_id\nHAVING COUNT(DISTINCT product_name) = 2"
}
},
{
"kind": "critique",
"content": {
"passed": false,
"feedback": "Unknown table: CUSTOMER_PURCHASES"
}
},
{
"kind": "routing",
"content": {
"decision": "strong",
"reason": "Weak model hallucinated a helper table; query requires self-join and date-window reasoning."
}
},
{
"kind": "plan",
"content": {
"approach": "strong model retry",
"sql": "SELECT DISTINCT a.customer_id, a.order_date AS widget_a_date, c.order_date AS widget_c_date\nFROM orders a\nJOIN orders c ON a.customer_id = c.customer_id\nJOIN products pa ON a.product_id = pa.product_id\nJOIN products pc ON c.product_id = pc.product_id\nWHERE pa.product_name = 'Widget A'\nAND pc.product_name = 'Widget C'\nAND a.order_date BETWEEN '2024-04-01' AND '2024-06-30'\nAND c.order_date BETWEEN a.order_date AND DATE(a.order_date, '+30 days')\nORDER BY a.customer_id"
}
},
{
"kind": "critique",
"content": {
"passed": true,
"feedback": "SQL passes read-only, schema, and shape checks."
}
},
{
"kind": "execute",
"content": {
"rows": [
{"customer_id": 5001, "widget_a_date": "2024-04-15", "widget_c_date": "2024-05-20"},
{"customer_id": 5003, "widget_a_date": "2024-06-25", "widget_c_date": "2024-06-28"}
],
"row_count": 2
}
},
{
"kind": "judge",
"content": {
"helpfulness": 5,
"correctness_confidence": 5,
"justification": "Query correctly reasons over temporal self-join."
}
},
{
"kind": "final",
"content": {
"status": "success",
"sql": "SELECT DISTINCT a.customer_id, a.order_date AS widget_a_date, c.order_date AS widget_c_date\nFROM orders a\nJOIN orders c ON a.customer_id = c.customer_id\nJOIN products pa ON a.product_id = pa.product_id\nJOIN products pc ON c.product_id = pc.product_id\nWHERE pa.product_name = 'Widget A'\nAND pc.product_name = 'Widget C'\nAND a.order_date BETWEEN '2024-04-01' AND '2024-06-30'\nAND c.order_date BETWEEN a.order_date AND DATE(a.order_date, '+30 days')\nORDER BY a.customer_id",
"result": {
"rows": [
{"customer_id": 5001, "widget_a_date": "2024-04-15", "widget_c_date": "2024-05-20"},
{"customer_id": 5003, "widget_a_date": "2024-06-25", "widget_c_date": "2024-06-28"}
],
"row_count": 2
}
}
}
]
}
The routing decision is itself a form of feedback-driven design. The weak model costs $0.055 per routine query; the strong model path costs $0.125 for this query. But the alternative — running every query on the strong model — is more expensive overall, and running every query on the weak model silently fails on complex ones. The loop discovers which tasks belong where.
3. Evaluation Is the Backbone, Not an Afterthought
In 2026, the evaluation stack for agents has become as important as the agent itself. The leading practice is to combine deterministic checks — for things like SQL shape, known tables, and read-only safety — with
LLM-as-a-judge for softer qualities like helpfulness, tone, and reasoning quality.
def validate_sql(sql: str, schema: dict) -> CriticResult:
# Read-only guard
if any(token in sql.upper() for token in ["DELETE", "UPDATE", "INSERT", "DROP"]):
return CriticResult(False, "Write operations are forbidden.")
# Known-table guard
for table in re.findall(r"FROM\s+(\w+)", sql.upper()):
if table.lower() not in schema:
return CriticResult(False, f"Unknown table: {table}")
return CriticResult(True, "SQL passes read-only and schema checks.")
These checks are cheap and exact. They should never be delegated to a judge model.
def llm_judge_score(task: str, output: str, judge_call) -> dict:
"""
Soft-quality scoring for things deterministic checks can't catch.
IMPORTANT: the judge is itself a probabilistic model and can be fooled
by fluent-but-wrong output — "who evaluates the evaluator?" Treat judge
scores as a signal to route for human review, not as ground truth, and
periodically audit judge verdicts against actual human ratings.
"""
The judge score is a signal, not a verdict. The same fluency that makes LLM output readable can make it convincing when wrong, which is exactly why silent errors are dangerous. Trace-based evaluation is the practical answer. A trace captures not just the final answer, but every tool call, reasoning step, plan revision, and handoff. Modern platforms like LangSmith and DeepEval use these traces to detect drift, spot new failure modes, and keep evaluations calibrated as the underlying model changes. Tracing is the backbone: it shows
where a metric failed and surfaces failure modes you do not yet have metrics for.
4. Failure Modes You Must Design For Upfront
Production agents fail in specific, expensive ways. The 2026 consensus is clear: decide how you will handle the following
before you give an agent real permissions.
| Failure mode | What it looks like | Why it is hard to catch |
|---|
| Hallucinated actions | The agent calls a tool based on a premise that isn’t true. | The tool call itself is well-formed; the error is in the model’s internal reasoning. |
| Runaway cost | Long or looping traces burn inference credits. | It does not produce a functional failure; it quietly raises the bill. |
| Tool misuse | The right tool used with the wrong arguments, or the wrong tool chosen. | The call may succeed and return data, but the data is irrelevant. |
| Context loss | Important information drops out of the context window mid-task. | The agent keeps responding fluently; only the trace shows it forgot the constraint. |
| Over-automation | The agent keeps going when it should have escalated to a human. | Success metrics look good until a boundary case explodes. |
| Silent errors | The output looks plausible but is factually wrong. | Fluency makes wrong output convincing — the very thing that makes LLMs useful also makes them dangerous. |
The “who evaluates the evaluator” problem in the
llm_judge_score comment is not an edge case; it is the reason you cannot rely on a single judge score. Production-ready agents handle these failure modes gracefully with governance, runtime guardrails, and defined escalation paths built into the design loop from the start.
5. The Real Cost
This new loop is powerful, but it is expensive. Every iteration consumes inference credits, engineering time, and human judgment. You are not just debugging code; you are probing the behavior of a model that changes underneath you. The design process becomes a continuous experiment: change a prompt, run a batch of tasks, inspect failures, adjust the architecture, and repeat.
A 2025 survey on
Evaluation and Benchmarking of LLM Agents notes that modern methods — repeated trials, human-in-the-loop assessment, and agent-as-a-judge techniques — are time- and resource-intensive, yet necessary to support iterative development (
arXiv:2507.21504). The survey also points out that most models still achieve low success rates on complex agent tasks, which is why evaluation cannot be a one-time benchmark.
Cost optimization is now a first-class concern. Teams cache prompts, compress context, route simpler tasks to smaller models, and use fine-tuned judge models to reduce evaluation spend. Yet some spending is unavoidable if you want reliability. In the SQL examples, making the agent schema-aware raised the routine-query cost from $0.025 to $0.055, and routing the complex query through a stronger model raised that path to $0.125. Both are small, measurable premiums for replacing hallucinations and capability gaps with verifiable chains.
6. Orchestration: Multi-Agent Systems Add Another Loop
Many 2026 agent systems are not single agents but orchestrated teams. LangGraph, CrewAI, AutoGen/AG2, OpenAI’s handoff SDK, and Google’s ADK represent different orchestration philosophies: directed graphs, role-based crews, conversational group chats, explicit handoffs, and hierarchical agent trees.
Each approach adds a layer of design uncertainty. Inter-agent schemas, state passing, and coordination protocols must be explicit — implicit data passing breaks when volume increases. The same build-evaluate-redesign loop applies, but at the team level: you must observe how agents interact, where handoffs fail, and whether the collective system is more reliable than the individual agents. This is the subject of the companion post on why multi-agent architectures need a meta-agent.
Conclusion
Feedback-driven design is not a refinement of traditional software engineering; it is a different shape of work. The teams that treat it as a first-class concern — building statistical evals, traces, guardrails, and human escalation into the loop from day one — will ship agents that recover from failure gracefully. The teams that do not will ship agents that look fine in demos and fail unpredictably in production. That gap is the moat. It is also the risk.
Random Instructions 
Leave a comment