Where Agents Fail and How to Build Reliable AI Workflows

A demo of an AI agent is a performance in which everything goes right. A deployed agent is a system that runs thousands of times on messy inputs, with nobody watching most of them. The gap between those two is where most of the engineering lives, and where most disappointments come from.

This article is about closing that gap. You do not need to build agents to benefit: the same thinking tells you how far to trust a workflow someone else built, or an assistant that offers to "handle it for you".

• The common ways agents and workflows fail.
• Design practices that make them more reliable.
• How to build effective human-in-the-loop checkpoints, with a runnable example.
• How to evaluate and roll out safely.

Knowing the shapes of failure helps you design against them and notice them early.

Several of these are the same issues from earlier articles, now with actions attached: invented content is more serious when the output is not text but a transaction.

These practices apply whether you are building a workflow with one AI step or a full agent. They echo advice from the model makers' own documentation: Anthropic's guide to building agents recommends starting simple and adding autonomy only when needed. Its prompt engineering overview assumes you have clear success criteria and ways to test against them. OpenAI's prompting guide stresses building tests and evaluation suites to monitor behaviour across versions.

Write down what "correct" means, in terms you can check. "Summarise tickets well" is vague. "Counts match the database exactly; each theme is supported by at least three tickets; no customer names appear" is checkable. If you cannot define it, you cannot test it, and you cannot tell whether a change made things better.

Give the system the smallest job and the smallest set of tools that can do it. Narrow systems fail less, fail more visibly and are easier to test.

Do not use an agent to decide an order that you already know. Use code for the sequence and the model for the bits that need language.

Check each model output before the next step uses it: shape, types, allowed values, ranges, required fields. Check each tool result too. See the validator in article 8. Use ordinary code for exact things such as sums, dates and lookups.

A failed step can be retried once or twice, perhaps with the error message included. Then the item goes to a person. Infinite retries and silent failure are both bad.

• Reversible where possible: drafts instead of sent messages, a trash folder instead of deletion, a branch instead of the main line.
• Idempotent: running a step twice does not double the effect (no double payments).
• Dry-run first: show what would happen before it does.
• Capped: limits on number of items touched, amount of money, number of steps and time.

Record the inputs, each decision, each tool call, each result and the final outcome. Logs make failures diagnosable and audits possible. Avoid logging sensitive data you do not need.

Someone must be able to pause or kill the system quickly, and know how.

"A human reviews it" is the most common safety claim and the most often hollow one. It works only if the human can actually catch errors. For that, the human needs:

• Time. Enough minutes per item, at a realistic volume.
• Evidence. The inputs, the reason for the proposal and the source of each claim, not just the output.
• Authority. Genuine ability and permission to reject, edit or escalate, without pressure to approve.
• A good interface. Differences highlighted, risky items first, one-click reject with a reason.
• Feedback. Their corrections feed back into improving the system.

OWASP's guidance on prompt injection, which ranks it first among risks for LLM applications, lists human approval for high-risk operations and least-privilege access among its recommended mitigations, which is consistent with this approach.

A key design rule is default-deny: if nobody approves, nothing happens. The opposite, "send unless someone objects", turns silence into consent.

The program below simulates a small AI-assisted support workflow. draft_reply stands in for a language model, with fixed outputs so that results are repeatable. For each ticket, automatic checks run first. Anything that fails the checks is escalated. Anything that passes still needs a human to approve it. If no decision exists, the default is to hold. An audit log records what happened.

# File: approval_gate.py# A small AI-assisted workflow with checks and a human approval gate.# draft_reply stands in for a language model so the output is repeatable.AUDIT_LOG = []def draft_reply(ticket):    drafts = {        1: "Hi Meena, your refund of Rs 499 has been issued and will arrive in 5 to 7 days.",        2: "Hi Dev, we guarantee a full refund of Rs 25000 today.",        3: "Hi Isha, thanks for writing. Our team will reply by Friday.",    }    ticket_id = ticket["id"]    return drafts[ticket_id]def automatic_checks(ticket, draft):    problems = []    if "guarantee" in draft.lower():        problems.append("contains a promise ('guarantee')")    if ticket["refund_amount"] > 1000:        problems.append("refund above the 1000 limit needs a person")    return problemsdef process(ticket, human_decisions):    draft = draft_reply(ticket)    problems = automatic_checks(ticket, draft)    if problems:        AUDIT_LOG.append((ticket["id"], "escalated", "; ".join(problems)))        return    decision = human_decisions.get(ticket["id"], "reject")  # no answer means no send    if decision == "approve":        AUDIT_LOG.append((ticket["id"], "sent", draft))    else:        AUDIT_LOG.append((ticket["id"], "held", "reviewer did not approve"))tickets = [    {"id": 1, "refund_amount": 499},    {"id": 2, "refund_amount": 25000},    {"id": 3, "refund_amount": 0},]reviewer = {1: "approve"}  # the reviewer approved ticket 1 onlyfor ticket in tickets:    process(ticket, reviewer)for entry in AUDIT_LOG:    print(entry)

(1, 'sent', 'Hi Meena, your refund of Rs 499 has been issued and will arrive in 5 to 7 days.')(2, 'escalated', "contains a promise ('guarantee'); refund above the 1000 limit needs a person")(3, 'held', 'reviewer did not approve')

Read it as three different outcomes:

• Ticket 1: passed the checks, the reviewer approved, so it was sent.
• Ticket 2: the draft makes a promise and the amount exceeds a limit. It never reaches the approval step: it is escalated to a person who can handle it properly. The checks are ordinary code, not the model's judgment.
• Ticket 3: passed the checks, but no reviewer decision exists, so by default it is held. Nothing is sent without an explicit yes.

Everything is logged. This is not sophisticated, and that is the point: much of reliability is plain, boring code around the AI step.

Systems need to know when not to act. Be careful here: a model's stated confidence ("I'm 95 percent sure") is not a reliable measure. Better signals:

• Validation results (does the output meet the rules?).
• Agreement across runs or methods, where disagreement suggests uncertainty.
• Rule triggers (amount over a limit, sensitive keywords, unfamiliar category).
• Missing information (required fields blank).
• An explicit "cannot determine" option in the output format, with a route to a person.

You do not know a workflow is reliable until you have measured it.

1. Collect real examples, 20 to 50 to start, covering normal cases, edge cases and cases that should be refused or escalated.
2. Write the expected outcome for each, using your success definition.
3. Run the system on all of them and compare. Record failures by type (misclassified, invented detail, wrong action, should have escalated).
4. Change one thing at a time, such as the prompt, the model or a rule, and rerun the set.
5. Rerun after model or tool updates, since behaviour can shift.
6. Keep adding hard cases from production failures.
7. Check a sample of live outputs against reality regularly, because test sets age.

This is the discipline the model makers' own guidance asks for. A small, honest evaluation set beats an impressive demo.

Do not go from zero to autonomous. A cautious path:

Move to the next stage only when the data supports it, and be ready to move back.

Goal. Help the finance team process employee expense claims.

AI handles the two language-heavy steps, code handles everything exact, and a human decides on money. If extraction accuracy proves high over months, finance might auto-approve very small, clean claims, with sampling.

• Success is defined in checkable terms.
• Scope and tools are as narrow as possible.
• Outputs are validated at every step. Exact work uses code.
• Retries are limited, and failures escalate to a person.
• Risky actions need approval, default-deny.
• Actions are reversible, idempotent or capped.
• Untrusted content is treated as data, not instructions.
• Everything is logged, and logs avoid unnecessary sensitive data.
• There is an evaluation set, and I know the error types.
• There is a stop button, and an owner.
• The rollout is staged, with monitoring.

• Launching from a successful demo.
• Using "human in the loop" as a slogan without time, evidence or authority for the human.
• Letting the model grade its own output as the only check.
• Retrying forever, or failing silently.
• Treating silence as approval.
• No evaluation set, so changes cannot be judged.
• Skipping logs.
• Automating the high-risk cases first.

• Next: Design Your Personal AI Operating System begins Module 9, where you bring the course together.
• For the risks of connecting tools, see Privacy and Security: What Not to Paste, Connected Tools and Prompt Injection.
• For structured outputs and validators, see Examples, Output Formats and Structured Output.