Operate: the weekly loop

An agent that passed its tests can still drift, overspend or swamp the people behind it. So once an agent is live, what do you check, and how often? Check four things every week. What did the agent do? What did each finished piece of work cost? Can the people handling its exceptions keep up? Should it change, or stop? This is step 6 on Start here. It covers the operate, change, periodic review and retire stages of the agent lifecycle. Judge oversight by the busiest ten minutes of the week, not by the daily average.

The four weekly checks

1. Observability: what the agent did

Every run leaves a traceTrace. The step-by-step log of one agent run: what it read, what it called, and what it decided.Read the page. It records the session, each model call, each tool call and the cost. The trace is collected outside the agent. The agent cannot forge or hide it. This is the observability and audit trail capability at work. Check that every run has a complete trace. Check costs against the budget. Review each incident and its cause. Watch guardrail block rates in both directions. A guardrailGuardrail. A filter that uses an AI model to catch bad inputs or outputs. Useful as an early warning, but not reliable enough to be the thing that grants or denies an action.Read the page is a fixed check that stops an unsafe action. A spike can mean an attack. A drop can mean a broken setting. The observability and FinOps layer (R12) sets out the trace, the budgets and the change gate.

2. Cost: what each outcome cost

Track cost per resolved outcomeCost per resolved outcome. What it costs to actually solve one case, including supervision time and the cost of wrong answers, rather than what one run costs.Read the page, not raw spend. It counts model, tool and platform cost, including failed attempts. It adds supervision time and the cost of fixing wrong results. Then it divides by the outcomes the business accepted as resolved. Each agent has an agent budgetAgent budget. A spending allowance for an agent with hard caps per run and alarms, owned by its business sponsor.Read the page with hard caps per run. Its sponsor owns that budget. Circuit breakers stop spending at set limits. When a budget runs out, the work goes to a person. It does not retry on its own. Measure value and full cost gives the formula. Token efficiency covers where the tokens go and what to meter for each agent.

3. Supervision: whether people keep up

Exceptions land in a queue for a person to decide. Capacity depends on how long items wait, not on how many agents there are. Span of controlSpan of control. How many agents or tasks one person can supervise at once. It shrinks as waiting times grow.Read the page is how many agents one person can supervise. It shrinks as waiting times grow. Measure wait time per item in the busiest ten minutes, never as a daily average. That peak figure is the burst rateBurst rate. Demand measured at its busiest moments rather than averaged over a day. Oversight capacity is planned against bursts.Read the page. Track the intervention rate and its trend. A falling rate can mean tired reviewers rather than a better agent. Split escalations by who raised them: a person or the automated monitor. Run blind checks as well. People misjudge their own speed and accuracy. The operating model and supervision layer (R13) sets out the queueing calculation.

4. Lifecycle: whether the agent should change or stop

Eight kinds of change make an approval out of date, from a new model to a new sponsor. Each one re-runs the evals and the risk profile before release. Check the week's change triggersChange trigger. A change that forces an agent to be re-evaluated and re-profiled before release: a new model, prompt, tool, permission, data source, memory policy, autonomy level or sponsor.Read the page against the registryAgent registry. The official list of agents: what each is for, who owns it, what it may touch, and which version is live.Read the page record. Check which periodic reviews are due. Each quarter, the sponsor confirms purpose, access, budget and need. Retire agents that are no longer needed. Retiring means revoking access and removing memory, not just switching off. The agent lifecycle sets out the triggers, the review and the retirement steps.

One week with a billing-exception agent

Take a water utility's billing-exception agent. It drafts fixes for meter reads that fail validation. A billing officer approves each fix. This week, every run has a full trace. Cost per resolved exception rose. The agent re-read more meters than usual. Wait time broke its limit on Monday morning, when a batch of failed reads arrived at once. The daily average looked fine. A model update was logged as a change trigger, and the evals ran again before release. The agent stays at its current autonomy level until Monday's peak is handled. This is an illustrative week, not a measured deployment.

What the weekly review reads

The gate to step 7

Step 7 lets a proven agent do more. Before that, oversight capacity must hold at burst. That means the busiest ten minutes, not an average day. A daily average can look fine while people are swamped in every burst. Alarm standards for control rooms use per-ten-minute rates for this reason. This guide asks for three things from production. Wait time stays inside its limit at peak. The intervention rate is not falling as autonomy grows. Blind checks agree with what reviewers report. Until all three hold, the agent stays at its current autonomy levelAutonomy level (A0 to A5). How much an agent may do without a person in the loop, from manual (A0) to governed lights-out operation (A5). Set per task, not per agent.Read the page. The autonomy contract sets out the calculation.

Who runs the loop

The programme lead owns step 6 and runs the weekly review. The agent sponsor answers for purpose, results and budget. The agent owner runs the configuration, evals and incidents. The FinOps and agent economics lead reads the cost lines. FinOps is the practice of managing cloud and agent spend. Exception supervisors work the queues as part of their jobs.

These roles sit inside a wider design for how the teams are organised. A thin office writes the rules and holds the money. One platform team turns the rules into automatic checks. Three small lanes build each new pattern once, then step back. Business teams own their agents. This guide calls that design One Platform, Three Lanes. How to organise the teams sets it out.

Evidence and limits

The span of control relation and the alarm-standard bands are peer-reviewed or published standards. Both come from supervision work outside AI agents. No credible published figure exists for how many agents one person can supervise. Measure your own queues. The quarterly review follows AWS and Microsoft guidance. Both ask for quarterly reviews of live agents. The billing-exception week above is an illustration.

Next step: before the agent does more, read the autonomy contract. To see who does this work, read how to organise the teams.

Explore this topic

• Operating model findings
• The economics model
• Observability and FinOps findings
• AWS Well-Architected agentic AI lens, AGENTSUS03-BP04, undated, read 15 October 2026. It names quarterly portfolio reviews.
• Microsoft Cloud Adoption Framework, Manage AI agents across your organization, updated 26 June 2026. It asks for quarterly reviews to retire agents that no longer deliver value.