Observability and FinOps

What the observability layer must provide for agents: a record of every run that the agent cannot change, a tested gate on every change, and a budget with hard caps that a named sponsor owns.

The observability and FinOps layer (R12) is one of the fourteen layers you already own. It shows what systems did and what they cost. FinOps is the practice of managing cloud and AI spend. Today this layer holds application monitoring, interface monitoring, service management and cloud cost tools. Almost none of it traces a model call or a tool call. For agents, this layer must provide three things. Every run leaves a step-by-step record that the agent cannot change. No change to an agent's behaviour reaches everyone before it is tested. And every agent has a budget with hard caps, owned by a named sponsor.

What sits in this layer today

The research found a similar monitoring estate in all four typical organisations. It watches systems and messages. It does not yet watch what an agent reasoned, which tools it called, or what each case cost. Vendors in brackets are examples, not a claim about any named organisation.

System category	Electricity distributor	Water utility	Hospital network	Generator, oil and gas, or miner
Application monitoring	Dynatrace, Splunk or Datadog	Same as electricity	Dynatrace or AppDynamics	Dynatrace or Splunk
Infrastructure and network monitoring	Visibility of the OT network	Visibility of the OT network	SolarWinds or SCOM	An OT asset inventory. Fleet network health for miners
Interface monitoring	SAP Cloud ALM or MuleSoft Anypoint. Market message monitoring in the gateway	SAP Cloud ALM or MuleSoft Anypoint	Integration engine queues and error dashboards	Historian and interface health, watched by OT teams
IT service management	ServiceNow, run to ITIL	ServiceNow, run to ITIL	ServiceNow	ServiceNow
Cost and licence tools	Cloud cost tools	Cloud cost tools	Cloud cost tools and licence management	Cloud cost tools. OEM licence management for miners
Tracing of model and tool calls	Almost none	Almost none	None found	None yet

System category: Application monitoring
Electricity distributor: Dynatrace, Splunk or Datadog
Water utility: Same as electricity
Hospital network: Dynatrace or AppDynamics
Generator, oil and gas, or miner: Dynatrace or Splunk

System category: Infrastructure and network monitoring
Electricity distributor: Visibility of the OT network
Water utility: Visibility of the OT network
Hospital network: SolarWinds or SCOM
Generator, oil and gas, or miner: An OT asset inventory. Fleet network health for miners

System category: Interface monitoring
Electricity distributor: SAP Cloud ALM or MuleSoft Anypoint. Market message monitoring in the gateway
Water utility: SAP Cloud ALM or MuleSoft Anypoint
Hospital network: Integration engine queues and error dashboards
Generator, oil and gas, or miner: Historian and interface health, watched by OT teams

System category: IT service management
Electricity distributor: ServiceNow, run to ITIL
Water utility: ServiceNow, run to ITIL
Hospital network: ServiceNow
Generator, oil and gas, or miner: ServiceNow

System category: Cost and licence tools
Electricity distributor: Cloud cost tools
Water utility: Cloud cost tools
Hospital network: Cloud cost tools and licence management
Generator, oil and gas, or miner: Cloud cost tools. OEM licence management for miners

System category: Tracing of model and tool calls
Electricity distributor: Almost none
Water utility: Almost none
Hospital network: None found
Generator, oil and gas, or miner: None yet

Standards and interfaces: OpenTelemetry and the FinOps Framework in utilities. ITIL and service levels in vendor contracts in hospitals. OpenTelemetry and ITIL in energy and mining. The full estates are on the utilities, healthcare and energy pages.

Interface monitoring shows whether a message moved. It does not show whether the business outcome happened. Tracing for agents will be needed outside each vendor's agent. That way the organisation holds its own evidence.

What agents add

Agents are managed as actors with costs, not as services with uptime. This layer adds four things. They are a record of every run, a budget per agent, a change gate and one view of spend.

layer · architecture

Traces the agent cannot forge

Every agent is traced end to end on a translation layer over unstable conventions; budgets are enforced deterministically, and telemetry collection sits outside the agent's control.

01Agent
Agent
Agent execution
02Control
Control
Out-of-band collection
Agents cannot forge their own traces
03System
System
Translation layer
Conventions unstable; avoid deep SDK coupling
04Evidence
Evidence
Traces + cost
Session, model, tool spans keyed to agent identity
05Control
Control
Budget enforcement
Hierarchical caps; 429 on exhaustion; fail to human queues
06Control
Control
Change gates
Shadow, 1-5% canary, full; judges calibrated

An observability outage must not blind the kill switch.

Session, model-call, and tool-call spans keyed to agent identity and cost.

A record of every run

Every run leaves a trace. A trace is the step-by-step record of one run. It holds the session, each model call inside it, and each tool call. Tokens and cost are counted on every step. The whole trace is keyed to the agent's own identity. That key ties cost and audit evidence to one agent.

The trace is collected outside the agent's control. The agent cannot forge it or hide it. That is what makes it usable as evidence for governance, risk and sovereignty (R11).

A trace that holds prompts and outputs is also personal data. So retention, redaction and residency rules apply to it. A trace takes the classification of the data it captured. Deleting a person's data must reach their traces too, as agent data engineering (R14) sets out.

One rule protects the stop. A monitoring outage must never block the kill switch. The stop path never depends on the measurement path.

A complete agent run goes from user request through agent step, model call, tool call and decision to outcome. It writes a tamper-resistant trace that the agent cannot alter. Budgets apply per run, per agent and per team. When a budget is reached, the run stops and the work goes to a human queue. The measure to manage is cost per solved case. — **Every run leaves evidence and stays inside budget**The trace covers the whole run and sits outside the agent's control. Budgets are enforced outside the model at several levels, and cost is measured per solved case.

A budget for every agent

Every agent has an agent budget owned by its business sponsor. The budget has hard caps and alerts for unusual spend. Fixed rules enforce the caps, not the model. They apply at three enforcement points.

The harness. The harness is the engineering shell around the model. It sets hard caps per run. AgentCore caps iterations (default 75), time (default 3,600 seconds) and tokens, each with cost tags [vendor].
The gateway. Budgets are set per key, user, team and organisation. When one runs out, the gateway refuses further calls with HTTP 429, "too many requests". LiteLLM is the common open-source pattern.
The platform's billing policies. Agent 365 ships agent billing policies with budget limits and threshold alerts (November 2025) [vendor]. Agentforce Digital Wallet shows cost by agent, action and channel, at $0.10 per action [vendor pricing]. Copilot Credits cost $0.01 each [vendor pricing].

When a budget runs out, the work goes to a person's queue. It does not retry on its own. Set this behaviour before the agent is given any autonomy.

Manage cost per resolved outcome, not raw spend. Tokens per request doubled in a year at the median, and quadrupled for heavy users [vendor]. So spend can rise while unit prices fall. Priced outcomes give an anchor. Automated resolutions are priced at $0.99 to $2.00 each [vendor pricing]. A case handled by a person costs $6 to $12 for comparison.

The usual cost failures are loops and retry storms. About 60 percent of model call errors come from rate limits [vendor]. A budget with a hard stop is the answer to both.

Showback comes first: each sponsor sees their agent's cost. Chargeback, billing the sponsor's unit, follows where costs split cleanly. The shared platform stays centrally funded.

A gate on every change

No change to an agent's behaviour reaches all traffic at once. A published pattern from a support agent with 100 million users has four steps (arXiv 2606.08867):

Offline evaluations on a test set with known right answers.
Shadow traffic, where the new version runs but its output is not used.
A canary on 1 to 5 percent of traffic. It is gated on session success, satisfaction and escalation rate.
Full release.

Feature flags separate switching a change on from deploying it. So rollback is one flag change, not a redeploy. A healthy escalation rate sits between 5 and 15 percent, tracked per release. Guardrail alerts work in both directions. A spike in blocks can mean an attack. A drop can mean a broken setting (Bedrock Guardrails metrics [vendor]).

Most online evaluations use a model as the judge. Judges drift. They need their own checks against human labels. Intelligence and learning (R06) and the learning flywheel set out those checks. Traces from this layer feed both.

A behaviour change moves through offline evaluation, shadow traffic, a small canary and wider release. Gates check quality, safety, cost and a comparison with people. A model judge is calibrated against humans. When a regression is detected, the change rolls back. — **No behaviour change goes straight to every user**Offline tests start the release. Shadow and canary steps catch failures that only show in production. Any regression returns the agent to the last proven version.

One view of all agent spend

Agent costs arrive in two ways. Agents you build run through your gateway and are metered per use. Agents embedded in vendor products are licensed per seat and report through the vendor's console. The aim is one view of both.

The vendor side has gaps. Microsoft Graph Copilot usage reports became generally available in October 2025. They leave out unlicensed Copilot Chat use and lag by about 48 hours. Salesforce Digital Wallet reports in near real time. Anthropic ships an Admin Usage and Cost API and an Enterprise Analytics API [vendor].

Splitting the cost of a shared model across agents is the hard problem. FOCUS, the FinOps Open Cost and Usage Specification, answers it with split-cost allocation in version 1.3.

A buffer in front of the record format

The shared format for agent traces is still changing. OpenTelemetry's conventions for generative AI were all still in development status as of August 2026. In release v1.42.0 (12 June 2026) they moved to their own repository. That repository had no tagged release by mid-July 2026. Conventions for agent steps and the Model Context Protocol exist, and they are the ones to emit. OpenInference and OpenLLMetry still compete as ways to record traces.

So keep a translation step between recording and storage. Avoid deep ties to any vendor's software kit you may want to leave.

Part	What it does	Where it runs
Recording and translation	Records the session, model calls and tool calls. Redacts and tags data at capture	Agent runtimes and harnesses, with a collector
Trace store	Holds traces as operations data, evidence and evaluation material	The existing monitoring estate, or a tool that also runs evaluations
Evaluation service	Runs offline and online evaluations, with judges checked against human labels	The build pipeline, plus samples from production
Release controller	Moves a change through shadow, canary and full release	The delivery pipeline and runtime flags
Budget enforcement	Hard caps per run, and budgets per key, team and organisation	The harness and the model gateway
Cost control	Metering, showback then chargeback, and alerts on unusual spend	Cloud cost tools and the FinOps practice
One cost view	Joins metered use with licensed seats	Vendor admin interfaces, feeding cost control

Part: Recording and translation
What it does: Records the session, model calls and tool calls. Redacts and tags data at capture
Where it runs: Agent runtimes and harnesses, with a collector

Part: Trace store
What it does: Holds traces as operations data, evidence and evaluation material
Where it runs: The existing monitoring estate, or a tool that also runs evaluations

Part: Evaluation service
What it does: Runs offline and online evaluations, with judges checked against human labels
Where it runs: The build pipeline, plus samples from production

Part: Release controller
What it does: Moves a change through shadow, canary and full release
Where it runs: The delivery pipeline and runtime flags

Part: Budget enforcement
What it does: Hard caps per run, and budgets per key, team and organisation
Where it runs: The harness and the model gateway

Part: Cost control
What it does: Metering, showback then chargeback, and alerts on unusual spend
Where it runs: Cloud cost tools and the FinOps practice

Part: One cost view
What it does: Joins metered use with licensed seats
Where it runs: Vendor admin interfaces, feeding cost control

What must be true before an agent touches it

Before an agent touches a live system, these hold. Each is enforced outside the model. Enforcement outside the model shows the points.

Every run is traced, outside the agent. The collector sits in the runtime and the gateway, not in the agent's own code. The audit trail is written where the agent cannot touch it.
The trace is keyed to the agent's identity. The delegation chain links each action to the agent and to the person who asked.
The budget and its caps are set. The sponsor owns the budget. The caps sit in the harness and the gateway. A spent budget sends work to a person. This is part of the operate stage of the agent lifecycle.
The change gate is in place. Evaluations pass before release. Every later change goes through shadow and canary. A listed change re-runs the evaluations, as the lifecycle's change triggers require.
Traces follow the data's rules. Redaction runs at capture. Residency follows the source classification. Deletion reaches traces.
The stop does not depend on monitoring. The kill switch works even when the trace pipeline is down.

Choices and their verdicts

Each choice below is the common choice, tested against the simplest credible alternative. The full records are in the decision catalog.

Extend existing monitoring, or buy a specialist AI monitoring tool (decision record CD-15). The question dissolved. The specialists were bought up. Choose on open standards, evaluation depth and self-hosting.
How to charge for each agent. A sponsor-owned budget per agent, with showback first and chargeback where costs split cleanly. A fixed yearly cost per agent does not fit. Use swings too much.

Concerns this layer must honour

This layer owns two of the ten cross-cutting concerns and carries duties for four more.

Observability (owned here). Every run is traced, and trace coverage is measured.
Cost (owned here). Budgets, caps and cost per resolved outcome, with showback then chargeback.
Traceability and audit. Traces are evidence, collected outside the agent's control.
Privacy. Prompts and outputs are redacted in traces. Deletion reaches telemetry.
Sovereignty and residency. Traces stay where their source data may stay.
Resilience. A spent budget sends work to a person. A monitoring outage never blinds the kill switch.

Evidence and limits

The record of cost incidents is thin. Each one is graded for confidence. A bill of $1.3 million for 603 billion tokens in 30 days (May 2026) came from a deliberate fleet. A three-person team ran about 100 coding-agent instances, mostly on premium pricing. The team reported the figure itself, so confidence is medium. A widely shared $47,000 loop between two agents rests on one first-person account. Its weekly figures do not add up, so confidence is low.

Survey and telemetry figures come from vendors (LangChain, Harness, Datadog) and are marked. The Harness cost figures are paired with the FinOps Foundation's State of FinOps data. Every price on this page is a vendor list price.

The sponsor-owned agent budget is a recommended design. No enterprise had published per-agent budgets running in production as of August 2026. No published enterprise example yet sets an error budget for agent quality.

Next step

See this layer at work in investigating production discrepancies. The weekly checks that read these measures are on Operate. Reading the layers in order? The next one is operating model and supervision (R13).

Explore this topic

Observability and FinOps findings
Sources
Products named for orientation, on the one-page wall chart

Cost and survey figures. In the Harness 2026 survey, 52 percent report no clear AI cost owner [vendor survey]. The same survey estimates that 26 percent of AI spend is wasted [vendor survey]. 72 percent saw an unexpected spike in the past year [vendor survey]. 98 percent of FinOps practitioners now manage AI spend, up from 63 percent a year earlier (State of FinOps 2026). 90 percent of FinOps teams also manage software-as-a-service spend. The rate-limit figure comes from Datadog telemetry [vendor]. The token growth figure is vendor telemetry [vendor].

Consolidation behind decision record CD-15. Dynatrace agreed to acquire Arize for $915 million (announced 13 August 2026, not closed at research time). ClickHouse acquired Langfuse (January 2026) and kept its MIT licence. Cisco folded Galileo into Splunk (May 2026). CoreWeave took Weights & Biases (2025). LangSmith and Braintrust remain independent. Most enterprises land on their existing monitoring tool for correlation, plus one tool for evaluations. OpenAI bought Statsig, a flags and experimentation company, for $1.1 billion (September 2025).

The sponsor role in products. Identity platforms carry a sponsor role with automatic transfer when the sponsor leaves. Workforce systems took per-agent budgeting to general availability in February 2026, with thin mechanics and no named customer. Control platforms ship billing policies. The Tokenomics Foundation works with FOCUS on normalising token costs. Agents-console products bring third-party agents into the same cost view [vendor].

Refusals. A spring 2026 funding round attributed to the open-source tracing project circulates on content-farm sites. It contradicts the verified January 2026 acquisition. It is excluded. That class of aggregator source is treated as unreliable.

Re-verify each quarter. The most important item is the first tagged release, and the stability status, of the semantic-conventions-genai repository. Also check the close of the Dynatrace and Arize deal, and the fate of the Phoenix open-source project. Check how much per-agent budgeting workforce systems have really shipped. Check the general availability of agents-console products.