# Observability and FinOps
What the observability layer must provide for agents: a record of every run that the agent cannot change, a tested gate on every change, and a budget with hard caps that a named sponsor owns.
Author: Murali Sid (https://linkedin.com/in/muralisid)
Source: https://www.agenticarchitectureskills.com/layers/r12-observability-and-finops
Published: 2026-08-20
Updated: 2026-10-15
Licence: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/
Cite as: Murali Sid. Observability and FinOps. Agentic Architecture Skills. https://www.agenticarchitectureskills.com/layers/r12-observability-and-finops
---
The observability and FinOps layer (R12) is one of [the fourteen layers](/layers) you already own. It shows what systems did and what they cost. FinOps is the practice of managing cloud and AI spend. Today this layer holds application monitoring, interface monitoring, service management and cloud cost tools. Almost none of it traces a model call or a tool call. For agents, this layer must provide three things. Every run leaves a step-by-step record that the agent cannot change. No change to an agent's behaviour reaches everyone before it is tested. And every agent has a budget with hard caps, owned by a named sponsor.
## What sits in this layer today [#what-sits-in-this-layer-today]
The research found a similar monitoring estate in all four typical organisations. It watches systems and messages. It does not yet watch what an agent reasoned, which tools it called, or what each case cost. Vendors in brackets are examples, not a claim about any named organisation.
| System category | Electricity distributor | Water utility | Hospital network | Generator, oil and gas, or miner |
| ------------------------------------- | ---------------------------------------------------------------------------- | ---------------------------------- | ---------------------------------------------- | ------------------------------------------------------ |
| Application monitoring | Dynatrace, Splunk or Datadog | Same as electricity | Dynatrace or AppDynamics | Dynatrace or Splunk |
| Infrastructure and network monitoring | Visibility of the OT network | Visibility of the OT network | SolarWinds or SCOM | An OT asset inventory. Fleet network health for miners |
| Interface monitoring | SAP Cloud ALM or MuleSoft Anypoint. Market message monitoring in the gateway | SAP Cloud ALM or MuleSoft Anypoint | Integration engine queues and error dashboards | Historian and interface health, watched by OT teams |
| IT service management | ServiceNow, run to ITIL | ServiceNow, run to ITIL | ServiceNow | ServiceNow |
| Cost and licence tools | Cloud cost tools | Cloud cost tools | Cloud cost tools and licence management | Cloud cost tools. OEM licence management for miners |
| Tracing of model and tool calls | Almost none | Almost none | None found | None yet |
Standards and interfaces: OpenTelemetry and the FinOps Framework in utilities. ITIL and service levels in vendor contracts in hospitals. OpenTelemetry and ITIL in energy and mining. The full estates are on the [utilities](/use-cases/utilities), [healthcare](/use-cases/healthcare) and [energy](/use-cases/energy) pages.
Interface monitoring shows whether a message moved. It does not show whether the business outcome happened. Tracing for agents will be needed outside each vendor's agent. That way the organisation holds its own evidence.
## What agents add [#what-agents-add]
Agents are managed as actors with costs, not as services with uptime. This layer adds four things. They are a record of every run, a budget per agent, a change gate and one view of spend.
### A record of every run [#a-record-of-every-run]
Every run leaves a trace. A trace is the step-by-step record of one run. It holds the session, each model call inside it, and each tool call. Tokens and cost are counted on every step. The whole trace is keyed to the agent's own identity. That key ties cost and audit evidence to one agent.
The trace is collected outside the agent's control. The agent cannot forge it or hide it. That is what makes it usable as evidence for [governance, risk and sovereignty (R11)](/layers/r11-governance-risk-sovereignty).
A trace that holds prompts and outputs is also personal data. So retention, redaction and residency rules apply to it. A trace takes the classification of the data it captured. Deleting a person's data must reach their traces too, as [agent data engineering (R14)](/layers/r14-agent-data-engineering) sets out.
One rule protects the stop. A monitoring outage must never block the kill switch. The stop path never depends on the measurement path.
### A budget for every agent [#a-budget-for-every-agent]
Every agent has an agent budget owned by its business sponsor. The budget has hard caps and alerts for unusual spend. Fixed rules enforce the caps, not the model. They apply at three enforcement points.
1. **The harness.** The harness is the engineering shell around the model. It sets hard caps per run. AgentCore caps iterations (default 75), time (default 3,600 seconds) and tokens, each with cost tags \[vendor].
2. **The gateway.** Budgets are set per key, user, team and organisation. When one runs out, the gateway refuses further calls with HTTP 429, "too many requests". LiteLLM is the common open-source pattern.
3. **The platform's billing policies.** Agent 365 ships agent billing policies with budget limits and threshold alerts (November 2025) \[vendor]. Agentforce Digital Wallet shows cost by agent, action and channel, at $0.10 per action \[vendor pricing]. Copilot Credits cost $0.01 each \[vendor pricing].
When a budget runs out, the work goes to a person's queue. It does not retry on its own. Set this behaviour before the agent is given any autonomy.
Manage cost per resolved outcome, not raw spend. Tokens per request doubled in a year at the median, and quadrupled for heavy users \[vendor]. So spend can rise while unit prices fall. Priced outcomes give an anchor. Automated resolutions are priced at $0.99 to $2.00 each \[vendor pricing]. A case handled by a person costs $6 to $12 for comparison.
The usual cost failures are loops and retry storms. About 60 percent of model call errors come from rate limits \[vendor]. A budget with a hard stop is the answer to both.
Showback comes first: each sponsor sees their agent's cost. Chargeback, billing the sponsor's unit, follows where costs split cleanly. The shared platform stays centrally funded.
### A gate on every change [#a-gate-on-every-change]
No change to an agent's behaviour reaches all traffic at once. A published pattern from a support agent with 100 million users has four steps (arXiv 2606.08867):
1. Offline evaluations on a test set with known right answers.
2. Shadow traffic, where the new version runs but its output is not used.
3. A canary on 1 to 5 percent of traffic. It is gated on session success, satisfaction and escalation rate.
4. Full release.
Feature flags separate switching a change on from deploying it. So rollback is one flag change, not a redeploy. A healthy escalation rate sits between 5 and 15 percent, tracked per release. Guardrail alerts work in both directions. A spike in blocks can mean an attack. A drop can mean a broken setting (Bedrock Guardrails metrics \[vendor]).
Most online evaluations use a model as the judge. Judges drift. They need their own checks against human labels. [Intelligence and learning (R06)](/layers/r06-intelligence-and-learning) and [the learning flywheel](/architecture/learning-flywheel) set out those checks. Traces from this layer feed both.
### One view of all agent spend [#one-view-of-all-agent-spend]
Agent costs arrive in two ways. Agents you build run through your gateway and are metered per use. Agents embedded in vendor products are licensed per seat and report through the vendor's console. The aim is one view of both.
The vendor side has gaps. Microsoft Graph Copilot usage reports became generally available in October 2025. They leave out unlicensed Copilot Chat use and lag by about 48 hours. Salesforce Digital Wallet reports in near real time. Anthropic ships an Admin Usage and Cost API and an Enterprise Analytics API \[vendor].
Splitting the cost of a shared model across agents is the hard problem. FOCUS, the FinOps Open Cost and Usage Specification, answers it with split-cost allocation in version 1.3.
### A buffer in front of the record format [#a-buffer-in-front-of-the-record-format]
The shared format for agent traces is still changing. OpenTelemetry's conventions for generative AI were all still in development status as of August 2026. In release v1.42.0 (12 June 2026) they moved to their own repository. That repository had no tagged release by mid-July 2026. Conventions for agent steps and the Model Context Protocol exist, and they are the ones to emit. OpenInference and OpenLLMetry still compete as ways to record traces.
So keep a translation step between recording and storage. Avoid deep ties to any vendor's software kit you may want to leave.
| Part | What it does | Where it runs |
| ------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| Recording and translation | Records the session, model calls and tool calls. Redacts and tags data at capture | Agent runtimes and harnesses, with a collector |
| Trace store | Holds traces as operations data, evidence and evaluation material | The existing monitoring estate, or a tool that also runs evaluations |
| Evaluation service | Runs offline and online evaluations, with judges checked against human labels | The build pipeline, plus samples from production |
| Release controller | Moves a change through shadow, canary and full release | The delivery pipeline and runtime flags |
| Budget enforcement | Hard caps per run, and budgets per key, team and organisation | The harness and the model gateway |
| Cost control | Metering, showback then chargeback, and alerts on unusual spend | Cloud cost tools and the FinOps practice |
| One cost view | Joins metered use with licensed seats | Vendor admin interfaces, feeding cost control |
## What must be true before an agent touches it [#what-must-be-true-before-an-agent-touches-it]
Before an agent touches a live system, these hold. Each is enforced outside the model. [Enforcement outside the model](/architecture/enforcement) shows the points.
* **Every run is traced, outside the agent.** The collector sits in the runtime and the gateway, not in the agent's own code. The audit trail is written where the agent cannot touch it.
* **The trace is keyed to the agent's identity.** [The delegation chain](/architecture/delegation-chain) links each action to the agent and to the person who asked.
* **The budget and its caps are set.** The sponsor owns the budget. The caps sit in the harness and the gateway. A spent budget sends work to a person. This is part of the operate stage of [the agent lifecycle](/architecture/agent-lifecycle).
* **The change gate is in place.** Evaluations pass before release. Every later change goes through shadow and canary. A listed change re-runs the evaluations, as [the lifecycle's change triggers](/architecture/agent-lifecycle#the-change-triggers) require.
* **Traces follow the data's rules.** Redaction runs at capture. Residency follows the source classification. Deletion reaches traces.
* **The stop does not depend on monitoring.** The kill switch works even when the trace pipeline is down.
## Choices and their verdicts [#choices-and-their-verdicts]
Each choice below is the common choice, tested against the simplest credible alternative. The full records are in [the decision catalog](/decisions).
* **Extend existing monitoring, or buy a specialist AI monitoring tool** ([decision record CD-15](/decisions#cd-15)). The question dissolved. The specialists were bought up. Choose on open standards, evaluation depth and self-hosting.
* **How to charge for each agent.** A sponsor-owned budget per agent, with showback first and chargeback where costs split cleanly. A fixed yearly cost per agent does not fit. Use swings too much.
## Concerns this layer must honour [#concerns-this-layer-must-honour]
This layer owns two of [the ten cross-cutting concerns](/architecture/concern-matrix) and carries duties for four more.
* **Observability (owned here).** Every run is traced, and trace coverage is measured.
* **Cost (owned here).** Budgets, caps and cost per resolved outcome, with showback then chargeback.
* **Traceability and audit.** Traces are evidence, collected outside the agent's control.
* **Privacy.** Prompts and outputs are redacted in traces. Deletion reaches telemetry.
* **Sovereignty and residency.** Traces stay where their source data may stay.
* **Resilience.** A spent budget sends work to a person. A monitoring outage never blinds the kill switch.
## Evidence and limits [#evidence-and-limits]
The record of cost incidents is thin. Each one is graded for confidence. A bill of $1.3 million for 603 billion tokens in 30 days (May 2026) came from a deliberate fleet. A three-person team ran about 100 coding-agent instances, mostly on premium pricing. The team reported the figure itself, so confidence is medium. A widely shared $47,000 loop between two agents rests on one first-person account. Its weekly figures do not add up, so confidence is low.
Survey and telemetry figures come from vendors (LangChain, Harness, Datadog) and are marked. The Harness cost figures are paired with the FinOps Foundation's State of FinOps data. Every price on this page is a vendor list price.
The sponsor-owned agent budget is a recommended design. No enterprise had published per-agent budgets running in production as of August 2026. No published enterprise example yet sets an error budget for agent quality.
## Next step [#next-step]
See this layer at work in [investigating production discrepancies](/use-cases/production-measurement). The weekly checks that read these measures are on [Operate](/operate). Reading the layers in order? The next one is [operating model and supervision (R13)](/layers/r13-operating-model).
Explore this topic
* [Observability and FinOps findings](/library/layers/r12-observability-and-finops/findings)
* [Sources](/library/layers/r12-observability-and-finops/sources)
* [Products named for orientation](/architecture), on the one-page wall chart
**Cost and survey figures.** In the Harness 2026 survey, 52 percent report no clear AI cost owner \[vendor survey]. The same survey estimates that 26 percent of AI spend is wasted \[vendor survey]. 72 percent saw an unexpected spike in the past year \[vendor survey]. 98 percent of FinOps practitioners now manage AI spend, up from 63 percent a year earlier (State of FinOps 2026). 90 percent of FinOps teams also manage software-as-a-service spend. The rate-limit figure comes from Datadog telemetry \[vendor]. The token growth figure is vendor telemetry \[vendor].
**Consolidation behind decision record CD-15.** Dynatrace agreed to acquire Arize for $915 million (announced 13 August 2026, not closed at research time). ClickHouse acquired Langfuse (January 2026) and kept its MIT licence. Cisco folded Galileo into Splunk (May 2026). CoreWeave took Weights & Biases (2025). LangSmith and Braintrust remain independent. Most enterprises land on their existing monitoring tool for correlation, plus one tool for evaluations. OpenAI bought Statsig, a flags and experimentation company, for $1.1 billion (September 2025).
**The sponsor role in products.** Identity platforms carry a sponsor role with automatic transfer when the sponsor leaves. Workforce systems took per-agent budgeting to general availability in February 2026, with thin mechanics and no named customer. Control platforms ship billing policies. The Tokenomics Foundation works with FOCUS on normalising token costs. Agents-console products bring third-party agents into the same cost view \[vendor].
**Refusals.** A spring 2026 funding round attributed to the open-source tracing project circulates on content-farm sites. It contradicts the verified January 2026 acquisition. It is excluded. That class of aggregator source is treated as unreliable.
**Re-verify each quarter.** The most important item is the first tagged release, and the stability status, of the semantic-conventions-genai repository. Also check the close of the Dynatrace and Arize deal, and the fate of the Phoenix open-source project. Check how much per-agent budgeting workforce systems have really shipped. Check the general availability of agents-console products.