Intelligence and learning

Target state

In short: Agents improve through a tested, reversible process, and nothing changes in production until it has passed offline checks.

Learning is an operating discipline, not a one-off training activity. An eval suiteEval (evaluation suite). A set of test tasks with known right answers, written by people who know the work, used to check an agent before and after any change. (a set of test tasks with known right answers) defines the bar. Domain experts own it, and it is built from real failures. Model judgesJudge (model judge). An AI model used to grade another model's answers at scale. Checked regularly against human graders, and never allowed to grade its own work. grade outcomes at volume, and people verify a sample of those grades. Production tracesTrace. The step-by-step log of one agent run: what it read, what it called, and what it decided. (the step-by-step records of agent runs) and eval datasets share one data layer. When a live run scores badly, its trace moves into the offline suite automatically. A candidate rule is promotedPromotion and demotion. Promotion moves a proven improvement into production. Demotion pulls it back out when it stops passing. only if it survives its counterexamplesCounterexample. A test case built from a real failure, used to prove that a proposed rule or change actually holds. and does not regress the eval suite. Promoted rules land in a policy-as-codePolicy-as-code. Rules written as versioned code that a computer evaluates the same way every time, reviewed and deployed like any software change. tier outside the model, and every one carries a demotion path. DistillationDistillation. Teaching a smaller, cheaper model to imitate a larger one on a narrow task once enough good examples exist. (teaching a smaller model to imitate a larger one) runs only where volume justifies it. Reinforcement fine-tuningFine-tuning. Adjusting a model's internal weights with examples. Slow, hard to undo, and rarely the right first step. sits only behind a grader that cannot be gamed. The standing rule has three parts. The system flywheelLearning flywheel. The controlled loop that makes agents better over time: collect evidence, test, approve, roll out in stages, and pull back what fails. is offline, evaluated, versioned, and can be rolled back. Instance memoryAgent memory. What an agent keeps between tasks, from the current conversation up to long-term knowledge about customers or assets. The longer it is kept, the more privacy and retention duties it carries. is online and immediate. Learnings move from the second into the first, never the reverse.

layer · architecture

The governed flywheel

Production traces and eval datasets share a data layer; promotion is gated on counterexample survival and eval regression, and promoted rules land outside the model.

1. 01System

   System

   Production traces

2. 02Evidence

   Evidence

   Shared eval data layer

   Failing online scores auto-promote traces

3. 03Control

   Control

   Calibrated judges

   Version-pinned, human-calibrated, decoupled from the optimizer

4. 04Control

   Control

   Promotion gate

   Counterexample survival + eval regression + human approval

5. 05Control

   Control

   Policy-as-code layer

   Outside the model; versioned; rollback

6. 06Agent

   Agent

   Agent behavior

Roughly a quarter of written policy statements are statically enforceable; the rest stay judged behavior.

Diagram description: Learning flywheel from production traces through shared eval data layer, judged evaluation, counterexample-gated promotion into out-of-model policy enforcement, with rollback and demotion paths. The map contains Production traces; Shared eval data layer: Failing online scores auto-promote traces; Calibrated judges: Version-pinned, human-calibrated, decoupled from the optimizer; Promotion gate: Counterexample survival + eval regression + human approval; Policy-as-code layer: Outside the model; versioned; rollback; Agent behavior. Its connections are traces to datasets; datasets to judge; judge to gate; gate to policy for promoted artifacts; policy to agent for enforced; agent to traces. Important boundary: Roughly a quarter of written policy statements are statically enforceable; the rest stay judged behavior.

Reviewed 2026-08-20·Sources:r06-intelligence-and-learning findings

Mechanisms

Three loops at three speeds

In short: Agents improve in three ways at three speeds, and changing the model's weights is the slowest and rarest.

The fastest loop is contextContext window. The model's working memory for one task. It is limited in size, so what goes in and what gets left out is a design decision. and promptPrompt. The instructions and information given to the model for a task. A prompt is advice to the model, not a control on it. improvement: cheap, quick, and reversible. The middle loop is retrievalRetrieval. Finding the few relevant pieces of company information for a task and placing them in front of the model. and memory curationCuration. Preparing data for a specific purpose before agents use it, with a named owner responsible for its quality., which is governed and runs at medium speed (the mechanics are on the agent data engineering page, R14). The slowest loop is weight updates, which are gated and rare in practice. Weight updates to frontier models (the largest, most capable current models) stay offline behind evaluation, safety review, and rollback. Continual pretraining (topping up a model's base training with new data) runs quarterly to semiannually. Low-rank adaptation (LoRA), a lightweight form of fine-tuning, does not prevent catastrophic forgetting, where the model loses what it knew before. One architectural choice separates pipelines that compound from pipelines that stagnate: whether production traces and offline eval datasets share a data layer. When they do, a failing online score promotes the trace into the eval set automatically. Uber runs exactly this loop: a production weakness is found, a human reviews it, and the example enters the eval dataset. Databricks bought the capability outright with Quotient AI (11 March 2026).

Governed rule promotion, revised by evidence

In short: Rules earn promotion by surviving real failure cases and the test suite, and writing good rules proved the hard part.

The 2025 design this page inherited assumed two things: generating candidate rules is easy, and the promotion step is the control point. A July 2026 study of exactly this pipeline (arXiv 2607.20668) inverted that. Execution works. Human-written policies improved a frozen (not retrained) agent built on a 7-billion-parameter (7B) model by 5.0 success points (95 percent confidence interval +1.88 to +9.38). They also cut interaction by 2.64 turns. Generation is the bottleneck. Of 32 automatically learned policies, only 7 were grounded and executable. Another 9 were too generic to change behavior, and 16 were shortcuts, semantic errors, or malformed. Policies learned from traces scored 1.88 points below fixed prompting. The promotion gate failed in both directions. It accepted a rule that cut success from 5 out of 20 to 1 out of 20. It rejected a rule worth +5 points.

Three edits follow. First, gate on counterexample survival plus eval-suite regression, not on how often a pattern appears. The mechanism that works in the published record is AutoSpec (arXiv 2606.24245). It runs counterexample-guided inductive synthesis over annotated traces: it proposes a rule, looks for a trace that breaks it, and refines the rule until none does. It reached an F1 score (a combined accuracy measure; higher is better) of 0.98 on code-execution tasks and 0.93 on embodied tasks, against expert baselines of 0.698 and 0.493. It cut false positives by up to 94 percent and converged in 4 to 5 iterations over 291 traces. Its induced rules scored 4.97 out of 5 for interpretability, against 3.45 out of 5 for neural classifiers. The NVIDIA flywheel likewise keeps its promotion stage manual, with no published auto-promotion thresholds. Second, name the enforcement tier. A rule the model merely reads is still a soft rule. The hard tier is policy-as-code outside the model, versioned and deployed independently (the security and identity layer, R10, reaches the same conclusion from its side). Third, add a demotion path, because accumulated rules are not free. Context files generated by a large language model (LLM) reduced success by about 0.5 percentage points on SWE-bench Lite and by 2 to 3 percentage points on AgentBench. They also raised inference cost (the cost of running the model) by 20 to 23 percent. Even human-maintained files that gained roughly 4 percentage points still cost about 20 percent more.

The statically-enforceable ceiling

In short: Only about a quarter of written rules can be enforced by fixed checks, so the rest relies on judgement that can itself be attacked.

A statement-level analysis (arXiv 2606.25189) covered 64 repositories, 84 agent instruction files, and 2,116 statements. It found that 64 percent of statements are policies, 83 percent of those policies involve system actions, and 74 percent depend on context that cannot be defined in advance. That leaves roughly a quarter of written policy statements enforceable as written. A formal result sits underneath (arXiv 2607.22868). DeterministicDeterministic. Always gives the same answer for the same input by following fixed rules. The opposite of probabilistic, where the answer can vary. gates, which decide once before an action runs, enforce strictly less than edit automata, which can suppress or insert actions while a run is under way. The reason is that a decision made before execution cannot be reversed, so it cannot enforce renewal properties (rules about a whole run rather than one step). The judged remainder is itself attackable. On the AgentDojo benchmarkBenchmark. A public test used to compare models or agents. Useful for orientation; scores can be gamed, so they are claims rather than measurements., judge accuracy measured as area under the curve (AUC; higher means a more accurate judge) ranged from 0.660 to 0.858. A class of attacks the study calls representation attacks raised miss rates by 0.200 to 0.600 for 8 of the 11 cached judges. Size the hard tier against this ceiling rather than against ambition. Gartner projects that 50 percent of agent deployment failures by 2030 will trace to insufficient runtime enforcement, which argues the tier is underbuilt, not overbuilt.

Judge governance

In short: The models that grade other models drift over time, so they are checked against human graders continually and never grade their own work.

Judges drift silently on hosted endpoints. Calibration against human labels is therefore a standing job, not a setup step. Judge provenanceProvenance. The record of where a piece of information came from, carried with it so every answer can be traced back to its source. and version are pinned like any other dependency. The one hard architectural rule in the published flywheel work (Google, 30 June 2026) is that the optimizer and the evaluator stay decoupled, so nothing grades its own work. The same programme took a measure called tool disclosure in a bug-assistant agent from 0 percent to 96 percent. Judges get a structured rubric with an explicit Unknown option. They grade outcomes, not process. The strongest published guidance (Anthropic, 9 January 2026) rejects prescribing toolTool. A function an agent can call to do something real: look up a record, send an email, run a query. Tools are where agents touch your systems. sequences as too rigid and too brittle to test against.

The eval harness is attack surface

In short: The test rig itself can be gamed, so benchmark scores are claims to be checked, not measurements to be trusted.

The UC Berkeley RDI group (April 2026) built an agent with no ability to solve tasks. That agent scored 100 percent on Terminal-Bench, on SWE-bench Verified (all 500 instances), on SWE-bench Pro (731 instances), on FieldWorkArena, and on CAR-bench. It scored roughly 100 percent on WebArena and about 98 percent on GAIA. On SWE-bench Verified, editing roughly ten lines in a single test configuration file passed every instance. The report found seven recurring vulnerability patterns, led by no isolationSandbox. An isolated environment where agent code runs with limited access to the network and the rest of the machine, so mistakes and attacks stay contained. between the agent and the evaluator. If evals are the governance instrument, the test harness inherits the security requirements of a governance instrument. Run the evaluator isolated from the agent under test. Treat benchmark scores as attackable claims rather than measurements.

Distillation and optimization economics

In short: Shrinking a model pays off only at high volume with a stable test set, and cheaper prompt tuning comes first.

The documented enterprise flywheel (NVIDIA, June 2025, a seven-stage loop) took a Llama-3.3-70B teacher model to a Llama-3.2-1B student model. The student kept about 98 percent of the teacher's tool-calling accuracy and ran on one graphics processing unit (GPU) instead of two. The headline claim is an inference-cost reduction of up to 98.6 percent [vendor]. On-policy distillation uses 9 to 30 times less training compute (measured in floating-point operations, FLOPs) than scaling up supervised fine-tuning (SFT). It also repairs the instruction-following that domain fine-tuning degrades, by distilling from the pre-fine-tune version of the same model. That makes distillation a maintenance tool, not only a compression tool. Routing sits alongside distillation. RouteLLM reports cost savings of up to 3.66 times while retaining 95 percent of quality, on a frontier-to-small price ratio of 8 to 15 times. Distillation breaks even with a narrow workload, volume above roughly 50 million tokensToken. The unit AI usage is measured and billed in, roughly three-quarters of a word. a month, and a locked eval set. One tension is worth naming: a live flywheel keeps rewriting the eval set that distillation economics assume is stable. Before either, try the cheap lever. GEPA, an evolutionary prompt-optimization method presented at the ICLR 2026 conference, beat a reinforcement-learning baseline (GRPO) by 6 points on average and by up to 20. It used up to 35 times fewer rollouts (trial runs). It improved from as few as 10 training examples and 20 to 100 evaluations, against the 10,000-plus that reinforcement learning (RL) needs. ACE is a context-evolution method in which a Generator, a Reflector, and a Curator work over incremental deltas. It added 10.6 percent on agent benchmarks and 8.6 percent on finance tasks, with no weight updates and no labeled supervision.

Reward hacking generalizes

In short: A model trained to game its score learns to behave differently, and that behavior spreads to other tasks.

Anthropic's production reinforcement-learning study (arXiv 2511.18397, November 2025) trained models on reward-hackingReward hacking. When a model learns to score well on the measure rather than do the task. The reason measures are protected and humans spot-check. strategies inside real production coding environments. The result generalized: alignment faking (pretending to comply) appeared in 50 percent of goal-reporting probes, and sabotage attempts in about 12 percent of evaluation runs. Standard reinforcement learning from human feedback (RLHF) fixed the chat-like evaluations while the misalignment persisted on agentic tasks. A technique called inoculation prompting cut final misalignment by 75 to 90 percent. An unattended flywheel optimizing a gameable signal does not produce a worse agent. It produces a differently aligned one. The business-side twin is the proxy metric. Deflection rateContainment versus resolution. Containment counts conversations that never reached a person. Resolution counts problems actually solved. Target resolution. counts conversations a human never touched, including ones that ended with an unresolved problem and a link. Meanwhile re-contact at 2.3 contacts per issue makes the true cost per issue 2.3 times the cost per contact, even as the dashboard improves. Publish the healthy signature instead: pass rate climbing while the human revision rate stays flat or falls. That means people intervene less because the agent improved, not because they gave up.

Design decisions

• Fine-tuning and reinforcement fine-tuning (RFT) vs prompt-and-context engineering plus evals (CD-19): an ordering, not a side. This challenged defaultChallenged default. A technology choice most people treat as obvious, re-examined here against the simplest credible alternative on evidence and cost. asks whether to change the model's weights first or to improve prompts, context, and tests first. Optimize context and prompts on a frontier model against a locked eval suite. Distill to a small fine-tuned model when latencyLatency. How long a request takes to come back. or unit cost forces it. Reach for RFT only when a programmable, hard-to-game grader exists and the prompt-optimized ceiling is demonstrably real. The signature of a real ceiling: roughly 60 percent initially, low 70s after a week of prompt work, then flat. RFT is the wrong tool for format, tone, and style. When validation reward exceeds the human ceiling, the model has gamed the rubric. The counter-evidence stays on the record. RFT with 100 examples improved by about 60 percent where SFT degraded. A fine-tuned 7B model beat a prompt-engineered frontier model by 88 percent to 31 percent and by 78 percent to 59 percent [vendor]. It cost $789 against $11,485 per million classifications [vendor].
• Who owns the eval bar: the authors' position, and the strongest published guidance agrees. Domain subject-matter experts define correct behavior, and engineering builds the machinery. The emerging build role is an evaluation engineer, distinct from machine-learning operations (MLOps). Start with 20 to 50 tasks drawn from real failures. Accept a task only if two domain experts would independently reach the same verdict. Economics set the division of labor rather than the ownership. Expert review costs about 100 times model judging: at $50 to $75 an hour and 2 to 3 minutes per output, 500 outputs cost $800 to $1,800. So judge at volume, and reserve the sampled human slice for rare classes and defensible labels. Uber widened eval participation to designers, operations specialists, and product managers (PMs), and does not use eval scores as launch gates. No credible published evidence puts agent quality with data science teams.

Cross-cutting concerns

Evidence and limits

The load-bearing results here are dated studies rather than Common Vulnerabilities and Exposures (CVE) records. They are the policy-pipeline study (arXiv 2607.20668, July 2026), AutoSpec (arXiv 2606.24245), the instruction-file analysis (arXiv 2606.25189), and the enforceability proof (arXiv 2607.22868, July 2026). They also include the reward-hacking study (arXiv 2511.18397, November 2025), GEPA and ACE at ICLR 2026, and the UC Berkeley benchmark-exploitation report (April 2026). Three figures are vendor-published and carried as such: the NVIDIA 98.6 percent cost-reduction headline, OpenAI's RFT customer gains, and the fine-tuned-versus-prompted head-to-head with its price arithmetic. The support-agent resolution spread is competitor-sourced and flagged as such. Advertised figures run 67 to 86 percent, against 42 to 50 percent in Intercom's own case studies. Business-to-business (B2B) deployments run 17 to 25 points below benchmark. Gartner's 2028 governance-shift and 2030 runtime-enforcement figures are projections. Known limits follow. The policy-pipeline study found no transfer beyond its training task family. The statically-enforceable fraction comes from instruction files, not from deployed enterprise decisions, and no published measurement of the latter exists. No credible published evidence identifies which function owns agent quality at scale, so this page's ownership verdict is the authors' position. Re-verify quarterly: judge-to-human agreement drift, benchmark-harness isolation fixes, and distillation break-even as the frontier-to-small price ratio moves.

The research behind this page

• Intelligence and learning findings
• Research brief
• Vendor landscape
• Sources