Choose what to change

An agent schedules an inspection that was completed yesterday. Should we buy a more capable model, rewrite its instructions, or train it? First find out what prevented it from doing the job correctly.

A stronger model may not fix the problem

It is tempting to treat every wrong answer as insufficient intelligence. In this example, the completion record may never have reached the agent. It may concern a different inspection case. Or a failed tool call may have been mistaken for a successful lookup. Changing the model without checking these possibilities leaves the cause unresolved.

The intelligence ladder explains the available ways to build capability. This page helps choose which part to change. The aim is a better completed task at an acceptable full cost.

Start with a failed task

Keep the request, allowed actions, sources available at the time, tool results, and actual outcome. Ask the inspection owner what an acceptable result would have been. A useful answer might have three parts. Identify the completed inspection. Check unresolved findings. Request a further inspection only if the evidence supports it.

That gives us something concrete to test. "Make the agent more reliable" does not.

Run a comparison that separates the causes

First supply the correct completion record to the same model, keeping the task and other settings comparable. If its answer improves consistently, investigate the information path. This is evidence for a retrieval or context problem, not proof that the whole system is fixed.

Then test the proposed fix through the real retrieval path. Include inspections that were not completed, old records, wrong asset identifiers, and missing information. Supplying the answer manually is a diagnostic test; it is not the production solution.

Check the information

Could the agent access the correct source? Did parsing retain the completion date and asset identity? Was the current version retrieved? Did the evidence actually reach the model? Use context testing and memory organization to examine this path.

Check the task and tools

If evidence is adequate but the requested result is unclear, try clearer instructions and examples. If the agent selects a sensible action but cannot complete it, inspect permissions, tool inputs, errors, and confirmation. A submitted inspection request is not a completed inspection. Test retries without creating duplicate work.

Check the workflow

If the same steps are always required, compare a fixed workflow. If the next check depends on the result of the previous one, compare a bounded investigation. If separate investigations are useful, evaluate one agent or several. Changing coordination should address an observed difficulty.

Then consider changing the model

When the task, evidence, and tools are adequate, compare another model, better examples, or training against the remaining limitation. If capability is already good but expensive, test whether a smaller model or distillation can preserve the required quality.

Use tasks held apart from development. Count unsuccessful attempts, expert review, latency, and maintenance alongside model charges. Retain changes that improve the agreed outcome enough to justify their cost. See value and full cost.

A new model, prompt or tool is a change to a running agent. It re-runs the evals at the change stage of the agent lifecycle. The learning flywheel sets out how the change is tested and promoted.

Evidence and limits

This diagnostic order and the inspection case are teaching proposals, not reported experimental results. Anthropic's Building effective agents supports starting simply and testing added complexity (19 December 2024, reviewed 18 September 2026).

Next step

Start with the cheapest change, step 1: instructions.