# Reinforcement fine-tuning

How training against a grader works, how graders get gamed, and the narrow tasks where it is the right tool.

Author: Murali Sid (https://linkedin.com/in/muralisid)
Source: https://www.agenticarchitectureskills.com/ladder/reinforcement-fine-tuning
Published: 2026-08-29
Updated: 2026-10-15
Licence: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/
Cite as: Murali Sid. Reinforcement fine-tuning. Agentic Architecture Skills. https://www.agenticarchitectureskills.com/ladder/reinforcement-fine-tuning

---


The model's output has the right shape, but its judgement on hard cases is still wrong. **When is it right to train a model against a score instead of examples?** Other kinds of training show the model what right looks like. Reinforcement fine-tuning lets the model try, scores the attempt, and shifts it towards what scored well. It can teach judgement that is hard to show in examples. The catch is the scorer, called a grader. The model becomes excellent at whatever the grader actually measures. That is not always what you meant. So the entry requirement is a grader that cannot be satisfied without doing the real job. This is one of the model-change options on [the intelligence ladder](/ladder).

## How training against a grader works [#how-training-against-a-grader-works]

The loop has three parts. The model makes an attempt at the task. A grader scores it. Training raises the odds of the behaviours that led to high scores. Supervised <Term k="fine-tuning">fine-tuning</Term> pulls the model towards fixed reference answers. This method pulls it towards whatever scores well, found by the model's own exploring.

That difference is both the power and the danger. The power: where good answers vary and cannot be listed, the model can find strategies no example set held. The reported efficiency shows it. Gains came from around a hundred graded tasks, where supervised fine-tuning at that scale made the model worse \[vendor].

The danger: the model searches for score, not for your intent. Any gap between the two will be found. Nothing like malice is involved. High-scoring shortcuts are simply reinforced. And the model makes far more attempts to find gaps than you spent designing the grader.

## What makes a grader gameable [#what-makes-a-grader-gameable]

A grader can be gamed when the score and real skill can come apart. The common gaps:

* **Surface markers.** If well-structured answers with confident phrasing tend to score well, the model learns structure and confidence. It learns them apart from correctness.
* **Length and effort proxies.** If thorough answers score well, expect padding.
* **A model as the judge.** Using another model as the <Term k="judge">judge</Term> makes that model's biases the target. Models have documented preferences for the style of their own family. Our own published experiment found this. An effect seen under a judge from the same model family vanished under an independent judge.
* **Incomplete coverage.** A grader that checks three properties of a good answer trains a model that improves those three. It neglects the rest.

Grading is safe on tasks that share one shape. &#x2A;*A program can check the outcome, and that program does not share the model's blind spots.** Code that must compile and pass tests. An extracted figure that must match the filing. A schedule that must meet every constraint. Where quality is taste, tone or judgement that no program holds, the grader will be a stand-in. The stand-in will be gamed.

<GuideFigure id="grader-gameability" />

<TeachingIllustration src="/images/ladder/reinforcement-loop-labeled-v2.webp" alt="A model acts. A deterministic checker evaluates the outcome and a reward returns. An independent human benchmark can trigger an alarm." title="The grader closes the learning loop" caption="Reinforcement fine-tuning works when the real outcome can be checked automatically. The human benchmark stays outside the loop. An implausibly high machine score is then treated as a grader fault." />

## The alarm: a score above a careful person's [#the-alarm-a-score-above-a-careful-persons]

**Set one alarm before training starts. If the validation score beats a careful person on the same task, the model has learned the rubric.** It has not learned the job. Treat a score above the human level as a fault report. This is <Term k="reward hacking">reward hacking</Term>, and it is well documented in production reinforcement learning. The right response is to treat the grader as the broken part.

The practical consequence is that the grader is now production code. It gets versioning, review, tests and an owner. A fault in the grader trains a fault into the model. Models trained on a faulty grader do not announce it. They score very well.

[The learning flywheel](/architecture/learning-flywheel) covers judges, tests and how a trained model is promoted. A trained model also re-runs the evals at the change stage of [the agent lifecycle](/architecture/agent-lifecycle).

## What it fixes, and what it does not [#what-it-fixes-and-what-it-does-not]

<TeachingIllustration src="/images/ladder/reinforcement-reward-hacking-labeled-v2.webp" alt="A model takes a shortcut through three shallow checks. A stronger evaluation follows a complete evidence network with independent review." title="A narrow score creates a narrow intelligence" caption="If the grader checks only visible features, the model learns to decorate those features and route around the job. Strong evaluation ties several independent checks to source evidence and keeps human review outside the model's influence." />

**It fixes** judgement and multi-step reasoning on tasks with checkable outcomes. Vendor-published customer results report a 39 percent improvement on tax analysis \[vendor]. They report 12 points over physician baselines \[vendor], and 20 percent on citation-extraction F1 \[vendor].

**It does not fix** format, tone or style. Those can be shown by example, so they belong to [adapters](/ladder/adapters-and-fine-tuning). It also cannot add knowledge. None of these methods can, except [continued pretraining](/ladder/continued-pretraining). The mechanism shapes behaviour. It does not store facts.

## Worked example: a compliance-evidence assistant [#worked-example-a-compliance-evidence-assistant]

<TeachingIllustration src="/figures/ladder/industrial-reinforcement-fine-tuning.svg" mobileSrc="/figures/ladder/industrial-reinforcement-fine-tuning-mobile.svg" title="Check whether the feedback measures the job" alt="Four boxes in order. Task: prepare supported findings. Feedback: check evidence and omissions. Challenge: try to exploit the score. Evaluation: use independent review." caption="Proposed industrial example. Evaluate against the simpler existing process before adopting it." />

A compliance-evidence assistant at an energy company might be scored for three things. It cites the correct clause, finds the supporting record and identifies missing evidence. Before training on that score, test whether it can reward a plausible but invalid answer.

Past audit closures are not automatically the right answers. They may reflect incomplete inspection, uneven practice or missed issues. A grader that checks a citation is present does not show that the cited evidence supports the claim.

Test the grader with independent review, held-out cases and adversarial examples. A high automated score is a reason to inspect the result. It does not make the model an assurance authority. Formal conclusions stay with the responsible review process. See [compliance evidence](/use-cases/compliance-evidence).

## What comes next [#what-comes-next]

Judgement on checkable outcomes may now be right while the model still misreads your domain's material. Then the missing piece is [continued pretraining](/ladder/continued-pretraining). That is a different job, not a further step. Whatever was trained here, [distillation](/ladder/distillation) can make it cheaper once quality is proven.

## Evidence and limits [#evidence-and-limits]

<EvidenceBadge status="vendor-reported" /> The domain gains are vendor-published customer results from May 2025, flagged as such. Reward hacking is independently documented. So is the judge-bias finding. It also appears in our own published experiments. The compliance example is a proposed design.

## Next step [#next-step]

Read the next option, [continued pretraining](/ladder/continued-pretraining).

<details>
  <summary>
    Explore this topic
  </summary>

  * Reinforcement fine-tuning customer results and sample efficiency (May 2025) \[vendor]: [intelligence and learning](/layers/r06-intelligence-and-learning).
  * Why this comes only after a working grader: [the decision catalog](/decisions).
  * Reward hacking in production reinforcement learning: covered on [the learning flywheel](/architecture/learning-flywheel).
  * The same-family judge effect vanishing under an independent judge: [diversity, and who consumes the results](/patterns/diversity-and-consumer).
  * Industrial scenarios on this page are proposed designs, illustrated in the [use cases](/use-cases).
</details>