Reinforcement fine-tuning

How training against a grader works, how graders get gamed, and the narrow tasks where it is the right tool.

The model's output has the right shape, but its judgement on hard cases is still wrong. When is it right to train a model against a score instead of examples? Other kinds of training show the model what right looks like. Reinforcement fine-tuning lets the model try, scores the attempt, and shifts it towards what scored well. It can teach judgement that is hard to show in examples. The catch is the scorer, called a grader. The model becomes excellent at whatever the grader actually measures. That is not always what you meant. So the entry requirement is a grader that cannot be satisfied without doing the real job. This is one of the model-change options on the intelligence ladder.

How training against a grader works

The loop has three parts. The model makes an attempt at the task. A grader scores it. Training raises the odds of the behaviours that led to high scores. Supervised fine-tuning pulls the model towards fixed reference answers. This method pulls it towards whatever scores well, found by the model's own exploring.

That difference is both the power and the danger. The power: where good answers vary and cannot be listed, the model can find strategies no example set held. The reported efficiency shows it. Gains came from around a hundred graded tasks, where supervised fine-tuning at that scale made the model worse [vendor].

The danger: the model searches for score, not for your intent. Any gap between the two will be found. Nothing like malice is involved. High-scoring shortcuts are simply reinforced. And the model makes far more attempts to find gaps than you spent designing the grader.

What makes a grader gameable

A grader can be gamed when the score and real skill can come apart. The common gaps:

Surface markers. If well-structured answers with confident phrasing tend to score well, the model learns structure and confidence. It learns them apart from correctness.
Length and effort proxies. If thorough answers score well, expect padding.
A model as the judge. Using another model as the judge makes that model's biases the target. Models have documented preferences for the style of their own family. Our own published experiment found this. An effect seen under a judge from the same model family vanished under an independent judge.
Incomplete coverage. A grader that checks three properties of a good answer trains a model that improves those three. It neglects the rest.

Grading is safe on tasks that share one shape. A program can check the outcome, and that program does not share the model's blind spots. Code that must compile and pass tests. An extracted figure that must match the filing. A schedule that must meet every constraint. Where quality is taste, tone or judgement that no program holds, the grader will be a stand-in. The stand-in will be gamed.

foundation · comparison

Where grading is safe, and where it will be gamed

The model becomes excellent at whatever the grader actually measures. Score and competence must be inseparable.

Mixed evidence

Safe to grade

Code compiles, tests pass
Figure matches the filing
Every claim traces to a source
Checkable by program

Will be gamed

Style and confidence proxies
Length as effort
An LLM judging its own family
Three checks, the rest neglected

The alarm

Humans agree 92 to 95 percent
Model scores 99 percent
The checker has a hole
File it as an incident

A score above what careful people achieve on the same task means the rubric was learned, not the job. Treat it as a defect report.

A model acts. A deterministic checker evaluates the outcome and a reward returns. An independent human benchmark can trigger an alarm. — **The grader closes the learning loop**Reinforcement fine-tuning works when the real outcome can be checked automatically. The human benchmark stays outside the loop. An implausibly high machine score is then treated as a grader fault.

The alarm: a score above a careful person's

Set one alarm before training starts. If the validation score beats a careful person on the same task, the model has learned the rubric. It has not learned the job. Treat a score above the human level as a fault report. This is reward hacking, and it is well documented in production reinforcement learning. The right response is to treat the grader as the broken part.

The practical consequence is that the grader is now production code. It gets versioning, review, tests and an owner. A fault in the grader trains a fault into the model. Models trained on a faulty grader do not announce it. They score very well.

The learning flywheel covers judges, tests and how a trained model is promoted. A trained model also re-runs the evals at the change stage of the agent lifecycle.

What it fixes, and what it does not

A model takes a shortcut through three shallow checks. A stronger evaluation follows a complete evidence network with independent review. — **A narrow score creates a narrow intelligence**If the grader checks only visible features, the model learns to decorate those features and route around the job. Strong evaluation ties several independent checks to source evidence and keeps human review outside the model's influence.

It fixes judgement and multi-step reasoning on tasks with checkable outcomes. Vendor-published customer results report a 39 percent improvement on tax analysis [vendor]. They report 12 points over physician baselines [vendor], and 20 percent on citation-extraction F1 [vendor].

It does not fix format, tone or style. Those can be shown by example, so they belong to adapters. It also cannot add knowledge. None of these methods can, except continued pretraining. The mechanism shapes behaviour. It does not store facts.

Worked example: a compliance-evidence assistant

Four boxes in order. Task: prepare supported findings. Feedback: check evidence and omissions. Challenge: try to exploit the score. Evaluation: use independent review. — **Check whether the feedback measures the job**Proposed industrial example. Evaluate against the simpler existing process before adopting it.

A compliance-evidence assistant at an energy company might be scored for three things. It cites the correct clause, finds the supporting record and identifies missing evidence. Before training on that score, test whether it can reward a plausible but invalid answer.

Past audit closures are not automatically the right answers. They may reflect incomplete inspection, uneven practice or missed issues. A grader that checks a citation is present does not show that the cited evidence supports the claim.

Test the grader with independent review, held-out cases and adversarial examples. A high automated score is a reason to inspect the result. It does not make the model an assurance authority. Formal conclusions stay with the responsible review process. See compliance evidence.

What comes next

Judgement on checkable outcomes may now be right while the model still misreads your domain's material. Then the missing piece is continued pretraining. That is a different job, not a further step. Whatever was trained here, distillation can make it cheaper once quality is proven.

Evidence and limits

Vendor reported The domain gains are vendor-published customer results from May 2025, flagged as such. Reward hacking is independently documented. So is the judge-bias finding. It also appears in our own published experiments. The compliance example is a proposed design.

Next step

Read the next option, continued pretraining.

Explore this topic

Reinforcement fine-tuning customer results and sample efficiency (May 2025) [vendor]: intelligence and learning.
Why this comes only after a working grader: the decision catalog.
Reward hacking in production reinforcement learning: covered on the learning flywheel.
The same-family judge effect vanishing under an independent judge: diversity, and who consumes the results.
Industrial scenarios on this page are proposed designs, illustrated in the use cases.