Target Policy Optimization

Left: MNIST with dense reward, where TPO learns fastest. Right: reward only at the end of the sequence. DG never gets going, GRPO still fails half the time after 2,000 episodes, and TPO ends below 10% error. 20 seeds, mean ± standard error.

One prompt, a handful of answers

Most RL on language models takes a prompt, samples $K$ answers from the current model, checks each one with a verifier, and updates the model so the right answers become more likely.

GRPO, the current workhorse, standardizes the rewards within the group to get advantages $A_i = (R_i - \text{mean}\,R) / \text{std}\,R$, then moves each answer's log-probability up or down in proportion to its advantage:

That one line answers two questions at once:

1. What should change? Which answers should gain probability, and how much?
2. How should the weights move to make that change happen?

The advantages give a direction. How far the model travels along it is left to the learning rate, the number of passes over the batch, and the clip. If it moves too little, a rare success gets wasted; if it moves too much, the model overfits one lucky batch.

TPO answers the two questions one at a time: first decide where the probability should go, then move the model there.

Step 1: decide where the probability should go

TPO needs two numbers per answer: how likely the model was to write it, $p_i^{\text{old}}$ (renormalized over the $K$ sampled answers so they sum to one), and its standardized score $A_i$, the same one GRPO uses. The target reweights the old probabilities by the exponentiated score:

In words: start from what the model already does, scale good answers up and bad ones down, renormalize. $\eta$ is a temperature; I use $\eta = 1$ everywhere in the paper.

Some things to try in the widget:

• Pick All wrong. Every answer has the same reward, so every $A_i = 0$, $q = p^{\text{old}}$, and the update is exactly zero.
• Pick Lone breakthrough. The only correct answer starts at 3% and the target gives it 24%. That's an eightfold boost, but it doesn't take over the group, which I think is the right amount of trust for a single batch.
• Pick Common vs rare mistake. Both wrong answers get the same score, yet the target takes 45 points of mass from the 70% mistake and barely touches the 2% one. This matters more than it looks (section 2 below).
• Move η. Smaller η makes the target greedier; larger η keeps it closer to $p^{\text{old}}$.

Why standardize? Scores of (1, 0, −1) and (100, 0, −100) rank the answers the same way, but exponentiating the second gives an essentially one-hot target while the first gives a gentle tilt. After standardizing, the target only depends on how the answers compare with each other, which is why one $\eta$ works across tasks.

The target is a trade-off

The formula for q comes out of:

over distributions $r$ on the sampled answers: get as much advantage as you can without moving too far from what the model already does. With three answers, every distribution is a point in a triangle, so you can see the trade-off directly.

The dashed curve shows where the target lands for each $\eta$. Large $\eta$ keeps it on top of $p^{\text{old}}$; as $\eta$ shrinks, it slides toward the correct corner. The orange arrow is GRPO's first step on the same group. It points the same way, but it's only a direction, and where GRPO stops depends on the optimizer settings. TPO's target is a point you can compute before taking a single step.

Step 2: fit the target

Now make the model match $q$. TPO uses plain cross-entropy with $q$ as soft labels:

where $p_i^\theta$ is the current model's probability of answer $i$, again renormalized over the group. The gradient with respect to each answer's log-probability is

which is zero exactly when the model matches the target. This is the part I like most: the update stops on its own. It matters as soon as you reuse a batch for several gradient steps, which PPO and GRPO do routinely.

TPO settles on its target at any learning rate; a larger step only gets it there sooner. GRPO's surrogate without the clip and KL penalty has no stopping point, and keeps going until the model is certain about this one batch's winners. That's what the clip and the KL penalty are there to prevent. With TPO, the loss itself stops at the target.

The code is short:

def tpo_target(log_scores, adv, eta=1.0):
    return F.softmax(
        F.log_softmax(log_scores, -1)
        + adv / eta, -1)

q = tpo_target(
    log_scores, adv).detach()
log_p = F.log_softmax(log_scores, -1)
loss = -(q * log_p).sum(-1).mean()

log_scores holds the model's log-probabilities for the $K$ sampled answers, and adv holds their standardized scores. The one subtlety is detaching $q$, because it's a fixed label and gradients shouldn't flow through it. If you reuse a batch for more epochs, keep $q$ frozen at its rollout-time value.

What the target changes

So why would this beat GRPO? GRPO's one-number-per-answer advantages leave signal on the table in at least three places. The first two are diagnostics from Ian Osband's Delightful Policy Gradient.

1. Hard prompts get their share

Take the simplest setting I can think of: many prompts, each with one correct answer out of ten, and exact updates. Every method then pushes each prompt in the same direction, toward its correct answer. The methods only differ in how hard they push, a weight $\beta(p)$ that depends on the prompt's current pass rate $p$. If the total step size is fixed, $\beta$ decides how the step gets split between prompts.

GRPO's weight goes to zero on hard prompts and blows up on easy ones: a prompt the model solves 95% of the time gets 19× the update of one it solves 5% of the time. That's backwards if the goal is to learn what the model can't do yet. Supervised learning would split the step evenly. TPO gets close, at 1.7×. Its target multiplies the correct answer's odds by the same factor (about 28 here) on every prompt, so hard prompts still get a big update.

2. Common mistakes get pushed harder than rare ones

Back to the four answers from the first widget. GRPO sees two wrong answers with the same reward and gives them the same advantage, so it pushes both logits down equally hard, whether the model makes that mistake 70% of the time or 2% of the time. TPO's push on each answer is $q_i - p_i^{\text{old}}$, the amount of probability that has to move.

TPO pushes 35× harder on the habit than on the slip. GRPO spends as much of its update on a 2% slip as on the mistake the model keeps making.

3. Each group gets its own stopping point

PPO and GRPO stop a reused batch with a clip: each answer's probability ratio $\pi_\theta / \pi_{\text{old}}$ may only move within $[1-\varepsilon,\, 1+\varepsilon]$, usually with $\varepsilon = 0.2$. Every answer gets the same band. Once a breakthrough that the model finds 1% of the time reaches 1.2%, the clip stops pushing it, while an easy answer at 90% can coast all the way to 100%. DAPO's "clip-higher" trick was introduced for this reason. TPO has no clip. It stops at $q$, which each group sets from its own evidence: a lone success at 1% in a group of eight gets a target of 17%.

Does it work?

The experiments in the paper go from tabular bandits up to 1.7B-parameter LLMs. The short version: with dense reward, TPO is about as good as GRPO, PPO and DG, sometimes a bit better. With sparse reward, it pulls away.

Sparse reward

A small transformer learns to reverse a binary sequence of length $H$ and only gets a reward if the whole sequence is right. Longer sequences make a correct rollout rarer: at $H = 10$, a random guess is right one time in 1,024.

Stale rollouts

Large RL systems often generate rollouts with slightly outdated weights. In an MNIST bandit where the actors use parameters from $D$ steps ago, TPO's error stays at 2.3% all the way from $D = 0$ to $D = 1000$.

LLMs

I also swapped GRPO's loss for TPO's in verl, on Qwen3-1.7B and DeepSeek-R1-Distill-Qwen-1.5B with 16 rollouts per prompt, and changed nothing else:

GSM8K ends in a near tie, though TPO gets there faster. On graph coloring with Qwen3-1.7B, GRPO peaks early and then degrades, while TPO keeps climbing to about 0.96.

Independently, a team at Tencent came up with a closely related objective, Listwise Policy Optimization, which also frames group-based RLVR as projecting onto a target on the response simplex.

An old idea, in closed form

Reweight-then-fit is old. It goes back at least to Dayan and Hinton's EM for RL (1997) and runs through REPS, MPO and SPU, each of which had to work around something expensive. Group-based RL makes this much easier: with a finite group of scored answers, the target is just a softmax, so you need neither a critic nor a dual problem.

Cite

@article{kaddour2026tpo,
  title   = {Target Policy Optimization},
  author  = {Kaddour, Jean},
  journal = {arXiv preprint arXiv:2604.06159},
  year    = {2026}
}