Target Policy Optimization

TPO (blue) against GRPO (orange) and DG (red). Left: with dense reward on MNIST, TPO learns fastest. Right: with reward only at the end of a sequence, DG never gets going and GRPO still fails half the time after 2,000 episodes, while TPO gets below 10% error. 20 seeds, mean ± standard error.

One prompt, a handful of answers

Here is the loop behind most RL on language models today. Take a prompt. Sample $K$ answers from the current model. Score each one with a verifier: right or wrong. Then update the model so the right answers become more likely.

GRPO, the current workhorse, turns the rewards into advantages by standardizing them within the group, $A_i = (R_i - \text{mean}\,R) / \text{std}\,R$, then moves each answer’s log-probability up or down in proportion to its advantage:

That one line answers two different questions at once:

1. What should change? Which answers should gain probability, and how much?
2. How should the weights move to make that change happen?

The advantages set a direction. But how far the model actually travels depends on the learning rate, on how many times the batch is reused, and on where the clip kicks in. Travel too little and a rare success is wasted. Travel too far and the model overfits one lucky batch.

TPO answers the two questions separately. First decide where the probability should go. Then move the model there.

Step 1: decide where the probability should go

TPO needs two numbers per answer. The first is how likely the model was to write it, $p_i^{\text{old}}$, renormalized over the $K$ sampled answers so they sum to one. The second is its standardized score $A_i$, the same one GRPO uses. The target reweights the old probabilities by the exponentiated score:

Start from what the model already does, scale the good answers up and the bad ones down, and renormalize. $\eta$ is a temperature; the paper uses $\eta = 1$ everywhere.

A few things to try:

• All wrong. Every answer has the same reward, so every $A_i = 0$ and $q = p^{\text{old}}$. No signal, no update.
• Lone breakthrough. The only correct answer had 3% of the mass. The target gives it 24%: an eightfold boost, but not a takeover. One batch is evidence, not proof.
• Common vs rare mistake. Both wrong answers get the same score. Yet the target takes 45 points of mass from the 70% mistake and barely touches the 2% one. We’ll come back to this.
• Turn η down and the target gets greedier; turn it up and it stays closer to $p^{\text{old}}$.

Why standardize? Scores of (1, 0, −1) and (100, 0, −100) rank the answers identically. But exponentiating the second makes the target essentially one-hot, while the first gives a gentle tilt. Standardizing within the group makes the target depend only on how answers compare with each other, which is why a single $\eta$ works across tasks.

The target is a trade-off

The formula for q isn’t arbitrary. It is the exact solution of a small optimization problem:

over distributions $r$ on the sampled answers. Collect as much advantage as you can without drifting too far from what the model already does. With three answers, every distribution is a point in a triangle, so you can watch this trade-off directly.

The dashed curve traces the target for every $\eta$. When $\eta$ is large, the target sits on top of $p^{\text{old}}$. As $\eta$ shrinks, it slides toward the correct corner. The orange arrow is GRPO’s first step on the same group. It points the same way, but it is only a direction: where GRPO ends up depends on the learning rate, the number of epochs and the clip. TPO’s target is a place.

Step 2: fit the target

Now make the model match $q$. TPO uses the plainest loss there is, cross-entropy with $q$ as soft labels:

where $p_i^\theta$ is the current model’s probability of answer $i$, again renormalized over the group. Its gradient with respect to each answer’s log-probability is

which is zero exactly when the model matches the target. The update switches itself off. That matters as soon as you reuse a batch for several gradient steps, which PPO and GRPO routinely do.

TPO settles on its target whatever the learning rate; a bigger step only gets there sooner. GRPO’s surrogate, stripped of its clip and KL penalty, has no such stopping point. It keeps going until the model is certain of this one batch’s winners. That is why PPO and GRPO need a clip and a KL penalty. TPO’s stopping point comes with the loss.

The whole method fits in a few lines:

def tpo_target(log_scores, adv, eta=1.0):
    return F.softmax(
        F.log_softmax(log_scores, -1)
        + adv / eta, -1)

q = tpo_target(
    log_scores, adv).detach()
log_p = F.log_softmax(log_scores, -1)
loss = -(q * log_p).sum(-1).mean()

log_scores holds the model’s log-probabilities of the $K$ sampled answers, and adv their standardized scores. The only subtlety is detaching $q$: the target is a label, not something to optimize. When a batch is reused for more epochs, $q$ stays frozen at its rollout-time value.

What the target changes

Why would this beat GRPO? Here are three places where GRPO’s one-number-per-answer advantages leave signal on the table. The first two are diagnostics from Ian Osband’s Delightful Policy Gradient.

1. Hard prompts get their share

Take the simplest possible setting: many prompts, each with one correct answer out of ten, and exact updates. Every method then pushes each prompt in the same direction, toward its correct answer. They differ only in how hard they push: a weight $\beta(p)$ that depends on the prompt’s current pass rate $p$. With the total step size fixed, $\beta$ decides how the step is split between prompts.

GRPO’s weight vanishes on hard prompts and blows up on easy ones. A prompt the model solves 95% of the time gets 19× the update of one it solves 5% of the time. That is backwards if the point is to learn what the model can’t do yet. Supervised learning would split the step evenly. TPO stays close to even, at 1.7×, because its target multiplies the correct answer’s odds by the same factor (about 28 here) on every prompt, so hard prompts still get a large update.

2. A common mistake is not a rare one

Back to the four answers from the first widget. GRPO sees two wrong answers with the same reward and gives them the same advantage. So it pushes both of their logits down equally hard, whether the model makes that mistake 70% of the time or 2%. TPO’s push on each answer is $q_i - p_i^{\text{old}}$: how much probability actually has to move.

TPO pushes 35× harder on the habit than on the slip. GRPO spends as much of its update on a 2% slip as on the mistake the model actually keeps making.

3. Every group gets its own stopping point

PPO and GRPO stop a reused batch with a clip. Each answer’s probability ratio $\pi_\theta / \pi_{\text{old}}$ may only move within $[1-\varepsilon,\, 1+\varepsilon]$, typically with $\varepsilon = 0.2$. That is the same band for every answer. Once a breakthrough the model finds 1% of the time reaches 1.2%, the clip stops pushing it, while an easy answer at 90% can coast all the way to 100%. DAPO’s “clip-higher” trick exists for this reason. TPO has no clip. Its stopping point is $q$, set for each group by the evidence: a lone success at 1% in a group of eight gets a target of 17%.

Does it work?

The paper tests TPO on everything from tabular bandits to 1.7B-parameter LLMs. It matches GRPO, PPO and DG when reward is dense, and pulls away when reward is sparse.

Sparse reward. A small transformer learns to reverse a binary sequence of length $H$, and is rewarded only if the whole sequence is right. Longer sequences make a correct rollout rarer: at $H = 10$, a random guess is right one time in 1,024.

Stale rollouts. Large RL systems generate rollouts with slightly outdated weights. In an MNIST bandit where actors use parameters from $D$ steps ago, TPO’s error doesn’t move from $D = 0$ to $D = 1000$.

LLMs. On Qwen3-1.7B and DeepSeek-R1-Distill-Qwen-1.5B with 16 rollouts per prompt, swapping only the loss:

On GSM8K, TPO learns faster and both methods end near the same accuracy. On Reasoning Gym graph coloring with Qwen3-1.7B, GRPO peaks early and then degrades, while TPO keeps climbing to about 0.96.

A team at Tencent independently arrived at a closely related objective, Listwise Policy Optimization, which also treats group-based RLVR as projecting onto a target on the response simplex.

An old idea, in closed form

Reweight-then-fit is not new. It goes back at least to Dayan and Hinton’s EM for RL in 1997 and runs through REPS, MPO and SPU. Each had to work around something expensive. Group-based RL removes the obstacle: with a finite group of scored answers, the target is a softmax. No critic, no dual.

Limitations

• TPO can only move probability among the answers it sampled. If all $K$ are bad in the same way, the target has nothing to say. It also still needs $K$ rollouts per prompt, just like GRPO.
• Standardizing makes tiny score differences look sharp when a group has little variance: one answer scoring 0.001 and the rest 0 produces a very confident target. This is the same difficulty bias that has been studied for GRPO.
• The LLM experiments stop at 1.7B parameters. Whether the gains hold at 7B and beyond, on MATH or AIME, is still open.

Cite

@article{kaddour2026tpo,
  title   = {Target Policy Optimization},
  author  = {Kaddour, Jean},
  journal = {arXiv preprint arXiv:2604.06159},
  year    = {2026}
}