CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

Authors: Bo-Wen Zhang, Junwei He, Wen Wang, Song-Lin Lv, Wentao Ma, Rongyi Lin, Shuhan Zhong, Lan-Zhe Guo

arXiv ID: 2607.25659

Problem: Rubric-based reinforcement learning evaluates model outputs against explicit criteria, but in GRPO-style pipelines those structured judgments are reduced to a scalar response-level reward, converted to a response-level advantage, and broadcast uniformly to every generated token. Different criteria are grounded in different spans, formatting decisions, or semantic choices, yet there is no mechanism to allocate credit within a response.

Key Methodology:

Counterfactual replay: rescore the same sampled response under the original rubric-conditioned prompt and a matched criteria-free prompt
The tokenwise log-likelihood contrast between the two scorings serves as a proxy for dependence on the rubric context
Contrasts are mapped to bounded, response-normalized weights that redistribute the signed GRPO advantage across tokens
No auxiliary token scoring model, no change to the response-level reward, no separate relevance-learning stage

Key Results:

Improves over matched response-level GRPO in the vast majority of comparisons, with an average gain of 4.4 percentage points
Competitive with learned token-level credit baselines while avoiding their extra training stage
Works across instruction-tuned models and reward granularities

What it means for developers: Policy-internal counterfactual likelihood contrasts are an effective training signal for within-response credit allocation while retaining the simplicity and stability of GRPO. Teams training with rubric-conditioned RL can get token-level credit without buying or training a token scorer - the model's own likelihood under matched prompt pairs is the credit signal.

Paper: arXiv:2607.25659