Scaling Laws for Generative Reward Models

Updated 23 Oct 2026

Static judge accuracy and downstream training quality can disagree. In our matched online-DPO study, Answer-Only rewarders produce higher peak and top-five policy ratings than Thinking rewarders in seven of eight settings under the learned reference evaluator, despite the Thinking judges’ higher static accuracy.

I led the implementation of the distributed online-DPO training system and the experimental program behind this first-author work. The public research package provides the manuscript, derived results and reproducible CPU analysis. The custom training implementation is maintained privately.

A good judge is not necessarily a good rewarder

A generative reward model reads an instruction and two responses, then decides which response is better. That makes it useful as an evaluator. Put the same model inside a training loop, however, and it has a second job: its preferences determine how another model changes.

Those jobs expose different weaknesses. A fixed benchmark tests judgments on an existing response distribution. Online training continually changes that distribution. A policy can improve according to its own judge while moving away from the behavior that a separate reference evaluator prefers.

Our study asks how to choose and train generative reward models for that second job. We examine reward-model capacity, policy capacity, judge training, reasoning, training duration and online-DPO regularization in creative writing. The central measurement is the behavior of the complete training loop.

What we varied

The study uses Qwen3 models, LitBench’s curated, vote-derived preferences over human-written Reddit stories, and policy-training prompts derived from WritingPrompts. A fine-tuned 32B evaluator supplies reference judgments. Smaller generative reward models learn from its labels and then supervise online policy optimization. Gold, proxy and policy validation reuse the same 1,000 source pair records and their prompts; the paper spells out the distinct labels and response references.

These are matched and fixed-axis sweeps, not every combination of every axis. The comparison also changes the judge’s training method: it does not isolate the causal effect of verbalized reasoning alone.

The evaluator–rewarder gap

Thinking judges are more accurate than Answer-Only judges on the fixed Gold-labeled preference set across the model sizes studied. That ranking does not reliably carry through to the policies they train.

In the shared global Gold arena, Answer-Only rewarders produce higher peak policy Elo in seven of eight matched configurations. The same seven-of-eight pattern holds when averaging each run’s five highest-rated snapshots. The smallest-policy configuration is the exception; some other gaps are small.

Intermediate reward-model checkpoints supply another view of the same problem. A checkpoint that improves static judging accuracy need not be the checkpoint that produces the best downstream policy. Selecting the rewarder is therefore part of the policy-optimization problem, rather than a preprocessing decision settled by validation accuracy alone.

Capacity on both sides of the loop

Increasing reward-model size substantially improves Gold-evaluated policy quality in the measured sweeps. For a fixed 4B policy, the Answer-Only 14B judge outperforms its smaller counterparts—even though the reward model is much larger than the model being trained. Thinking rewarders also benefit from scale, but not monotonically: the 8B Thinking judge slightly exceeds the 14B Thinking judge in the global arena.

Scaling the policy matters too. The paper studies the interaction between the capacity that generates responses and the capacity that evaluates them, rather than treating reward-model size as an incidental implementation setting. Post-hoc fits summarize associations with policy and reward-model size. They describe the measured grid; they are not independently validated predictions for unmeasured model sizes.

The training system behind the study

I implemented the custom online-DPO trainer and actor/loss path in a fork of verl, and carried out the distributed training and evaluation program with my coauthors. The system coordinates three distinct model roles: a trainable current policy, a frozen reference policy for DPO likelihood ratios, and a generative pairwise judge.

verl provided the distributed foundation. The custom work connected fresh policy generations to pairwise judge preferences and online DPO updates, without requiring a precomputed preference dataset between training rounds. That made the judge itself an experimental component of the learning loop.

Each update performs an online preference-learning cycle:

1. Sample current-policy behavior. Generate two fresh responses for each prompt, retaining the response tokens and prompt identity through distributed batching.
2. Construct preferences. Ask the generative judge about both response orderings, normalize and aggregate its verdicts, and filter zero-sum results. A verdict parsed as invalid contributes zero, so a surviving pair need not have two valid votes. The judge has its own tokenizer, chat template and generation settings.
3. Train on the resulting pairs. Score chosen and rejected responses under the current and frozen-reference policies. Apply the sequence-level DPO objective with response masking, dynamic microbatching and distributed gradient updates.

Periodic validation and checkpointing track the evolving policy. In-loop validation uses the configured proxy judge; a separate evaluation and analysis path measures saved policy outputs under Gold. This keeps the training signal distinct from the reference used to compare the resulting policies.

Ray coordinates distributed workers, FSDP supplies sharded training and vLLM supports generation. Independent multi-GPU training jobs made it possible to compare judge sizes, policy sizes, training recipes and regularization settings in parallel. The engineering work included preference-pair identity, reward construction, distributed actor updates, evaluation and sharded checkpointing.

This research predates the managed agent-learning platform described in my systems article. Both involve the same broader concern: connecting real model behavior, trustworthy feedback and distributed model updates. They are separate systems and research efforts.

Code and reproducibility

The public research repository connects the paper to its numerical evidence and analysis code:

• CPU analysis: tools for verifying recorded results, reproducing figures, summarizing comparisons and inspecting descriptive fits.
• Research evidence: derived numerical tables, prompt templates and provenance linking the plots to their inputs.
• Manuscript artifacts: the paper and its versioned research release.

As of 23 October 2026, this public package excludes the custom training implementation and private training recipes. The paper’s original implementation availability statement predates this packaging change. The system described above is my implementation work; the public release supports inspection and analysis of the recorded study rather than a fresh execution of that trainer.

An earlier exploratory archive preserves original data-recipe and distillation comparisons, separately from the paper’s matched study.

The release makes CPU analysis reproducible without model calls. It exposes the conditional Elo uncertainty method alongside the retained comparison results.

Interpreting the result

The strongest conclusion is operational: evaluate a reward model through the optimization it induces, not only through its static preference accuracy. In this study, more accurate fixed-distribution judges can be worse rewarders under a specified learned reference evaluator.

That reference is not a human oracle. The result does not establish universal Answer-Only superiority, a cross-domain law, or a causal disadvantage of reasoning. A retained diagnostic using a different, larger learned judge is inconclusive overall, and preference gains do not establish better stylistic diversity. The paper reports these boundaries, the matched-setting exceptions, and uncertainty conditional on the stored comparison data.

The broader question connects to earlier work on reward-model evaluation and downstream performance and evaluation through reward overoptimization. Our focus is the generative-rewarder loop: practical judge-training pipelines, rewarder/policy capacity, checkpoint selection and online-DPO dynamics.

Manuscript and citation

The PDF linked at the top is the versioned arXiv manuscript. The public analysis release and source links are collected under Resources. The manuscript contains the experimental setup, matched-condition comparisons, readable chronological training trajectories, and an expanded empirical appendix. It documents the Elo estimator and uncertainty procedure, and distinguishes descriptive scaling fits from independently validated laws. The version record also retains the separately hashed website copy. It is not advertised as an identical second primary manuscript.

@misc{raychev2026scaling,
  title  = {Scaling Laws for Generative Reward Models},
  author = {Angel Raychev and Nathan Lile and Rafael Rafailov},
  year   = {2026},
  eprint = {2610.30765},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  url    = {https://arxiv.org/abs/2610.30765}
}