Literature Review: Tricks or Traps? A Deep Dive into RL for LLM Reasoning

This survey-esque paper presents a systematic empirical evaluation of existing reinforcement learning (RL) techniques applied to large language model (LLM) reasoning tasks. Rather than proposing new algorithms, the authors reproduce and analyze a broad set of commonly used RL variants—such as PPO, GRPO, and DAPO—and practical “tricks” used in recent LLM training pipelines. Their goal is to clarify the often contradictory findings in the literature by conducting controlled comparisons within a unified framework (the open-source ROLL platform), thereby offering practical insights and guidelines for selecting and combining RL methods in reasoning-focused training.

Key Insights

1. Advantage normalization determines training stability.
   Group-level normalization works best across reward types, while batch-level normalization scales better for large or diverse rewards. The most robust combination uses group-level means and batch-level standard deviations.

2. Clip-Higher restores exploration in aligned models.
   By raising PPO’s clipping bound, models avoid entropy collapse and maintain diversity in reasoning paths. Smaller models benefit steadily from higher bounds, while larger ones plateau around moderate values.

3. Loss aggregation must match model type.
   Token-level aggregation improves convergence for unaligned base models, but aligned models perform better with sequence-level aggregation that preserves structural reasoning coherence.

4. Overlong filtering helps short reasoning but not complex tasks.
   Filtering overly long responses stabilizes training for medium-length problems but offers diminishing returns on long-form or competition-level reasoning datasets.

5. Lite PPO outperforms complex variants.
   A simple combination of advantage normalization and token-level aggregation enables critic-free PPO to surpass GRPO and DAPO, showing that minimal, well-chosen components can outperform heavy designs.

Example

Consider training Qwen3-8B on OlympiadBench. Using group-level normalization stabilizes reward variance, while a Clip-Higher bound of 0.28 maintains entropy and exploration. Applying token-level loss aggregation ensures balanced gradient updates across varying output lengths. When combined, these adjustments—without any critic—yield superior reasoning accuracy compared to more intricate baselines like DAPO.

Personal Perspective

This work functions as a well-executed meta-analysis for RL in LLM reasoning—a field where inconsistent setups often obscure genuine insights. Its strength lies in reproducibility and empirical rigor: standardizing experiments across GRPO, DAPO, and PPO variations gives practitioners a rare comparative baseline. However, it remains more engineering-oriented than theoretical; the paper largely catalogs empirical behaviors without offering deeper mechanistic explanations for why certain techniques interact favorably (e.g., why global std stabilizes gradients, or why clipping scales differently across model sizes).