Spurious Advantage Hidden in GRPO
Group Relative Policy Optimization exhibits spurious advantage when guesses match correct answers in bounded-answer tasks.
Search the full wire by company, model, lab, or keyword. Every story we have ever aggregated.
Group Relative Policy Optimization exhibits spurious advantage when guesses match correct answers in bounded-answer tasks.
Self-Routing post-training framework routes LLM samples to GRPO, distillation, or regularization based on rollout correctness without external teachers.
Study evaluates when learned perceptual predictors align with human listeners in codec-based speech models using GRPO with CER constraints.
Evolution Strategies show broader reasoning coverage than GRPO in LLM post-training with lower memory overhead.
LLM learns adaptive reasoning effort allocation (NoThink/Short/Long) via GRPO to optimize test-time compute per problem difficulty.
Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy. Group-relative methods like GRPO reduce gradient variance by sampling multiple rollouts per prompt, but provide only sequence-level credit…
MCR-GRPO assigns marginal box contributions in multimodal RL for structured visual perception tasks without response-level broadcast.
ReCo reweights GRPO to reduce distributional concentration and restore reasoning coverage, addressing model collapse in post-training.
Off-Context GRPO uses privileged training information (solution prefixes) to enable RL with verifiable rewards on hard reasoning problems avoiding zero-gradient plateaus.
LongStraw: GPU-efficient execution stack for million-token RL post-training with GRPO, bridging inference context length vs. post-training gap for agents.
Controlled study finds GRPO RL fails to improve 4B–8B scale web agents over supervised baselines across learning rate and hyperparameter grids, questioning RL value at small scale.
AdaPrefix-GRPO adapts solution prefix length during training to maintain 50% success rate, improving gradient signal on hard reasoning tasks.
Asynchronous single-rollout RL for agentic LLM post-training addresses training stability and efficiency limits of batch-synchronous GRPO.
TREK uses distillation for exploration support expansion in GRPO, enabling reasoning on hard prompts via black-box or white-box teacher trajectories.
Theoretical analysis of staleness effects in asynchronous GRPO-based RLHF reveals per-step surrogacy gap and learning-rate scaling laws.
Introduces Graph-PRefLexOR, a graph-native reasoning model fine-tuned with GRPO for traceable materials discovery hypothesis generation.
Large language model (LLM)-based approaches for automated code compliance (ACC) of building regulations are prone to generating incorrect and hallucinated computer-processable rules. This paper introduces P4IR, a two-stage framework that uses supervised fine-tuning (SFT) to…
Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies. We study this question through a three-stage pipeline that segments GUI…
Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level…
Multi-turn tool-use RL is bottlenecked by the rapid depletion of informative samples in static datasets. We observe that the gradient signal in GRPO concentrates on tasks with the highest rollout reward variance, a consequence of the Popoviciu upper bound. Consequently, samples…
Long-form chain-of-thought reasoning can improve LLM performance on complex tasks, but models often continue generating unnecessary reasoning after a correct answer has emerged. We refer to this behavior as overthinking. We study this phenomenon from the perspective of…
When applying Group Relative Policy Optimization (GRPO) for GUI Grounding, rollouts are sampled from a single screenshot view; groups often become either all failures on difficult instances or all successes on easy ones, yielding no useful relative advantage. We propose VISTA…
Recent work has demonstrated that online reinforcement learning (RL) can substantially improve the quality and alignment of flow matching models for image and video generation. Methods such as Flow-GRPO and CPS cast the denoising process as a Markov Decision Process and apply…
Reinforcement learning (RL) has become a key component of post-training large language models (LLMs). In practice, LLM RL is often off-policy because of training-inference mismatch and policy staleness, making trust-region control essential for stable optimization. Mainstream…
AI red teaming must continually adapt to evolving attackers and defenders. Reinforcement learning offers a promising approach to discovering novel attacks, and co-training methods can produce more robust defenders in tandem. Recent works have demonstrated the efficacy of…
Recent advancements in reasoning language models have been driven by Reinforcement Learning (RL) fine-tuning. Most often, these rely on the Group Relative Policy Optimization (GRPO) algorithm or modifications thereof to steer the models to produce Chain-of-Thought (CoT) traces…
Post-training of language models is commonly framed as a sample-score-update loop implemented by gradient descent. A recent line of work, exemplified by RandOpt, relocates this loop to weight space, sampling Gaussian perturbations around a pretrained model and ensembling the…
Vision-language models with extended reasoning succeed on complex problems, but many real-world problems require external tools that internal reasoning alone often cannot resolve. Agentic reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the…
Peak-then-collapse failure in GRPO tool-use training on knowledge graphs: four recurring failure modes shift rather than resolve.