The Archive

MobileSpec trains specialist LLMs on-device via forward-only optimization and RLVR, enabling adaptation without backpropagation memory overhead.

ReSI-Math reveals recoverable capability frontier in LLM reasoning failures; shows RLVR can elicit solutions via trajectory exploration even after direct failure.

Reinforcement learning with verifiable rewards (RLVR) can enable generalizable improvement for large language model (LLM) reasoning. This paper studies how to incentivize such generalization via reasoning strategies. We study this question with a controllable equation-solving framework: finding multiple inputs x for equations f(x)=y, where f is a non-bijective string function. This setting makes reasoning strategies observable while supporting a clear definition of easy-to-hard out-of-distribution (OOD) shifts, e.g., more solutions, longer y, etc. We uncover a subtle RLVR failure: it teaches ...

Group-based reinforcement learning with verifiable rewards (RLVR) learns only from prompts whose sampled completions disagree: all-pass and all-fail groups produce zero relative advantage and burn generation budget without moving the policy. Adaptive schedulers try to steer rollouts toward informative prompts, yet they rely on per-prompt pass-rate trackers that entangle two quantities evolving at very different speeds---the policy's shifting competence across reasoning skills and each prompt's intrinsic difficulty. When a single exponential average or Beta counter tracks both at once, short m...

Dataset mismatch in group relative policy optimization reduces RLVR training efficiency; selective rollout methods must account for distribution shifts.

GRIT uses gradient masking to fix visually misgrounded reasoning in vision-language models, improving RLVR task performance.

Theorem quantifies fundamental trade-off in RLVR: pass@1 improvements require proportional pass@k degradation under exclusive specialization.

Self-reinforcement in label-free RL causes policy collapse at scale due to collective instability across tasks sharing parameters.

RLVR checkpoint analysis on OLMo-3 empirically tests whether reinforcement learning induces instrumental convergence (power-seeking, deception).

HP-RLVR assigns token-level credit in single-rollout RL without group sampling by using trajectory hindsight, improving on GRPO.

Two-clock framework analyzes why larger rollout batches in RLVR slow learning but preserve collapse timing under square-root scaling.

Token-wise importance sampling for off-policy RLVR with KL regularization achieves unbiased policy gradients under coverage conditions without full-response weighting.

EDEN method improves LLM reinforcement learning with verifiable rewards by diversifying rollout trajectories via embedding-space directional exploration with anchored noise.

Decoupling exploration from optimization in RLVR improves discovery of novel reasoning strategies without degrading model quality on narrow reward signals.

Difficulty labels in RLVR are unreliable due to limited sampling; harder prompts improve at ~1/3 learnable rate, contradicting unlearnability claims.

Study of reward hacking in RLVR: characterizes when verifiers fail, shows feedback alone insufficient to detect errors, proposes audit-based correction.

ThinkPrior uses zero-rollout difficulty priors to avoid cold-start waste in RLVR prompt selection, reducing silent-group sampling from 39% to minimal.

Verifier-guided framework combines QLoRA, MoE routing, and symbolic solvers (Z3, FOL) for explainable LLM reasoning in education.

Two-stage OPD-then-RL sequential scheme outperforms joint on-policy distillation and RLVR fusion for reasoning LLM post-training on logic and math tasks.

GMTS: gradient magnitude-based token selection improves RL training for LLM reasoning tasks.

Method uses weak-model guidance to preserve generative diversity in LLM reasoning during RLVR training, addressing entropy collapse.

Comparative study of three RLVR fusion paradigms (Merge, Mix, MOPD) for consolidating domain-specific capabilities across LLM tasks.

Graph-structured online difficulty estimation optimizes RLVR exploration budgets for LLM reasoning by adaptive sample-difficulty scheduling.

Modern reinforcement learning (RL) post-training pipelines for large language models (LLMs) increasingly combine rollout workloads across multiple domains and feedback paradigms. Prefix-aware routing improves inference efficiency through cache reuse and load balancing, but it does not control how heterogeneous rollout sessions compete for KV-cache capacity. When reinforcement learning with verifiable rewards (RLVR), reinforcement learning from human feedback (RLHF), and agentic rollouts share an asynchronous inference service, their distinct sequence structures, interaction patterns, and KV-r...

Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significantly impact downstream performance. Many existing methods control exploration in the action-space, for example, using temperature scaling. However, these methods cannot reorder tokens but only influence the variance in the output distribution. This limits exploration and can lead to divergence or stalled training. Here, we investigate parameter-space exploration, where rollouts...

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level. On-policy self-distillation (OPSD) mitigates this sparsity by querying a privileged teacher at student-visited prefixes and providing dense token-level distributional supervision. Although this dense supervision alleviates signal sparsity, we find that standard OPSD still underexploits the temporal structure of the rollout. It assigns every local divergence the same c...

ISO optimization framework operationalizes spectral inheritance in RLVR, enabling weight-space updates via singular structure analysis.

Open-weight reasoning models fine-tuned via RLVR for thermal energy storage control, achieving building-scale load shifting with 30 prompts.

Adversarial generator-discriminator framework augments RLVR with learned style discriminator to prevent diversity collapse and reward hacking in LM training.

Reinforcement learning (RL) is central to aligning language models, from reinforcement learning with human feedback (RLHF) within AI assistants to newer... Reinforcement learning (RL) is central to aligning language models, from reinforcement learning with human feedback (RLHF) within AI assistants to newer reinforcement learning with verifiable rewards (RLVR) workflows for reasoning and agent tasks. RL is now becoming a practical technique for specialized AI where enterprises need more accurate agents for domain-specific workflows. Source