The Archive

RL post-training of frontier language models is increasingly bottlenecked by autoregressive rollout generation, making rollout acceleration a central systems challenge. Many existing efficiency methods improve throughput by changing the rollout or optimization regime, for example, through off-policy execution, replay, or lower-precision generation. We study speculative decoding as a lossless acceleration primitive for RL rollouts that preserves the target model's output distribution. We implement speculative decoding in NeMo-RL with a vLLM backend, supporting both synchronous and asynchronous...

Qwen 3.6 27B achieves 60 tok/s throughput and 204k context on dual RTX 5060 Ti 16GB with vLLM.

Mistral-Medium 3.5 (128B) model reference discovered in vLLM repository commit, suggesting potential unreleased weight release.

Enthusiast demonstrates multi-device local LLM inference setup using vLLM, LM Studio, and various open-weights models.

Qwen3.6-27B-INT4 achieves 100+ tokens/sec with 256k context on RTX 5090 via vLLM 0.19, with KLD quantization validation.

Qwen3.6-35B-A3B KLD divergence analysis comparing INT quantization vs NVFP4 using vLLM logit measurements on RTX 6000.

Qwen3.6-27B achieves ~80 tokens/sec at 218k context on single RTX 5090 using vLLM 0.19 with NVFP4 quantization.

Cohere preparing new MoE model with vLLM optimization support, signaling open-weights or community-accessible release.

Reddit user offers to run LLM experiments on new workplace hardware via vLLM, soliciting community requests.

Mistral shares technical deep-dive on debugging a memory leak in vLLM inference framework.