The Archive

NVIDIA Nemotron 3.5 Super VL is a 120B-parameter mixture-of-experts model that pairs reasoning and tool use with multimodal understanding. This post explores its architecture and use cases, then shows how to deploy it on Amazon EKS, EKS Auto Mode, and Amazon SageMaker HyperPod EKS with NVIDIA Dynamo and vLLM.

Sequential Isolation Methodology reduces LLM inference benchmark variance via controlled regression testing on vLLM across GPU concurrency levels.

py-kvcache characterizes external KV cache tradeoffs in vLLM across GPU/CPU/NVMe for long-context LLM serving optimization.

Cohere releases North Mini, a decode megakernel LLM serving engine achieving 1.58× speedup over vLLM in production.

Cohere releases W4A8 quantization with vLLM integration; compares W4A16 and W8A8 schemes on NVIDIA Hopper.

vLLM-based inference pipeline for unified audio understanding and generation with native support for multi-token prediction and delay-pattern interleaving.

AI agents automate vLLM fork maintenance, cutting upstream sync time from weeks to days by auto-rebasing, testing, and fixing conflicts.

Context-heavy agents place unusual pressure on the key-value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving system can keep GPUs utilized. We study 4-bit KV-cache compression for this setting, using TurboQuant-style rotation and codebook quantization as a quality anchor and vLLM FP8 KV caching as the deployment anchor. We report three contributions. First, we frame 4-bit KV caching around multi-round agent workloads where task quality, cache residency, and serving throughput must be measured jointly. Second, we describe the pract...

Visual information helps resolve ambiguity in coreference resolution, leading to notable performance gains. However, existing Multi-modal Coreference Resolution (MCR) methods require training with (partially) annotated data from the target dataset before they can be applied, preventing their direct usability and raising concerns about generalization. While Vision-Language Large Models (VLLMs) with billions of parameters offer promising zero-shot capabilities, they remain largely inaccessible. Their massive size limits deployability, and many are only accessible through paid APIs. In this pape...

As scalable inference services become popular, the cold start latency of an inference engine becomes important. Today, vLLM has evolved into the de facto inference engine of choice for many inference workloads. Although popular, due to its complexity and rapid evolution, there has not been a systematic study of its startup latency. With major architectural innovations such as the V1 API and the introduction of torch.compile, this paper presents the first detailed performance characterization of vLLM startup latency. We break down the startup process into six foundational steps and demonstrate...

Worth taking a look to see if this affects any of you. Surprised nobody has posted it yet.

Qwen 3.6 27B performance benchmarks across llama.cpp backends on RTX 3090: ik_llama.cpp achieved 1261 tok/s prefill, 72.9 tok/s decode with 156k context.

RTX 5060 Ti local LLM configuration guide covering vLLM and llama.cpp serving of Qwen 27B/35B models with quantization and long-context presets.

Shipped this for the AMD x lablab hackathon. Attached video is one of the actual reels the pipeline produced - one English sentence in, finished mp4 with characters, story, music, and voice-over out (fast demo video, not the best quality). ~45 minutes end-to-end on a single AMD Instinct MI300X. Every model is Apache 2.0 or MIT. **Pipeline (8 stages, all sequential on the same GPU):** 1. **Director Agent** - Qwen3.5-35B-A3B (vLLM + AITER MoE) plans 6 shots from one sentence, returns structured JSON with character bibles, shot prompts, music brief, per-shot voice-over script, narration langua...

Reddit discussion comparing vLLM vs llama.cpp for single-user local inference on AMD GPUs.

Benchmark comparing Gemma 4 multi-token prediction vs. DFlash speculative decoding on H100 using vLLM and SPEED-Bench dataset.

DeepSeek-V4-Flash W4A16+FP8 with retrofitted MTP self-speculation achieves 85.5 tok/s @ 524k context on dual RTX PRO 6000 Max-Q via vLLM patching.

Lemonade adds vLLM ROCm as experimental backend, enabling direct .safetensors inference without GGUF conversion.

Gemma 4 26B achieves 578 tok/s on RTX 5090 using DFlash speculative decoding in vLLM, 2.5× faster than baseline.

User demonstrates Qwen3.6 27B running 200k context on single RTX 5090 with NVFP4 quantization in vLLM, sharing exact configuration and parameters.

vLLM merged TurboQuant quantization support for Qwen 3.5+, enabling 4-bit/3-bit KV-cache inference via new command-line flags.

FastDMS achieves 6.4× KV-cache compression on Llama 3.2 1B via learned token eviction, matching vLLM performance with lower memory overhead.

llama.cpp adds beta MTP (Multi-Token Prediction) support, starting with Qwen3.5, closing performance gap with vLLM on token generation.

27B/31B vision model comparison: Qwen 3.6 benchmarks higher than Gemma 4 but underperforms in real-world tasks; suggests benchmark gaming.

Qwen3.6-27B Windows native vLLM launcher achieves 72 tok/s on RTX 3090 with portable installation, no WSL.

Intel releases AutoRound, a low-bit quantization algorithm optimized for CPU/XPU/CUDA with vLLM and Transformers compatibility.

Community member achieves 9.7 tok/s output throughput on Kimi K2.6 with 32x AMD MI50 GPUs using vLLM fork.