The Archive

Link: github.com

Multi-token prediction + TurboQuant quantization achieves 40% throughput gain (21→34 tokens/s) on Qwen 27B/35B via LLaMA.cpp on M-series Mac.

Qwen 3.6 27B achieves 52.8 tokens/sec throughput on MI50 GPU with full precision, enabling on-device agent inference.

Community Docker images for llama.cpp with MTP support to simplify local model inference setup across CUDA versions.

Benchmark comparing Gemma 4 multi-token prediction vs. DFlash speculative decoding on H100 using vLLM and SPEED-Bench dataset.

Unsloth released GGUF quantizations of Qwen3.6 27B and 35B models with MTP training.

Benchmark study of 300+ tests shows speculative inference speeds up coding tasks but slows creative text generation; task type dominates performance.

DeepSeek-V4-Flash W4A16+FP8 with retrofitted MTP self-speculation achieves 85.5 tok/s @ 524k context on dual RTX PRO 6000 Max-Q via vLLM patching.

User reports 1.5–2x speedup running Qwen 27B with MTP optimization on dual AMD MI50 GPUs via llama.cpp.

User achieves 80 tok/sec with 128K context on RTX 4070 Super using Qwen3.6 35B quantization and llama.cpp MTP implementation.

Reddit user asks about llama.cpp timeline for Vulkan/HIP MTP support on Strix Halo Windows 11.

MTP speculative decoding shows task-dependent gains: 1.53× speedup on code, 0.5× slowdown on JSON; effectiveness tied to draft acceptance rate.

Developer achieves 80+ tokens/sec on Qwen 3.6-27B with MTP + TurboQuant quantization on single RTX 4090 at 262K context.

Z-lab releases Gemma-4-26B with DFlash inference optimization, claiming improved performance over MTP via stateful parallel block diffusion.

Benchmark shows TP=2 pinned to NVLink GPU pairs yields +25–53% throughput vs PCIe on Qwen 3.6 27B; TP=4 degrades performance due to cross-pair PCIe bottleneck.

Multi-Token Prediction optimization for LLaMA.cpp achieves 40% speedup on Gemma 4 quantized models via parallel token drafting.

Qwen3.5/3.6 inference optimization guide: NextN MTP speculative decoding achieves 2.9× speedup on RTX 3090 Ti via llama.cpp with zero quality loss.

Community fine-tune of Qwen 3.6 27B with reduced safety filters released in multiple quantization formats.

User achieves 50 tokens/sec with Qwen 3.6 27B on RTX 3090 using MTP speculative decoding at 100k context.

MTP speculative decoding ported to Qwen 3.6 35B shows modest 2.5-6% speedup vs. 2-2.5x on 27B; architecture may limit gains.

User demonstrates Qwen3.6 27B running 200k context on single RTX 5090 with NVFP4 quantization in vLLM, sharing exact configuration and parameters.

Qwen3.6-27B with Multi-Token Prediction achieves 2.5x throughput via Unsloth quantization and llama.cpp integration.

Qwen 3.6 27B achieves 2.5x inference speedup via MTP speculative decoding in llama.cpp; 262k context on 48GB with fixed chat templates.

Qwen 27B achieves 54 t/s on V100 GPU with MTP optimization in llama.cpp, nearly 2x baseline speed for code review and tool use tasks.

User reports successful MTP speculative decoding on AMD Strix Halo (AI Max 395) with llama.cpp achieving 60-80 tok/s on Qwen 3.6B GGUF.

Google releases Gemma 4 multi-token prediction drafters in 4 quantized sizes for local deployment.

MTP format support coming to llama.cpp; DeepSeekv3, Qwen3.5, GLM4.5, and other models compatible pending native weights.

MTPLX inference engine achieves 2.24× throughput speedup for MTP-equipped models on Apple Silicon M5 Max via rejection sampling.

llama.cpp adds beta MTP (Multi-Token Prediction) support, starting with Qwen3.5, closing performance gap with vLLM on token generation.

IK_LLAMA.cpp adds Qwen3.5 MTP support with 50% throughput gain (18-20→30 tok/s) via pipeline parallelism on 27B model.