The Archive

BitMTP combines ultra-low-bit quantization with multi-token prediction for efficient LLM decoding without quality loss.

LoopMTP combines looped transformers with multi-token prediction to improve parameter-efficient reasoning via intermediate supervision.

Windowed-MTP optimizes speculative decoding at million-token context by eliminating full-KV attention overhead in multi-token prediction draft heads.

Gemma 4 is now significantly faster in Ollama 0.31 on Apple Silicon via multi-token prediction (MTP), powered by MLX. Performance is now up to 90% faster when used with coding agents, as measured using the Aider polyglot benchmark.

Reinforcement learning (RL) has become a key component in modern large language models, yet the rollout stage remains the key bottleneck in RL training pipelines. Although Multi-Token Prediction (MTP) offers a natural solution to accelerate rollouts through speculative decoding, many studies have observed that MTP acceptance rates degrade significantly during RL training, leading to limited speedup performance. To address this bottleneck, we present Bebop, a systematic study of MTP in LLM post-training, and offer practical recipes to integrate MTP into large-scale RL pipelines. First, we reve...

So, last week I tried to update my unused local LLM setup. I had to stop using it because quality was too low and deepseek was too cheap. First thing I stopped using Ollama and now I only use llama.cpp built in server that works really great. The quality improvement from Q4 to Q6 is outstanding and finally a local LLM server can work very similarly to paid APIs. That's great! And MTP makes a big performance gain, on a dual 3090 (downvolted and limited to 65°C) it generates from 20 to 50 tokens per second with minimal heat generation. So yes, that time has finally arrived! Local coding age...

I picked up a 7900 XTX earlier which runs qwen3.6-27b fine, but not to my like. Its compute performance is quite unstable for me. With MTP the decode speed can reach 40-60 t/s, but prefill is just too slow. Regardless of whether I used ROCm or Vulkan, the prefill speed varies between 300t/s and 500 t/s, even with very long prompts. I've been itching to try out an ultra-budget 24GB setup using dual 3060s. I managed to snag a second 3060 at a reasonable price in last few days. So I took out the 7900 XTX, installed the 3060s, and began testing. # Test Configuration * **Test Platform:** i7 477...

Long chain-of-thought reasoning has made autoregressive decoding the dominant inference cost of modern large language models. Existing methods target either the input side (latent compression) or the output side (speculative decoding and multi-token prediction, MTP), but the two lines of work have been pursued independently. Moreover, output-side methods must incur an expensive verifier pass to validate the unreliable draft tokens predicted by MTP. To address these issues, we propose \textbf{Pair-In, Pair-Out (PIPO)}, which unifies both sides by viewing a latent compressor and an MTP head as ...

Safetensors, llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved: [https://huggingface.co/llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved](https://huggingface.co/llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved) GGUFs, llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved-GGUF [https://huggingface.co/llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved-GGUF](https://huggingface.co/llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved-GGUF) NVFP4, llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-N...

Community finetune of Qwen 3.6 35B with quantized weights; testing on consumer hardware shows stability at 200k context.

llama.cpp b9274 fixes VRAM leak in speculative decoding by properly freeing draft context and decoder resources on server sleep.

User reports 110 tok/s inference speed on Qwen 35B with 12GB VRAM using ik_llama.cpp, outperforming standard llama.cpp after MTP merge.

llama.cpp PR #23287 optimizes MTP (multi-token prediction) draft sampling by moving logic to backend, improving inference performance.

ByteShape releases Qwen 3.6 35B GGUF quantizations in NTP and MTP variants with empirical comparisons across GPU/CPU hardware.

Early-stage Gemma 4 MTP compilation work-in-progress shared on LocalLLaMA.

Qwen 3.6 35B MoE benchmark on RTX 5080: 56 tok/s at 128k context; Multi-Token Prediction offers no speed gain at scale.

LM Studio 0.4.14 adds MTP Speculative Decoding support via llama.cpp 2.15.0 for faster inference.

llama.cpp PR #23269 introduces MTP (Multi-Token Prediction) improvements for faster local LLM inference.

llama.cpp MTP speculative decoding merged; Qwen 27B/35B inference speedups 1.24–2.44× on consumer GPUs/APUs with quantization.

Llama.cpp performance improved 1.5–1.8x in recent updates; MTP and prompt processing issues partially resolved.

MTP KV cache quantization in llama.cpp Qwen models reduces VRAM overhead without apparent inference degradation.

llama.cpp PR #23198 optimizes prompt decode by avoiding logits copying in MTP, improving inference speed.

Llama.cpp multi-token prediction on Qwen 3.6 27B shows 42% prefill slowdown but 85% token generation speedup on RTX 3090.

Benchmarking llama.cpp MTP (multi-token prediction) on Qwen 3.6 with RTX 5090, comparing inference speed with draft-mtp flag toggled.

llama.cpp release b9180 ships MTP support, enabling improved inference optimization for local LLM deployment.

Community discussion of Qwen 27B quantization and inference optimization on single RTX 3090 GPU.

Qwen3.6 27B-MTP shows 11.5% faster wall-clock time vs base on single-turn; 35B-MTP regresses 11.2%, with generation speedups offset by prompt processing slowdowns.

MTP support merged into llama.cpp master branch (PR #22673); no context on feature impact.

MTP PR merged into llama.cpp; technical details absent.

Looks like it finally happens... MTP getting approved for llama.cpp. Time to prepare for the update.