The Archive
Search the full wire by company, model, lab, or keyword. Every story we have ever aggregated.
That's a good news...
Looks like it finally happens... MTP getting approved for llama.cpp. Time to prepare for the update.
Built a fully offline suitcase robot around a Jetson Orin NX SUPER 16GB. Gemma 4 E4B, ~200ms cached TTFT, 30+ sensors, no WiFi/BT/cellular. He has opinions.
Engineer deployed Gemma 4 E4B on Jetson Orin NX with 30+ sensors, 200ms TTFT, offline multimodal robotics stack using llama.cpp and Piper TTS.
club-5060ti: practical RTX 5060 Ti local LLM notes and configs
RTX 5060 Ti local LLM configuration guide covering vLLM and llama.cpp serving of Qwen 27B/35B models with quantization and long-context presets.
Llama-Studio, WebUI for llama-server Management
Llama-Studio: open-source WebUI for managing multiple local llama-server instances and configurations.
Automated AI researcher running locally with llama.cpp
Hugging Face releases ml-intern, an agent framework for local LLM research automation supporting llama.cpp/ollama with Qwen and Claude models.
Multi-Token Prediction (MTP) for Qwen on LLaMA.cpp + TurboQuant
Multi-token prediction + TurboQuant quantization achieves 40% throughput gain (21→34 tokens/s) on Qwen 27B/35B via LLaMA.cpp on M-series Mac.
24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context)
Qwen 3.6 35B and Gemma 4 26B MoE models achieve 20–24.5 tok/s on GTX 1080 with 128k context via llama.cpp quantization.
llama.cpp docker images to run MTP models
Community Docker images for llama.cpp with MTP support to simplify local model inference setup across CUDA versions.
Is using vLLM actually worth it if you aren't serving the model to other people?
Reddit discussion comparing vLLM vs llama.cpp for single-user local inference on AMD GPUs.
PSA: If your project has an ANTHROPIC_API_KEY in any .env file, Claude Code will silently bill your API account instead of your Max plan — Anthropic calls it "intentional functionality"
r/ClaudeAI • also crosspost to r/LocalLLaMA and r/artificial I lost $187 to this and want to save others the same headache. **What happened** I run Claude Code headlessly via Windows Task Scheduler. My project repo has a `.env` file with `ANTHROPIC_API_KEY` set — legitimately, for a separate Express server doing AI-based transaction classification. Nothing to do with Claude Code itself. Claude Code reads environment variables from the `.env` in its working directory on launch. When it finds `ANTHROPIC_API_KEY` there, it silently uses that key for billing instead of your OAuth ...
Luce DFlash + PFlash on AMD Strix Halo: Qwen3.6-27B at 2.23x decode and 3.05x prefill vs llama.cpp HIP
Luce ships DFlash+PFlash optimizations for AMD Ryzen AI MAX+ 395, achieving 2.23x decode speedup on Qwen 3.6-27B vs llama.cpp HIP.
examples : add llama-eval by ggerganov · Pull Request #21152 · ggml-org/llama.cpp
llama.cpp adds llama-eval benchmarking tool supporting AIME, GSM8K, GPQA for local quantized model evaluation.
Stop wasting electricity
RTX 4090 power optimization for llama.cpp: reduce consumption 40% via power limits without performance loss.
Drastically improve prompt processing speed for --n-cpu-moe partially offloaded models
Tuning llama.cpp ubatch and n-cpu-moe parameters improves gpt-oss-120b prompt processing from 240 to higher tok/s on RTX 3090.
I catalogued every way local models break JSON output and built a repair library, here's what I found across 288 model calls
Empirical study across 288 model calls identifying JSON output failures in Llama 3, Mistral, Command R, DeepSeek, Qwen; failure modes consistent across open and closed models but vary by rate.
PSA: Watch out for extra spaces in chat-template-kwargs when using Qwen3.6 with llama-server
JSON parsing bug in llama-server's chat-template-kwargs for Qwen3.6 preserve_thinking parameter requires whitespace-free formatting.
ExLlamaV3 Major Updates!
ExLlamaV3 adds Gemma 4 support, improved caching, and DFlash optimization for faster LLM inference on consumer hardware.
I have DeepSeek V4 Pro at home
User successfully quantized and ran DeepSeek V4 Pro locally on AMD EPYC + RTX PRO hardware using modified llama.cpp with Q4_K_M compression.
Running Minimax 2.7 at 100k context on strix halo
User shares llama-server configuration for running Minimax 2.7 at 100k context on Strix Halo hardware with detailed tuning notes.
I am overwhelmed by Harnesses
Reddit user seeks advice on LLaMA inference harnesses; discusses fragmentation and compatibility issues with local LLM tooling.
BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)
BeeLlama.cpp fork adds DFlash, TurboQuant, and vision support; runs Qwen 3.6 27B Q5 on RTX 3090 with 200k context at 135 tps.
More Qwen3.6-27B MTP success but on dual Mi50s
User reports 1.5–2x speedup running Qwen 27B with MTP optimization on dual AMD MI50 GPUs via llama.cpp.
80 tok/sec and 128K context on 12GB VRAM with Qwen3.6 35B A3B and llama.cpp MTP
User achieves 80 tok/sec with 128K context on RTX 4070 Super using Qwen3.6 35B quantization and llama.cpp MTP implementation.
How long for llama.cpp official support of MTP?
Reddit user asks about llama.cpp timeline for Vulkan/HIP MTP support on Strix Halo Windows 11.
Where's the Plan? Locating Latent Planning in Language Models with Lightweight Mechanistic Interventions
Linear probing and activation patching reveal latent planning representations in Qwen3, Gemma-3, Llama-3; future constraints encoded at layer boundaries.
Multi-Token Prediction (MTP) for LLaMA.cpp - Gemma 4 speedup by 40%
Multi-Token Prediction optimization for LLaMA.cpp achieves 40% speedup on Gemma 4 quantized models via parallel token drafting.
guess what? if you are a chrome user, technically you are localllama member!
Chrome allegedly downloads 4GB LLM checkpoint without user consent, raising privacy and transparency concerns for browser-embedded AI.
feat: Add Mimo v2.5 model support by AesSedai · Pull Request #22493 · ggml-org/llama.cpp
Xiaomi releases Mimo v2.5, a 310B sparse MoE multimodal model with 1M token context supporting text, image, video, and audio.
Running Qwen3.5 / Qwen3.6 with NextN MTP (Multi-Token Prediction) speculative decode in llama.cpp — single RTX 3090 Ti GPU guide
Qwen3.5/3.6 inference optimization guide: NextN MTP speculative decoding achieves 2.9× speedup on RTX 3090 Ti via llama.cpp with zero quality loss.