Llama

This study investigates cross-lingual distributional skew (the Shibboleth Effect) in frontier large language models (LLMs) subjected to sustained adversarial conditions. We develop a multi-agent geopolitical wargame, the Cerulean Sea Crisis, a synthetic maritime territorial dispute designed to mirror the structural dynamics of Eastern Mediterranean conflicts. Six frontier models (GPT-4o, Llama-4, Mistral-Large, Gemini-3.1-Pro, Qwen3.6-Plus, and DeepSeek-R1) participate in a between-groups experiment (N = 10 games per arm, K = 5 rounds per game) in which the sole manipulation is the language o...

AutoMegaKernel (AMK) compiles a HuggingFace Llama-family model into a single persistent cooperative CUDA kernel that runs the whole forward pass in one launch, with no per-model hand-written CUDA. The contribution is the system, not raw speed. A frozen schedule-IR validator statically certifies deadlock-freedom and race-freedom via static graph checks (not a mechanized proof), so an unsafe agent-proposed schedule is rejected before launch: across 7,160 adversarial schedules (6,091 unsafe) it had zero false-accepts and accepted all 360 real lowerings. The same source retargets sm_80/sm_90/sm_1...

We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner that ensures stable weight conditioning throughout LLM training. The PC module reshapes the singular-value spectrum of weight matrices via low-degree polynomial preconditioning. After training, the preconditioned weights can be merged back into the original architecture, incurring no inference overhead. We demonstrate the advantage of the proposed PC layer over standard transformers in Llama-1B pre-training, for both the AdamW and Muon optimizers. Theoretically, we justify this spectrum-control pr...

AI-assisted coding agents are bottlenecked by input-token cost. Two pathologies of raw human input drive much of this overhead: tokenization inefficiency for non-English text and structural entropy in conversational prompts. Existing approaches act reactively by compressing already-bloated contexts or intervening after failures occur. We introduce a pre-flight, edge-side prompt-rewriting middleware that operates between the developer and the cloud agent. A local Llama 3.2 (3B) model performs cross-lingual translation into English, structural rewriting into a compact task-oriented format, and ...

Temperature-zero BF16 LLM inference is often treated as reproducible, yet the same request can emit different tokens when decoded alone or inside a larger batch. Existing fixes use batch-invariant operators or LLM-42's per-token verification, incurring cost even when most steps are stable. We ask whether verification can be applied exclusively to flipped tokens. Across five models, batch-induced token flips are sparse on the flip-rate benchmarks: on MATH500, Llama-3.1-8B flips on $0.48\%$ of synchronous decode steps, and all tested models stay within the 0.3-1.3% range on MATH500, GSM8K, and ...

So, last week I tried to update my unused local LLM setup. I had to stop using it because quality was too low and deepseek was too cheap. First thing I stopped using Ollama and now I only use llama.cpp built in server that works really great. The quality improvement from Q4 to Q6 is outstanding and finally a local LLM server can work very similarly to paid APIs. That's great! And MTP makes a big performance gain, on a dual 3090 (downvolted and limited to 65°C) it generates from 20 to 50 tokens per second with minimal heat generation. So yes, that time has finally arrived! Local coding age...

Here's my article with **38 quant pairs** thoroughly benchmarked in KLD with **3 different Qwen 3.6 27B configs**: Q5\_K\_S + 64k context, IQ4\_XS + 64k context, IQ4\_XS + 128k context. This allows us to track not only how cache quantizations affects the precision in a vacuum, but also how it interacts with noise from the model itself. All benchmarks were done using my [BeeLlama.cpp](https://github.com/Anbeeld/beellama.cpp) fork, allowing to include a number of quant types that are not present in mainline llama.cpp: vanilla TurboQuant, TCQ 3-bit/2-bit, and q6\_0. [https://anbeeld.com/articl...

[Cuda 13.3 Downloads](https://developer.nvidia.com/cuda-downloads) [Release Notes](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) Anybody already tried llama.cpp with 13.3?

Here's the PR by pedapudi. https://github.com/ggml-org/llama.cpp/pull/21344 It's merge request has been denied so it will not be in mainline llama.cpp. The changes are so small that I just put them into whatever the current release of llama.cpp is. Read the PR for more info. It will only work with MOEs. Also, it gives the most boost at low context. As the context rises, the gain diminishes. Pedapudi explains why that happens in the PR. Here are some numbers. It really works well. The tiny amount of time it takes me to apply the code to the current release of llama.cpp is time well spent. ...

I've fine-tuned Qwen 3.5 0.8B on the dataset provided by Pangram with their EditLens paper. It's available via a [Chrome extension](https://chromewebstore.google.com/detail/slop-hammer/gfjdmhfokmhedlgfggmmgchpppmhkdgg); you can just click selected text and it's going to give you the probability distribution of how likely it is AI-generated. It takes under 1s on my M1 MacBook Pro. Pangram did release Llama 3.2 3B trained on their dataset, but I found this model slightly too legacy (too big for the capabilities). Qwen 0.8B (base) ended up being as good after roughly 20h of fine-tuning on a sin...

llama.cpp adds fast Walsh-Hadamard transform (FWHT) for CUDA, yielding 1–2% prompt-processing and 7–9% token-generation speedups with quantized KV-cache.

Financial Times reports Heretic tool removes guardrails from Meta's Llama 3.3 in <10 minutes; 3,500+ decensored variants downloaded 13M times.

llama.cpp PR addresses checkpoint creation inefficiency when context optimization tools modify conversation history in agentic workflows.

Community discussion on local LLM frontends and limitations of llama-server as default interface.

Benchmark of vision LLMs vs. OCR pipelines on 30 long, image-heavy PDFs from MMLongBench-Doc shows LlamaCloud + Azure premium achieving 59.6%–58.5% accuracy; agentic RAG and native PDF vision approaches compared on cost and accuracy.

llama.cpp server adds native tool support (shell execution, file ops) via experimental --tools flag.

Wanted to share a workflow I tested on a real flight, in case anyone else is trying to set up offline Claude Code. The core idea: using ollama to pull the needed model of what you need, and then use it to run claude code The setup, in order: 1. Pull a model on home wifi the night before. \`ollama pull &lt;model&gt;\` — \~9 GB for a 14B, \~17 GB for a 26B. Don't try this at the gate. 2. In Claude Code, point at Ollama. The cleanest path I found is wrapping it in two aliases: alias claude-local='ollama launch claude --model gemma4:26b' alias claude-cloud='claude' 3. Verify on the ground with...

User reports Apex quantization for Gemma 4 26B achieves 38 tokens/sec at 90K context on RX 9060 XT with llama.cpp.

User configures dual AMD RDNA GPUs (48GB VRAM) with llama-cpp via Vulkan for local inference.

BeeLlama v0.2.0 achieves 4-5x token throughput gains on RTX 3090 via DFlash optimizations for Qwen 27B and Gemma 31B models.

Community fork of llama.cpp optimizes MoE inference on 12GB VRAM by loading only active experts rather than full layers.

cHunter789 releases Qwen-27B IQ4_KS quantization (14.1GB) optimized for 16GB NVIDIA GPUs via ik_llama.cpp.

SupraLabs released Supra-50M, a 50M-parameter Llama-style language model trained on 20B educational tokens with competitive benchmark performance.

llama.cpp b9274 fixes VRAM leak in speculative decoding by properly freeing draft context and decoder resources on server sleep.

Qwen 3.7 open-weights model released; community discussion on LocalLLaMA highlights adoption momentum.

llama.cpp PR #22929 fixes prompt processing performance issue affecting OpenCode and Pi model inference.

User reports 110 tok/s inference speed on Qwen 35B with 12GB VRAM using ik_llama.cpp, outperforming standard llama.cpp after MTP merge.

User shares llama.cpp configuration for running Qwen 3.6 27B locally with ROCm acceleration and optimizations.

llama.cpp PR #23287 optimizes MTP (multi-token prediction) draft sampling by moving logic to backend, improving inference performance.

Early-stage Gemma 4 MTP compilation work-in-progress shared on LocalLLaMA.

LM Studio 0.4.14 adds MTP Speculative Decoding support via llama.cpp 2.15.0 for faster inference.

llama.cpp PR #23269 introduces MTP (Multi-Token Prediction) improvements for faster local LLM inference.

llama.cpp MTP speculative decoding merged; Qwen 27B/35B inference speedups 1.24–2.44× on consumer GPUs/APUs with quantization.

Just wanted to share that I'm pretty happy about Qwen 35b a3b agentic coding performance. I'm running the model in q80 quant, kv cache both q8\_0 as well, with 262144 in 4090 + 5060 ti, via llama.cpp backend with claude code pointing to localhost. For demo/data analytics purposes, it works pretty well. I haven't used it for large codebases, but it definitely is better than gemma4 26b in my use case. One thing that surprises me is that it seems to get better outcome in agentic coding, than chat. When using it with just chat UI, i found the code qwen35b provide a bit too clunky. I wonder o...

Llama.cpp performance improved 1.5–1.8x in recent updates; MTP and prompt processing issues partially resolved.

MTP KV cache quantization in llama.cpp Qwen models reduces VRAM overhead without apparent inference degradation.

OpenBMB releases BitCPM4-CANN family (1B–8B params) with BitNet quantization; awaiting llama.cpp support.

Qwen 3.6 27B performance benchmarks across llama.cpp backends on RTX 3090: ik_llama.cpp achieved 1261 tok/s prefill, 72.9 tok/s decode with 156k context.

llama.cpp PR #23198 optimizes prompt decode by avoiding logits copying in MTP, improving inference speed.

llama.cpp fork adds quantized KV cache support for tensor parallelism across dual GPUs, addressing long-standing inference bottleneck.

Llama.cpp multi-token prediction on Qwen 3.6 27B shows 42% prefill slowdown but 85% token generation speedup on RTX 3090.

Benchmarking llama.cpp MTP (multi-token prediction) on Qwen 3.6 with RTX 5090, comparing inference speed with draft-mtp flag toggled.

User seeks faster inference alternatives to Ollama/LM Studio for local model serving (Gemma, Qwen, OpenBioLLM) on 64GB RAM.

llama.cpp release b9180 ships MTP support, enabling improved inference optimization for local LLM deployment.

Community discussion of Qwen 27B quantization and inference optimization on single RTX 3090 GPU.

Qwen3.6 27B-MTP shows 11.5% faster wall-clock time vs base on single-turn; 35B-MTP regresses 11.2%, with generation speedups offset by prompt processing slowdowns.

MTP support merged into llama.cpp master branch (PR #22673); no context on feature impact.

MTP PR merged into llama.cpp; technical details absent.

Looks like it finally happens... MTP getting approved for llama.cpp. Time to prepare for the update.

Engineer deployed Gemma 4 E4B on Jetson Orin NX with 30+ sensors, 200ms TTFT, offline multimodal robotics stack using llama.cpp and Piper TTS.

RTX 5060 Ti local LLM configuration guide covering vLLM and llama.cpp serving of Qwen 27B/35B models with quantization and long-context presets.

Llama-Studio: open-source WebUI for managing multiple local llama-server instances and configurations.

Hugging Face releases ml-intern, an agent framework for local LLM research automation supporting llama.cpp/ollama with Qwen and Claude models.

Multi-token prediction + TurboQuant quantization achieves 40% throughput gain (21→34 tokens/s) on Qwen 27B/35B via LLaMA.cpp on M-series Mac.

Qwen 3.6 35B and Gemma 4 26B MoE models achieve 20–24.5 tok/s on GTX 1080 with 128k context via llama.cpp quantization.

Community Docker images for llama.cpp with MTP support to simplify local model inference setup across CUDA versions.

Reddit discussion comparing vLLM vs llama.cpp for single-user local inference on AMD GPUs.

r/ClaudeAI • also crosspost to r/LocalLLaMA and r/artificial I lost $187 to this and want to save others the same headache. **What happened** I run Claude Code headlessly via Windows Task Scheduler. My project repo has a `.env` file with `ANTHROPIC_API_KEY` set — legitimately, for a separate Express server doing AI-based transaction classification. Nothing to do with Claude Code itself. Claude Code reads environment variables from the `.env` in its working directory on launch. When it finds `ANTHROPIC_API_KEY` there, it silently uses that key for billing instead of your OAuth ...

Luce ships DFlash+PFlash optimizations for AMD Ryzen AI MAX+ 395, achieving 2.23x decode speedup on Qwen 3.6-27B vs llama.cpp HIP.

llama.cpp adds llama-eval benchmarking tool supporting AIME, GSM8K, GPQA for local quantized model evaluation.

RTX 4090 power optimization for llama.cpp: reduce consumption 40% via power limits without performance loss.

Tuning llama.cpp ubatch and n-cpu-moe parameters improves gpt-oss-120b prompt processing from 240 to higher tok/s on RTX 3090.

Empirical study across 288 model calls identifying JSON output failures in Llama 3, Mistral, Command R, DeepSeek, Qwen; failure modes consistent across open and closed models but vary by rate.

JSON parsing bug in llama-server's chat-template-kwargs for Qwen3.6 preserve_thinking parameter requires whitespace-free formatting.

ExLlamaV3 adds Gemma 4 support, improved caching, and DFlash optimization for faster LLM inference on consumer hardware.

User successfully quantized and ran DeepSeek V4 Pro locally on AMD EPYC + RTX PRO hardware using modified llama.cpp with Q4_K_M compression.

User shares llama-server configuration for running Minimax 2.7 at 100k context on Strix Halo hardware with detailed tuning notes.

Reddit user seeks advice on LLaMA inference harnesses; discusses fragmentation and compatibility issues with local LLM tooling.

BeeLlama.cpp fork adds DFlash, TurboQuant, and vision support; runs Qwen 3.6 27B Q5 on RTX 3090 with 200k context at 135 tps.

User reports 1.5–2x speedup running Qwen 27B with MTP optimization on dual AMD MI50 GPUs via llama.cpp.

User achieves 80 tok/sec with 128K context on RTX 4070 Super using Qwen3.6 35B quantization and llama.cpp MTP implementation.

Reddit user asks about llama.cpp timeline for Vulkan/HIP MTP support on Strix Halo Windows 11.

Linear probing and activation patching reveal latent planning representations in Qwen3, Gemma-3, Llama-3; future constraints encoded at layer boundaries.

Multi-Token Prediction optimization for LLaMA.cpp achieves 40% speedup on Gemma 4 quantized models via parallel token drafting.

Chrome allegedly downloads 4GB LLM checkpoint without user consent, raising privacy and transparency concerns for browser-embedded AI.

Xiaomi releases Mimo v2.5, a 310B sparse MoE multimodal model with 1M token context supporting text, image, video, and audio.

Qwen3.5/3.6 inference optimization guide: NextN MTP speculative decoding achieves 2.9× speedup on RTX 3090 Ti via llama.cpp with zero quality loss.

User reports successful local inference with Qwen3.6-35B on AMD R9700 GPU, generating functional code and tests via llama-cpp.

Qwen3.6-27B with Multi-Token Prediction achieves 2.5x throughput via Unsloth quantization and llama.cpp integration.

Qwen 3.6 27B achieves 2.5x inference speedup via MTP speculative decoding in llama.cpp; 262k context on 48GB with fixed chat templates.

Qwen 27B achieves 54 t/s on V100 GPU with MTP optimization in llama.cpp, nearly 2x baseline speed for code review and tool use tasks.

Cyera reports critical unauthenticated memory leak vulnerability in Ollama enabling unauthorized data access.

User reports successful MTP speculative decoding on AMD Strix Halo (AI Max 395) with llama.cpp achieving 60-80 tok/s on Qwen 3.6B GGUF.

Meta is facing a class action lawsuit filed by five major book publishers and one author over claims the company "engaged in one of the most massive infringements of copyrighted materials in history" when training its Llama AI models, as reported earlier by The New York Times. In their suit, Macmillan, McGraw-Hill, Elsevier, Hachette, Cengage, and author Scott Turow allege that Meta "repeatedly copied" their books and journal articles without permission. The lawsuit accuses Meta of knowingly ripping copyrighted work from "notorious pirate sites," such as LibGen, Anna's Archive, Sci-Hub, Sci-M...

Unexpected email to wake up to but I am here for it! Model agnostic tools are the way! This is huge!

MTP format support coming to llama.cpp; DeepSeekv3, Qwen3.5, GLM4.5, and other models compatible pending native weights.

FastDMS achieves 6.4× KV-cache compression on Llama 3.2 1B via learned token eviction, matching vLLM performance with lower memory overhead.

llama.cpp adds beta MTP (Multi-Token Prediction) support, starting with Qwen3.5, closing performance gap with vLLM on token generation.

Quantized Llama 405B and DeepSeek models now achieve 20-100 tokens/sec on consumer hardware, up from 1 token/sec two years ago.

Developer reports local Qwen 27B setup with llama-server now competitive with Claude Code and Cursor for coding tasks, driven by cloud provider cost increases.

Community tool adds phrase-filtering capability to llama.cpp inference engine via GitHub script.

User shares Qwen3.6-27B quantized setup with RTX 5090 and llamacpp configuration parameters.

r/LocalLLaMA moderators report positive community response to new rules reducing spam after one week.

PFlash: speculative prefill technique achieves 10x speedup on 128K context with quantized 27B models on RTX 3090, open-source C++/CUDA implementation.

gemma-4-31B-it-DFlash open-weights model released on Hugging Face, pending llama.cpp integration.

User demonstrates DFlash speculative decoding in llama.cpp with Qwen3.5-35B-A3B on RTX 2080 SUPER 8GB, achieving inference on VRAM-constrained hardware.

llama-swap adds matrix grouping feature for multi-model orchestration and intelligent VRAM swap scheduling.

Developer built local PDF-to-audiobook app using Kokoro 82M TTS, Qwen, and llama.cpp with Tauri 2.0 on M1 Mac.

PS5 Linux exploit proposed as potential hardware for local LLM inference via llama.cpp.

IK_LLAMA.cpp adds Qwen3.5 MTP support with 50% throughput gain (18-20→30 tok/s) via pipeline parallelism on 27B model.