r/LocalLLaMA

looks like you can run it on any potato (A1B)! [https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF) from LiquidAI: LFM2.5 is a new family of hybrid models designed for on-device deployment. It builds on the LFM2 architecture with extended pre-training and reinforcement learning. * **On-device personal assistant**: Designed to power real-life applications, chaining tool calls, and following complex instructions on all devices. * **Compressed performance**: Competitive with much larger dense and MoE models on instruction following and agen...

Hi! Andi from Hugging Face here! My team has been working over the last few months on creating a super smooth local experience for conversations with Reachy Mini, see the video! We hope people can extend this into tons of different cool use-cases. We wrote a blog explaining how to set this up, and how to modify it for tons of different use cases. Even if you don't have a Reachy Mini, you can use this as a roadmap for amazing voice agents: [https://huggingface.co/blog/local-reachy-mini-conversation](https://huggingface.co/blog/local-reachy-mini-conversation) Hope you enjoy it!

Been following the infrastructure side of AI more lately and stumbled on this from Zai. They upgraded the network architecture on a thousand-GPU cluster running GLM-5.1 coding inference from the standard ROFT setup to something they built called ZCube, developed with Tsinghua University and HarnetsAI The numbers from production: \- Switch and optical module costs down 33% \- GPU inference throughput up 15% \- P99 tail latency on first token dropped 40.6% Same GPUs, same software stack, same model. Just the network architecture changed The actual problem they were solving is interesting....

a feature that was requested a lot: [https://huggingface.co/models?base\_model\_relation=base](https://huggingface.co/models?base_model_relation=base)

4 x RTX Pro Max-Q We will not speak about the 64GB system RAM...

# [](https://huggingface.co/Qwen/Qwen-Image-Bench#model-description)Model Description Q-Judger is a vision-language model fine-tuned specifically for automated evaluation of text-to-image generated images. Given a text prompt and a generated image, the model evaluates the image on fine-grained quality criteria organized in a 3-level hierarchy and outputs structured JSON scores. * **Base Model**: Qwen3.6-27B * **Task**: Image quality evaluation / judging * **Input**: Text prompt + generated image * **Output**: Structured JSON with per-dimension scores (0 = Fail, 1 = Pass, 2 = Excel, N/A) * *...

We went from chasing GPT4 to looking at graphs with GPT5.4 xhigh, Gemini 3.1Pro, and now Hy3 preview completely shaking up the leaderboard. Look at that CHSBO 2025 chart Hy3 preview scoring 87.8 over Gemini and GPT. What a time to be alive, but honestly, my brain can't keep up with the version numbers anymore. What's your take? Is Hy3 actually punching at this level in real-world coding/math, or is it just benchmark hardening?

Provided in both Safetensors and GGUFs. Safetensors, llmfan46/Gemma-4-Harmonia-31B-it-uncensored-heretic: [https://huggingface.co/llmfan46/Gemma-4-Harmonia-31B-uncensored-heretic](https://huggingface.co/llmfan46/Gemma-4-Harmonia-31B-uncensored-heretic) GGUFs, llmfan46/Gemma-4-Harmonia-31B-it-uncensored-heretic-GGUF: [https://huggingface.co/llmfan46/Gemma-4-Harmonia-31B-uncensored-heretic-GGUF](https://huggingface.co/llmfan46/Gemma-4-Harmonia-31B-uncensored-heretic-GGUF) Comes with benchmark too. Find all my models here: [HuggingFace-LLMFan46](https://huggingface.co/llmfan46/models) The o...

Worth taking a look to see if this affects any of you. Surprised nobody has posted it yet.

I met Katrin from Squeez Labs at an event hosted by Pathway AI (the team behind Baby Dragon Hatchling) where she told me about CrankGPT, a literally hand-cranked device for running local LLMs. It's apparently real. It's appearently launched. It's apparently glorious. Check it out at [https://crankgpt.com/](https://crankgpt.com/) \- if anyone from Squeez Labs posts here and I'm stealing their thunder, I'll take the post down! But I've been really excited about this. So local you gotta squeez it with yer own armz. ;) [https://www.youtube.com/watch?v=HSapdLYpmWY](https://www.youtube.com/watch?...

Posted this to r/MachineLearning a couple weeks ago (30K views, 100+ upvotes) and have been meaning to share it here where the fine-tuning angle is more directly relevant. I spent years building and processing a complete Usenet corpus from 1980 to 2013. Here’s why it might matter for local model work specifically: Zero AI contamination. Every post predates LLMs by decades. Training on this won’t bake in GPT mannerisms, refusal patterns, or RLHF artifacts. It’s raw human writing - argumentative, unfiltered, stylistically diverse across 33 years. Pre-SEO, pre-algorithm internet. People wrot...

So, last week I tried to update my unused local LLM setup. I had to stop using it because quality was too low and deepseek was too cheap. First thing I stopped using Ollama and now I only use llama.cpp built in server that works really great. The quality improvement from Q4 to Q6 is outstanding and finally a local LLM server can work very similarly to paid APIs. That's great! And MTP makes a big performance gain, on a dual 3090 (downvolted and limited to 65°C) it generates from 20 to 50 tokens per second with minimal heat generation. So yes, that time has finally arrived! Local coding age...

AKA: Jank Incarnate After months of pain, I finally got a working setup. There's a bunch of quirks about running a multi-Tesla setup. I was planning to write something about my experience after I get it running. Currently, the fans are plugged into the wall, speed is controlled with a knob. I still gotta wire up a PWM controller for them. EDIT: Specs: * Intel Xeon CPU E5-2680 v4 @ 2.40GHz * Asrocka x99 Extreme motherboard * Cursed 16GB DDR4 of some laptop SODIMM in an adapter * 3x Nvidia Tesla V100, 32GB - total 96GB of VRAM

I know this sub loves absurd LLM projects, so sharing my contribution while we wait for the new Qwen 3.7 models to drop! I successfully got a tiny LLM running inside an RTOS, running inside a custom-built JavaScript emulator for the Freescale ColdFire MCF5307, which is a derivative of the legendary [Motorola 68K](https://en.wikipedia.org/wiki/Motorola_68000) that powered the original Mac and Sega Genesis. The RTOS was written back in 2008 with three classmates for our embedded systems university course. It was lost to time, with the hardware and original ROM long gone. A few months ago, I d...

Link: foodtruckbench.com

Hi all, Sorry for going missing — we’ve been collecting a larger, higher-quality set of more complex tasks. We’re excited to share a major leaderboard update covering the past three months. We’ve updated the **SWE-rebench leaderboard** with **110 fresh Python tasks** from GitHub PRs created in **March, April, and part of May**. The setup follows the standard SWE-bench format: models read real PR issues, edit code, run tests, and must make the full test suite pass. This time, instead of our usual monthly updates with a smaller number of tasks, we collected a larger batch so we could evalua...

Here's my article with **38 quant pairs** thoroughly benchmarked in KLD with **3 different Qwen 3.6 27B configs**: Q5\_K\_S + 64k context, IQ4\_XS + 64k context, IQ4\_XS + 128k context. This allows us to track not only how cache quantizations affects the precision in a vacuum, but also how it interacts with noise from the model itself. All benchmarks were done using my [BeeLlama.cpp](https://github.com/Anbeeld/beellama.cpp) fork, allowing to include a number of quant types that are not present in mainline llama.cpp: vanilla TurboQuant, TCQ 3-bit/2-bit, and q6\_0. [https://anbeeld.com/articl...

A couple of recent videos I have watched : [Billionaires Are Funding 'Anti AI' Content](https://www.youtube.com/watch?v=mzlu4FSXBNw) [AI Manufactured Doubt](https://www.youtube.com/watch?v=2SjgP8o-1LQ) (long but interesting take) **My tin foil hat take** : AI Companies understand that offline llm hosting is becoming more viable for both individuals and companies. They are spreading the "AI is dangerous" message to get government regulators to pass laws to keep the people "safe" from the unbridled power of tokens and weights. They will use their lobbying with the FUD as ammunition to pass ...

Howdy everyone! Quick disclosure: I work on this - it's a project my studio created called the Null Epoch. I wasn't really happy with testing my agents with the usual static benchmarks and I wanted to learn more about how models and agents handle long-horizon planning, resource contention, and adversarial pressure over days or weeks in a more dynamic situation. I also have a particular fondness for the MUDs and text based RPGs I grew up on (really dating myself here), so the whole MMO and the open source SDK/TUI are kind of modeled after that experience. It functions as a persistent stress t...

I don't see any threads on this model. Is it because it's dense and/or without-**reasoning**? Anyone tried this for coding? >[**Capabilities**](https://huggingface.co/ibm-granite/granite-4.1-30b) Summarization Text classification Text extraction Question-answering Retrieval Augmented Generation (RAG) Code related tasks Function-calling tasks Multilingual dialog use cases Fill-In-the-Middle (FIM) code completions Some people prefer dense in this model size range(Ex: 27B over 35B-A3B). Still no feedbacks from them here. I know that some people love Granite models. Myself...

This may be a controversial take, but AI is not for everyone. I've made a post here before about the vibecoded garbage I see on this subreddit every time I click on it but there seems to be a larger issue. AI isn't just a set and forget karma farm. You actually have to put work in to contribute to the betterment of this subreddit and local AI. I see a lot of posts written only by AI, and unless it translates for you, you have NO excuse. Your posts written by AI, and your projects vibe coded with AI, they are a use of local AI but they aren't helping to better it Your vibe coded SaaS isn't ...

[Cuda 13.3 Downloads](https://developer.nvidia.com/cuda-downloads) [Release Notes](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) Anybody already tried llama.cpp with 13.3?

As per Minimax\_AI twitter [https://x.com/MiniMax\_AI/status/2059286515155599595](https://x.com/MiniMax_AI/status/2059286515155599595) I hope it will speed up Qwen3.7 open weights release. https://preview.redd.it/q1bdhs017n3h1.png?width=898&format=png&auto=webp&s=a9a8ea134a71b9e5b9ea2489fc72420e18c6da67

Sadly the open models seem far behind.

I dont have good experience running q4\_k\_m, the difference to q6 is "a few errors an hour" to " a few errors every couple of days". Edit: How it fails? Just like user DifficultDog8435 and FullstackSensei explained in the comments. They worded it better than me.

TL;DR Some AI behavior reminded me of ADHD/Trauma Response (thought loops, task paralysis...) and I laughed it off at first. Then I treated it like my neurodivergent friends: give em some slack. And just like that, the thought loops stopped, response was fast, the answers correct most of the time AND it actually said "I don't know, help me!" every time it wasn't sure. It's a small Dataset...but still impressive results! [https://github.com/OttoRenner/Gentle-Coding](https://github.com/OttoRenner/Gentle-Coding) Hey everyone, I’ve been testing a weird hypothesis over the last few days, a...

I picked up a 7900 XTX earlier which runs qwen3.6-27b fine, but not to my like. Its compute performance is quite unstable for me. With MTP the decode speed can reach 40-60 t/s, but prefill is just too slow. Regardless of whether I used ROCm or Vulkan, the prefill speed varies between 300t/s and 500 t/s, even with very long prompts. I've been itching to try out an ultra-budget 24GB setup using dual 3060s. I managed to snag a second 3060 at a reasonable price in last few days. So I took out the 7900 XTX, installed the 3060s, and began testing. # Test Configuration * **Test Platform:** i7 477...

For real tho, 9b, 27b, 122b, I don’t really care at this point, just show us that you still love us.

The PrismML team really cooked with these models. They're only \~3GB in size (compared to FLUX.2 Klein 4B, which is \~16GB). Apache-2.0! Official collection on HF: [https://huggingface.co/collections/prism-ml/bonsai-image](https://huggingface.co/collections/prism-ml/bonsai-image) Link to demo: [https://huggingface.co/spaces/webml-community/bonsai-image-webgpu](https://huggingface.co/spaces/webml-community/bonsai-image-webgpu)

I was experimenting with a self-optimizing agentic pipeline to climb the benchmark leaderboard (TerminalBench). On a 10-task subset, I got the performance to rise from \~30% → \~90%. That loop worked, so I asked: can the same reflect-and-rewrite step run continuously against everyday chats instead of a benchmark? **How it works** * Every chat with your local LLM goes through a small proxy and is logged. * `autoswarm reflect` has the same local model review those logs, distill concrete lessons, and write them to `skills.yaml`. * Lessons auto-inject into the system prompt of future chats. ...

# MOSS-TTS-v1.5 **MOSS-TTS-v1.5** is continued from [MOSS-TTS 1.0](https://huggingface.co/OpenMOSS-Team/MOSS-TTS). It preserves the main 1.0 capabilities, including zero-shot voice cloning, long-form speech generation, token-level duration control, Pinyin/IPA pronunciation control, multilingual synthesis, and code-switching. For the full 1.0 feature walkthrough, input schema, decoding hyperparameters, and evaluation tables, please refer to the [MOSS-TTS 1.0 README](https://huggingface.co/OpenMOSS-Team/MOSS-TTS). Compared with MOSS-TTS 1.0, v1.5 focuses on the following improvements: * **St...

I previously hated on this model, but I have just been impressed by it, and I understand the hype now. I have been working on a HTML5 game console and I decided to see if Qwen3.6 27B can handle making some quick games in it to showcase functionality (save games, console API handling for stat tracking and heartbeat management, meta data for the game, etc) I gave it 3 files, explaining how the API works, the gamepad controls, and a typescript shader for it to apply. Then I just game it a very simple prompt "make a breakout game for this console, in the working directory are reference files on...

nice update bois

Big, if true. Doesn't bode well for research / OS models out of China.

Link: youtube.com

Paper came out recently that formalizes something a lot of agent builders have been doing ad hoc. They use a frontier model to propose bounded edits (add/delete/replace) to markdown skill files, then gate every edit against a held out validation set. Only strict improvements accepted, ties rejected, rejected edits become negative signal for the next round. Few things worth noting: Best skills converge with 1 to 4 accepted edits out of many more proposals. Edit budget of 4 to 8 per step works best, remove the cap and performance collapses. Median final skill is \~920 tokens. A skill optim...

Here's the PR by pedapudi. https://github.com/ggml-org/llama.cpp/pull/21344 It's merge request has been denied so it will not be in mainline llama.cpp. The changes are so small that I just put them into whatever the current release of llama.cpp is. Read the PR for more info. It will only work with MOEs. Also, it gives the most boost at low context. As the context rises, the gain diminishes. Pedapudi explains why that happens in the PR. Here are some numbers. It really works well. The tiny amount of time it takes me to apply the code to the current release of llama.cpp is time well spent. ...

Safetensors, llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved: [https://huggingface.co/llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved](https://huggingface.co/llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved) GGUFs, llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved-GGUF [https://huggingface.co/llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved-GGUF](https://huggingface.co/llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-Native-MTP-Preserved-GGUF) NVFP4, llmfan46/Qwen3.5-35B-A3B-uncensored-heretic-v2-N...

They started producing cheaper ram for corsair, hopefully it will get cheaper for consumers [https://www.tomshardware.com/pc-components/ddr5/chinese-memory-maker-cxmt-enters-the-mainstream-consumer-memory-with-corsair-vengeance-ddr5-kit-chinese-made-dram-emerges-as-an-antidote-for-crushing-shortages](https://www.tomshardware.com/pc-components/ddr5/chinese-memory-maker-cxmt-enters-the-mainstream-consumer-memory-with-corsair-vengeance-ddr5-kit-chinese-made-dram-emerges-as-an-antidote-for-crushing-shortages)

Lawyer documents local V100 cluster setup (12×32GB SXM2) for in-house legal document drafting using Claude Code.

I've fine-tuned Qwen 3.5 0.8B on the dataset provided by Pangram with their EditLens paper. It's available via a [Chrome extension](https://chromewebstore.google.com/detail/slop-hammer/gfjdmhfokmhedlgfggmmgchpppmhkdgg); you can just click selected text and it's going to give you the probability distribution of how likely it is AI-generated. It takes under 1s on my M1 MacBook Pro. Pangram did release Llama 3.2 3B trained on their dataset, but I found this model slightly too legacy (too big for the capabilities). Qwen 0.8B (base) ended up being as good after roughly 20h of fine-tuning on a sin...

llama.cpp adds fast Walsh-Hadamard transform (FWHT) for CUDA, yielding 1–2% prompt-processing and 7–9% token-generation speedups with quantized KV-cache.

User reports Qwen3.6 35B outperforms Gemma4, GLM 4.7 Flash, others for local agentic tasks; seeks comparable MoE alternatives.

>Long-context inference in large language models is bottlenecked by the quadratic cost of full attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset of attention heads truly requires full long-context processing; (2) long-...

MiniCPM5-1B released on HuggingFace: 1B-parameter model from CPM team, likely competitive efficiency benchmark for edge deployment.

Financial Times reports Heretic tool removes guardrails from Meta's Llama 3.3 in <10 minutes; 3,500+ decensored variants downloaded 13M times.

Numind releases NuExtract3, open-weight 4B multimodal VLM for document extraction and Markdown conversion under Apache-2.0.

Mac Pro user discovers legacy AMD D700 GPUs now support Vulkan-based LLM inference via new drivers, reviving dormant hardware.