The Archive

a small win for american open models - Glimmer runs on a fits on a single RTX 3090!

Post-training quantization lets large text-to-image diffusion transformers run on consumer GPUs, yet the hardware-specific trade-offs are seldom measured directly. We quantize Ideogram 4.0 - a 9.3B flow-matching diffusion transformer (DiT), shipped as two separate-weight copies of a single-stream 34-layer backbone for classifier-free guidance and conditioned by a Qwen3-VL-8B encoder - for Ampere RTX 3090 GPUs, which lack FP8 tensor cores. Our INT8 W8A8 recipe (per-channel weights, per-token dynamic activations, SmoothQuant, and mixed-precision protection of a small high-fragility layer set) h...

BeeLlama v0.2.0 achieves 4-5x token throughput gains on RTX 3090 via DFlash optimizations for Qwen 27B and Gemma 31B models.

Quantization benchmark study on Qwen 27B comparing TurboQuant, TCQ, and symmetric q8 via PPL/KLD metrics on RTX 3090 with 64k-128k context.

Community benchmarks OmniVoice TTS across 21 GPUs (5GB VRAM peak) on Vast.ai, measuring xRT performance vs. RTX 3090 baseline.

llama.cpp MTP speculative decoding merged; Qwen 27B/35B inference speedups 1.24–2.44× on consumer GPUs/APUs with quantization.

Qwen 3.6 27B performance benchmarks across llama.cpp backends on RTX 3090: ik_llama.cpp achieved 1261 tok/s prefill, 72.9 tok/s decode with 156k context.

Llama.cpp multi-token prediction on Qwen 3.6 27B shows 42% prefill slowdown but 85% token generation speedup on RTX 3090.

Community discussion of Qwen 27B quantization and inference optimization on single RTX 3090 GPU.

User reports running local LLMs on dual RTX 3090 setup with club-3090 project, noting performance improvements over LM Studio.

Tuning llama.cpp ubatch and n-cpu-moe parameters improves gpt-oss-120b prompt processing from 240 to higher tok/s on RTX 3090.

BeeLlama.cpp fork adds DFlash, TurboQuant, and vision support; runs Qwen 3.6 27B Q5 on RTX 3090 with 200k context at 135 tps.

Qwen3.5/3.6 inference optimization guide: NextN MTP speculative decoding achieves 2.9× speedup on RTX 3090 Ti via llama.cpp with zero quality loss.

User achieves 50 tokens/sec with Qwen 3.6 27B on RTX 3090 using MTP speculative decoding at 100k context.

Dual RTX 3090 setup draws ~760W under LLM inference, 90W idle; practical hardware benchmark for on-premises deployment.

LDR framework with Qwen3.6-27B agentic search achieves 95.7% SimpleQA accuracy on single RTX 3090.

Qwen3.6-27B Windows native vLLM launcher achieves 72 tok/s on RTX 3090 with portable installation, no WSL.

PFlash: speculative prefill technique achieves 10x speedup on 128K context with quantized 27B models on RTX 3090, open-source C++/CUDA implementation.

Qwen3.6-27B achieves 218K context window at 50–66 TPS on single RTX 3090 with stable tool calls after PN12 fix.

User reports difficulty replicating reported inference speeds (30-100 tok/s) for Qwen 27B on RTX 3090, achieving 10-19 tok/s instead.

User reports Qwen 3.6 and Gemma 4 performing expert-level work locally on consumer hardware (RTX 3090), replacing skilled human labor.

Luce DFlash: open-source speculative decoding for Qwen3.6-27B achieves 1.98x throughput on RTX 3090 via GGUF port.

User shares GPU workstation build with 4x RTX 3090 cards in server case.

Technical walkthrough: Qwen 3.6 27B achieves 85 TPS, 125K context on single RTX 3090 using llama.cpp.

llama.cpp Vulkan and SYCL benchmarks comparing Nvidia RTX 3090 vs Intel Arc Pro B70 on prompt processing and token generation.

User demonstrates Qwen3.6-27B running locally via llama-server with 200k context on dual RTX 3090, achieving coding performance cheaper than Claude.