The Archive

OmniMoE-VL sparse vision-language model with coupled visual-depth routing for dynamic visual representation selection across mixture-of-experts.

As language models grow, scaling dense architectures becomes increasingly expensive. In a dense transformer, every token passes through every layer, so adding... As language models grow, scaling dense architectures becomes increasingly expensive. In a dense transformer, every token passes through every layer, so adding capabilities increases computation for both training and inference. Mixture-of-experts (MoE) architectures take a different approach to scaling by using many subnetworks, or experts, while activating only a small subset for each token. Source

Mixture-of-experts with diffusion models for cross-scenario physical layer security in 6G wireless networks.

L0-MoE accelerates dense LLMs via L0-regularized Mixture-of-Experts, achieving 2.5x speedup with minimal performance loss.

HOPE second-order pruning method for Mixture-of-Experts models reduces parameter count by modeling expert cooperation.

How can a 30B-parameter model activate only 3B parameters per token, and still use the capacity of the larger model? Nemotron 3.5 Lightning illustrates the... How can a 30B-parameter model activate only 3B parameters per token, and still use the capacity of the larger model? Nemotron 3.5 Lightning illustrates the answer: It uses a Mixture-of-Experts (MoE) architecture that selects only a subset of its parameters for each token. There are two dominant model architectures: Dense model and MoE. How a model organizes its parameters matters as much as… Source

SeqMoE predictive offloading for Mixture-of-Experts models via sequence-to-sequence expert activation forecasting.

Empirical study showing Mixture-of-Experts models overfit more severely to repeated training data than dense Transformers across 80M-1B scale.

Encode-prefill-decode (EPD) disaggregation is an inference optimization technique for multimodal models that separates the vision encoder stage from the prefill... Encode-prefill-decode (EPD) disaggregation is an inference optimization technique for multimodal models that separates the vision encoder stage from the prefill and decode stages. It is most effective for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts (MoE) models. This post shows when and how to use EPD disaggregation with NVIDIA Dynamo to achieve up to 5x… Source

Mixture-of-Experts (MoE) architectures provide an efficient paradigm for scaling large language models (LLMs), yet fixed top-k routing activates the same number of expert slots for every token, causing substantial redundant computation. Existing expert-skipping methods often rely on router confidence, calibration data, or additional training, and therefore cannot reliably estimate the actual contribution of routed experts. To this end, we propose ACE, a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping in MoE-based LLMs. ACE contains two c...

Verifier-guided framework combines QLoRA, MoE routing, and symbolic solvers (Z3, FOL) for explainable LLM reasoning in education.

Sparse MoE routing decisions across layers share common geometric structure aligned via control subspaces despite layer-specific coordinate systems.

Contrastive Routing Mechanism (CoRM) improves MoE expert specialization by scoring against EMA reference instead of absolute magnitude, concentrating routing to separable subspace.

Q-Strata hierarchically allocates mixed-precision bitwidths across MoE LLM experts to optimize model-level quantization objectives.

CE-MoE reduces training communication overhead in mixture-of-experts LLMs via heterogeneous layer patterns that concentrate expert capacity while maintaining depth.

MM-Spectrum: sparse Mixture-of-Experts framework for multimodal molecular structure elucidation from heterogeneous spectroscopic signals.

Robust CurveMoE uses mixture-of-experts with mode connectivity to defend Mixture-of-Experts models against multi-norm adversarial perturbations efficiently.

Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture for developers to experiment with and evaluate. It’s... Alibaba released the model weights for Qwen3.8-Flash-Next as a preview of the upcoming Qwen4 architecture for developers to experiment with and evaluate. It’s a multimodal mixture-of-experts (MoE) model with a 125B-parameter main model supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token. It has a native 262,144-token context window, extensible to 1M tokens… Source

DUMoE framework captures temporal interest drift in social media via sparse mixture-of-experts and multi-scale modeling for user representation.

MoTE uses task-level sparse mixture-of-experts for multi-task video understanding, improving over dense decoders for action recognition and forecasting.

Proposes DF-MoE sparse mixture-of-experts architecture combining 8+ audio-visual feature extractors for generalizable deepfake detection.

RARE framework enables representation engineering steering for MoE LLMs by decoupling routing from behavior-directed latent modifications.

Two-step hyperparameter transfer framework enables compute-efficient learning rate optimization for trillion-token Mixture-of-Experts models via scaling extrapolation.

Two Mixture-of-Experts (MoE) forward passes can share every weight yet route the same token through different experts. This creates a possible blind spot in same-weight self-distillation, where a demonstration-conditioned teacher supervises a query-only student. We study this mismatch in its single-step form, with frozen weights rather than as a proxy for a full training trajectory. An exact blockwise decomposition separates a routing term, which changes gates at fixed content, from a dense-like content term. Across seven open-weight checkpoints and two domains, the routing term spans only $1...

Sycophancy, the tendency of a language model to change its answer to match a user's stated belief, is a common alignment failure. Existing activation steering methods typically apply a single contrastive direction uniformly throughout the model, which is an unconditional intervention that alters activations even when no sycophantic behavior is present, trading knowledge retention for behavioral correction. In Mixture-of-Experts (MoE) models, prior work further suggests that behavior is encoded within expert computations rather than routing decisions alone, making precise behavioral steering p...

Mixture-of-Experts (MoE) models have been widely adopted in real-time interactive applications such as coding assistants, real-time audio-video interaction systems. To meet the extremely low response latency requirements of these scenarios, practitioners commonly employ small-batch decoding, under which MoE inference becomes memory-bound and is severely bottlenecked by expert weight loading. However, this bottleneck has received limited attention, and existing solutions such as post-training weight compression or fine-grained expert design during pre-training either degrade model accuracy or ...

Financial volatility is regime dependent, yet incorporating regime information into neural networks can also destabilize training. This paper asks where such information should enter a neural cross-sectional volatility forecasting model. We study five-day realized-volatility forecasts for 1,027 U.S. equities using a rolling walk-forward evaluation framework in which information, model capacity, hyperparameter tuning, and random seeds are matched across architectures. We propose RG-ResMoE, a regime-gated residual mixture-of-experts architecture in which regime information is used only for expe...

Long-running AI agents spend most of their time on high-volume execution: tool calls, result validation, and subagent delegation. Using a frontier reasoning... Long-running AI agents spend most of their time on high-volume execution: tool calls, result validation, and subagent delegation. Using a frontier reasoning model for every execution step adds cost and latency. NVIDIA Nemotron 3.5 Lightning is an open 30B mixture-of-experts (MoE) model with 3B active parameters built for that execution layer of always-on agents. It is designed for harnesses… Source

Retinal fundus images frequently exhibit multiple co-occurring pathologies, yet standard deep learning classifiers apply static, identical computation to every image regardless of the underlying disease distribution. We propose a novel architecture that resolves this via sparse conditional computation, pairing a Guided Context Gating (GCG) spatial attention front-end with a sparsely-routed Mixture-of-Experts (MoE) block operating over feature tokens. Crucially, this routing yields an interpretable, data-driven decomposition. Expert allocation is significantly disease-dependent (p < 0.001), wi...

Physics-calibrated mixture-of-experts framework learns from fragmented experimental data on plastic thermochemical upcycling.