The Archive

Provides first global convergence guarantee for gradient EM training of gated mixture-of-experts models with linear experts.

GC-MoE overlaps expert-parallel token dispatch with FP8/FP4 grouped GEMM via CUDA to reduce decode latency in mixture-of-experts model serving.

Most parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step. This memory grows with the number and size of the experts, so it rises as MoE models scale. We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away. It brings optimizer-in-backward from dense models to batched MoE expert tensors, so the GPU never holds more than one layer of expert gradients. When we fine-tune OLMoE-1B-7B and Granite-3.1-3B-A800M in bf16 with two expert-k...

Parameter-efficient continual learning on pre-trained Vision Transformers largely relies on a costly premise: preventing catastrophic forgetting requires either spawning a dedicated LoRA branch per session---causing adapter parameters to scale linearly with task horizon---or applying sparse Mixture-of-Experts gating symmetrically to both forward and backward passes, which sacrifices representational rank at test time. We argue that reading consolidated knowledge and writing new task credit are fundamentally asymmetric operations. Because visual features in a shared low-rank space distribute a...

VideoEditor-MoE: open-source mixture-of-experts framework scaling video editing from 5B to 56B parameters via MoE-Attention routing.

NVIDIA Nemotron 3.5 Super VL is a 120B-parameter mixture-of-experts model that pairs reasoning and tool use with multimodal understanding. This post explores its architecture and use cases, then shows how to deploy it on Amazon EKS, EKS Auto Mode, and Amazon SageMaker HyperPod EKS with NVIDIA Dynamo and vLLM.

Analysis of router probability collapse and recovery in mixture-of-experts models during gradient descent training.

Generating tokens faster starts with identifying what limits inference performance. In the batch-size-1 evaluation of Ling-3.0-flash from Inclusion AI on NVIDIA... Generating tokens faster starts with identifying what limits inference performance. In the batch-size-1 evaluation of Ling-3.0-flash from Inclusion AI on NVIDIA DGX Spark, profiling showed that decoding was memory-bound: reading model weights drove execution time. Two operations stood out—the mixture-of-experts (MoE) computation and the output-layer projection that produces scores across the model’… Source

Controlled study of mixture-of-experts for time-series forecasting shows channel-specific expert allocation improves beyond capacity matching.

Accordion enables flexible LLM inference across five budgets via mixture-of-experts layer looping, reducing cost of maintaining separate models.

Meta launches Muse Spark Open, a coding-optimized 816B mixture-of-experts (MoE) model with 42B active parameters and a 1M token context window. Trained on... Meta launches Muse Spark Open, a coding-optimized 816B mixture-of-experts (MoE) model with 42B active parameters and a 1M token context window. Trained on NVIDIA GB300 NVL72 Blackwell Ultra specifically for agentic coding workflows, it delivers higher first attempt accuracy, reliable tool calling, and deep codebase understanding across repositories. The 1M token context window holds an entire… Source

Analysis of routing exploration trade-offs in Mixture-of-Experts models with fine-grained experts and shared experts, proposing design mitigations.

Always-on agents are taking on longer, more complex enterprise workflows. A single request can involve reading a document, checking information across sources,... Always-on agents are taking on longer, more complex enterprise workflows. A single request can involve reading a document, checking information across sources, following company policy, and calling several tools. NVIDIA Nemotron 3.5 Super VL is an open, customizable multimodal model for these specialized workflows. Its 120B-parameter mixture-of-experts (MoE) architecture activates 12B… Source

Mixture-of-experts architecture using Hodge-Laplacian gating partitions PDE solutions by spectral scale, improving surrogate model performance on unstructured meshes.

OrionMoE-Small: compact tabular foundation model using distribution-aware dual-axis Transformer with dataset-conditioned sparse MoE for in-context learning.

UniW(A)M unifies world models and world-action models via mixture-of-experts to share dynamics while preserving distinct prediction and control capabilities.

Sparse Mixture-of-Experts models achieve compute savings but not sample efficiency; oracle inequalities bound adaptive routing cost.

Category-conditioned expert-selection biases in Mixture-of-Experts models mitigate data imbalance by preferentially routing minority categories to capacity.

DuoRate optimizes MoE serving by allocating per-expert precision separately for storage and decode traffic, addressing the store-read gap in quantized models.

Trained with reinforcement learning in real environments, Mellum2.1 is built for coding agents and fast sub-agents that run on your own hardware. Today, we’re releasing Mellum2.1, the next version of the 12B mixture-of-experts model we open-sourced in June. The architecture hasn’t changed since version 2: it’s still a compact, fast model with 2.5B active parameters, […]

Elastic Expert Routing smooths discrete top-k boundaries in sparse MoE training via stochastic expert budget sampling.

BRANCH-MoE uses binary decision tree routing for experts in mixture-of-experts layers to improve load balancing and reduce computation overhead.

Study of sparsity-inducing routing maps (sparsemax, entmax, normmax) in MoE language models reveals training dynamics and expert participation patterns.

HiRefPOU: hierarchical partition-of-unity mixture-of-experts architecture for operator learning on PDEs with sharp interfaces and multiscale structures.

SPEAR applies spectral-disentangled mixture-of-experts to PDE foundation models to mitigate knowledge interference and expert redundancy.

OpenTSLM TeeMoE unifies forecasting, contextual prediction, and temporal reasoning in a single time-series language model via mixture-of-experts.

UniAE-MoE unifies audio encoders via Mixture-of-Experts architecture for multi-task audio language model performance.

SplitMoE applies spatiotemporal-aware Mixture-of-Experts routing to video diffusion, breaking uniform expert-load regularization.

Foil method enables efficient looped mixture-of-experts by flattening experts and decoupling attention, improving parameter utilization in sparse models.