The Archive

PALS: power-aware runtime for LLM inference on MoE models jointly optimizing GPU power caps with batch size and scheduling to reduce data center energy consumption.

Shodh-MoE: Sparse mixture-of-experts architecture addressing negative transfer in multi-physics foundation models via selective routing.

Study reveals geometric coupling between routers and experts in sparse MoE models, explaining routing collapse and informing load-balancing improvements.

DECO: sparse MoE architecture matching dense Transformer performance with reduced storage/memory for edge device deployment.

UniPool proposes shared expert pool across MoE layers instead of per-layer isolation, reducing parameters while maintaining performance in production models.

Resource modeling and pipelined hybrid parallelism system for efficient large-scale Mixture-of-Experts training on HPC platforms.

AIR-MoE uses vector quantization for efficient routing in granular mixture-of-experts, reducing computational overhead of token-to-expert assignment.

Mixture-of-Experts framework for semi-supervised inference combining diverse predictors with limited labeled data via prediction-powered inference.

FaaSMoE: serverless multi-tenant MoE serving architecture decoupling control and execution planes to reduce resource underutilization in expert model deployment.

arXiv paper presenting first direct empirical comparison of Mixture-of-Experts vs dense model architectures.

Theoretical analysis of Bayesian softmax-gated mixture-of-experts models with focus on posterior distribution asymptotic behavior.

BAR framework trains domain-specific experts separately then composes via Mixture-of-Experts, avoiding catastrophic forgetting in multi-domain post-training.

Analysis of Mixture-of-Experts routing locality and KV cache sharing overlap across layers in multi-candidate code generation from shared prefixes.

In LLM training, Expert Parallel (EP) communication for hyperscale mixture-of-experts (MoE) models is challenging. EP communication is essentially all-to-all,... In LLM training, Expert Parallel (EP) communication for hyperscale mixture-of-experts (MoE) models is challenging. EP communication is essentially all-to-all, but due to its dynamics and sparseness (only topk experts per AI token instead of all experts), it’s challenging to implement and optimize. This post details an efficient MoE EP communication solution, Hybrid-EP, and its use in the… Source

Mistral AI releases Mixtral, a sparse mixture-of-experts model combining quality with efficient routing.