Rethinking Routing Exploration in Modern Mixture-of-Experts
Advances MoE architecture design for ML researchers working on scalable model optimization.
AI Summary
Researchers propose MirrorMoE to address the exploration dilemma in modern Mixture-of-Experts architectures between fine-grained segmentation and shared experts.
Excerpt
Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reduci