Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
Provides foundational research and practical guidelines for ML engineers on advanced model fusion techniques.
AI Summary
A new arXiv paper compares three paradigms—Merge, Mix RL, and multi-teacher distillation—for consolidating reinforcement learning with verifiable rewards (RLVR) capabilities across different domains.
Excerpt
Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. We organize three fusion paradigms by the artefacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and multi-teacher on-policy distillation (MOPD) uses both. Because they have largely been studied in isolation, how they compare and how to