Mask-Aware Policy Gradients for Diffusion Language Models
Advances the state-of-the-art in diffusion language model optimization with a novel RL formulation that could influence future training methodologies.
AI Summary
Researchers introduce Mask-Aware Policy Gradients, a novel reinforcement learning approach for Masked Diffusion Language Models that optimizes both placement and masking decisions during generation. This method achieves state-of-the-art results on mathematical reasoning (87.1% on GSM8K) and coding benchmarks (53.4% on MBPP) by addressing the intractable log-likelihood estimation problem in MDLMs. The paper formalizes MDLM generation as a two-stage action MDP and demonstrates significant performance improvements over existing approaches.
Excerpt
Reinforcement learning has proven effective for improving reasoning in large language models, but extending it to Masked Diffusion Language Models (MDLMs) remains challenging due to the intractability of the log-likelihood estimation. Existing approaches approximate this log-likelihood by modeling only the token predictions, ignoring the order in which positions are unmasked during generation. We observe that MDLM generation involves two decisions at each step: what tokens to place at each maske