Feature Information Dynamics in Diffusion
Provides theoretical foundation and practical tools for analyzing feature generation dynamics in diffusion models, relevant for ML researchers working on model interpretability and training optimization.
AI Summary
Researchers developed an information-theoretic framework to quantitatively measure when specific features are generated during denoising, revealing representation differences across pixel, SDVAE, VAVAE, and RAE models.
Excerpt
Diffusion models generate data through a continuum of denoising problems, and are widely observed to reveal coarse structure before fine detail. Yet, this intuition is mostly empirical and qualitative. We introduce feature information dynamics, an information-theoretic framework for localizing when a feature is generated during diffusion. Using the I-MMSE identity, we connect the rate of feature mutual information change to a gap between optimal unconditional and feature-conditional denoising lo