Generative Representation Learning for Scalable Latent Tokenization
Presents novel theoretical framework and empirical results for ML researchers working on representation learning and generative modeling efficiency.
AI Summary
Researchers introduce a new generative representation learning approach for scalable latent tokenization that achieves compute-optimal improvements over direct diffusion methods.
Excerpt
We introduce an approach for self-supervised representation learning designed to be used as a tokenizer for generative modeling. Our objective supervises a representation loss by factorizing the pointwise mutual information of the data, which we estimate by taking information differences between conditional predictions from a learned generative model. We justify our algorithm theoretically and empirically, demonstrating that our generative approach to representation learning avoids dimensional c