Designing Transformers for Continuous Diffusion Language Models
Presents novel architectural improvements for diffusion-based language generation that researchers can build upon.
AI Summary
Researchers introduce LangDiT, a architecture optimized for continuous diffusion language models that achieves state-of-the-art generation perplexity on OpenWebText.
Excerpt
Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalizati