Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Provides developers with massive multilingual dataset and reproducible pipeline for building Indic language models
AI Summary
NVIDIA and AI4Bharat release 2.8 trillion Indic language dataset and open-source GPU-accelerated data generation pipeline for 22 Indian languages.
