Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Hugging Face and NVIDIA have released a 2.8 trillion token corpus for 22 Indian languages, along with an open, GPU-accelerated data generation pipeline. The corpus includes native-script data for all 22 languages and romanized text for 12 languages. The pipeline allows developers to adapt the generation recipe to their own source material.