Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

{"

Workflow Overview":"The pipeline captures the full generation recipe, including prompts, LLM workflows, and quality checks, so developers can adapt it to their own source material and generate more data for their needs.","

Corpus Composition":"The corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. Counts use the Gemma-4 tokenizer. Document records include translations and script variants of shared sources; they are not counts of unique English documents.","

Language Configurations":"The corpus has two groups of language configurations: Native-script and romanized data. Native-script data includes 12 languages, while romanized data includes 10 languages. Native-script data is written in the native writing system, while romanized data is written in the Latin script.","

Workflow Details":"The workflow combines document-level translation, quality assessment, and targeted repair, with checks to preserve content such as equations, tables, and code. This data complements naturally authored Indic text, which remains essential for local knowledge and cultural context.","

Open Source Release":"The open-source release represents a substantial expansion of the open synthetic Indic training data that will be available to the community. It brings a broad mix of curated English material, including mathematics, code, and question-answer content, into Indian languages, giving teams more training material for building models that can explain technical concepts, work through problems, and answer questions in those languages."}