NVIDIA and IIT Madras Release 2.8 Trillion Tokens for Indian Languages

NVIDIA, together with India's IIT Madras and its AI4Bharat lab, is preparing to publish Nemotron-pretraining-Indic-v1 — roughly 2.8 trillion tokens across 22 Indian languages. These are the languages named in the country's constitution: first languages for about 97% of Indians, nearly 1.2 billion people at the 2011 Census. The corpus arrives alongside the open, GPU-accelerated pipeline used to build it.

The idea is straightforward: take curated English material — mathematics, code, question-and-answer content — and translate it into Indian languages so models can explain technical concepts and work through problems in those languages. Every record traces back to a source document, and each output is checked against its input, keeping the synthetic text grounded in real information.

What is in the corpus

Of the 2.8 trillion tokens, 2,601 billion are native-script text and another 224.6 billion are romanized (counted with the Gemma-4 tokenizer). Native-script data exists for all 22 languages, while Latin-script transliteration covers twelve of them, for 34 language-script configurations in all. Note that "native script" refers only to the writing system — the text is still translated from English.

Two pipelines

Sources come from 15 Nemotron-CC-v2 subsets. For twelve languages, Gemma-4-31B-it translates documents in full, preserving context across paragraphs, tables, and equations. The same model then scores each translation against a rubric inspired by the MQM framework: only outputs rated at least 75 out of 100 are kept, while weaker ones are repaired using the feedback and reassessed. This repair step recovered 438.4 billion tokens.

The other ten languages rely on Indic Translate, a model from the nonprofit Bodhan AI. Long Santali documents are translated in overlapping windows and reassembled; Manipuri is first translated into Bengali script, then converted to its native Meitei Mayek.

What it achieved

During continued pretraining of Nemotron 3.5 Super VL, 10 billion Indic tokens were added (10% of a 100-billion-token budget). Scores rose on all five reported Indic knowledge and reasoning benchmarks, with gains ranging from 4.98 to 9.82 percentage points.

AI4Bharat head Mitesh Khapra, named to the 2025 TIME100 AI list, long ago pinpointed the core issue: Indian-language technology trails English because "we do not have enough data." The team answers with both more data and an open recipe for making it. Still, they are candid about the limits: translation broadens access to source material but does not replace text originally written in Indian languages.

Source: Hugging Face Blog