Existing Indic benchmarks are either multi-language (which dilutes depth), narrowly scoped to academic exam MCQs, preference-only, or built primarily as training data. None provide a single-language, single-region, full-spectrum evaluation standard — one language tied to its primary geographic and social region, covering the practical and cultural dimensions that a speaker of that language actually shares.
ATX IndicEval is built to fill that gap, one language–region pair at a time.
Phase 1 — Marathi / Maharashtra (mr subset) is released here. Hindi, Gujarati and Sanskrit follow as additional subsets.
What this measures
Core cultural knowledge that a well-read speaker of the language would be expected to hold — not niche or specialist trivia, and not translated English content. Every question, answer and answer option is written in Marathi (Devanagari).
Note: Detailed technical report coming soon.
Back to reports ↑