Live Math & aptitude benchmark

Know which AI truly works in Indian languages.

Signoff.space benchmarks AI models on expert-verified questions written in Hindi, Telugu and English. We publish how every model performs in each language, and how much it loses outside English.

See the leaderboard Compare languages
हिंदी తెలుగు English
One question, three languages
Same answer key in every language

Profit and lossAnswer: 12.5%
Average pass rate across modelsMath & aptitude

Every question is checked by native-speaking experts before it counts.

Reviewer  Signed off

Figures come from the published leaderboard.

How models do in each language

Every question is asked in Hindi, Telugu and English with the same answer. The gap shows how many points a model loses outside English.

These are placeholder results to show the layout. Real results appear here once verified runs are published.

Leaderboard

Ranked by the Indian-language score: the average of Hindi and Telugu, with a 95% confidence interval. English is shown as the control.

Placeholder results. Real results appear here once verified runs are published.
RankModelType

Benchmark progress

Questions are written in all three languages, then verified by native-speaking experts. Only verified questions are used for published results.

Coming next

Domains being written, also in Hindi, Telugu and English

    One standard, many professions

    Every domain is graded in Hindi, Telugu and English. We open a domain once its questions have been written, verified and piloted.

    How a question earns its sign-off

    Every question follows the same path before it counts towards a published score.

    1. Written in three languages

      Experts write each question in Hindi, Telugu and English, with one shared answer key.

    2. Verified by native speakers

      A second expert checks the translation and the answer. Questions that don't match are fixed or dropped.

    3. Run on every model

      Each model answers every question several times through its API, under identical instructions.

    4. Graded and published

      Answers are graded automatically, results are reviewed, then published by model and by language.

    Inside a question

    A public sample from the math & aptitude set. Held-out questions are never published, so they can't leak into training data.

    Syllogism: the same question in three languages

    Hindi

    कथन: सभी गुलाब फूल हैं। कुछ फूल जल्दी मुरझा जाते हैं। कौन सा निष्कर्ष निकलता है?

    1. A) सभी गुलाब जल्दी मुरझा जाते हैं
    2. B) कुछ गुलाब जल्दी मुरझा जाते हैं
    3. C) गुलाबों के जल्दी मुरझाने के बारे में कोई निश्चित निष्कर्ष नहीं निकलता
    4. D) कोई भी गुलाब जल्दी नहीं मुरझाता
    Telugu

    వాక్యాలు: అన్ని గులాబీలు పువ్వులే. కొన్ని పువ్వులు త్వరగా వాడిపోతాయి. ఏ ముగింపు సరైనది?

    1. A) అన్ని గులాబీలు త్వరగా వాడిపోతాయి
    2. B) కొన్ని గులాబీలు త్వరగా వాడిపోతాయి
    3. C) గులాబీలు త్వరగా వాడిపోవడం గురించి ఖచ్చితమైన ముగింపుకు రాలేము
    4. D) ఏ గులాబీ కూడా త్వరగా వాడిపోదు
    English

    Statements: All roses are flowers. Some flowers fade quickly. Which conclusion follows?

    1. A) All roses fade quickly
    2. B) Some roses fade quickly
    3. C) No definite conclusion can be drawn about roses fading quickly
    4. D) No rose fades quickly

    Get a sign-off before you deploy.

    We evaluate models for the teams that build them and the companies that deploy them to Indian users.

    Request a private evaluationTest your model on held-out questions in Hindi, Telugu and English, with a report by language and topic. Submit a modelShare API access and we'll add your model to the public leaderboard.