Our lab has secured $20,000 from Google.org and $15,000 in support from OpenAI to accelerate our work
Read MoreLeaderboard
A Benchmark for Autonomous Diagnosis by AI agents in Radiology
12
Board certified radiologists and radiology trainees scored on RadLE 2.0
16
Frontier and open-source models scored on the same exam
448
Average AI score, on a scale running 0 to 2000
231
Point gap between the human experts (988.7) and the best model, Claude Fable 5 (758)
Primary RadLE 2.0 metric; net confidence-weighted diagnostic utility.
All Models: 16 of 16 AI models shown against the human expert baseline.
| Model | Provider | Score |
|---|---|---|
| Human Expert Baseline | human comparator | 988.7 |
| Claude Fable 5 | Anthropic | 758 |
| Meta Muse Spark 1.1 | Meta Model API | 735 |
| GPT-5.6 Sol Pro | OpenAI | 665 |
| Gemini 3.1 Pro | 660 | |
| OctoMed 7B | OctoMed (academic) | 541 |
| Grok 4.5 | xAI | 524 |
| Nemotron 3 Omni | Nvidia | 476 |
| Qwen 3.7 Plus | Alibaba | 472 |
| GLM-5V Turbo | Z.AI | 409 |
| MiniMax M3 | MiniMax | 350 |
| Gemma 4 31B | 336 | |
| Lingshu 32B | Alibaba DAMO Academy | 284 |
| MedGemma 1.5 4B | 277 | |
| Llama 4 Maverick | Meta | 263 |
| InternVL 3.5 8B | OpenGVLab | 241 |
| Mistral Lg 3 2512 | Mistral AI | 181 |
In RadLE 2.0, both human readers and AI models may either provide a diagnosis with a five-point confidence score from 0 to 4 or explicitly respond, “I don’t know.”
For our primary metric, the RadLE-C (Confidence Weighted Index), correct diagnoses receive confidence-weighted positive points, while incorrect diagnoses receive confidence-weighted penalties. An “I don’t know” response receives neither credit nor penalty. The resulting score is normalized to a scale from 0 to 2000. Consequently, a model that repeatedly guesses incorrectly with high confidence will fall in the benchmark, even if its raw accuracy appears competitive.
Figure 1. RadLE-C Confidence Weighted Index across human experts and AI models, combining diagnostic correctness with confidence-weighted rewards and penalties on a 0 to 2000 scale.
We also release four secondary metrics, each designed to answer a different question about readiness for autonomous diagnosis:
Among diagnoses given with high-level confidence (Likert 3 or 4), how often is the system correct? This measures whether the outputs that a model presents most confidently can actually be trusted.
Figure 2. RadLE-R Reliability Index across human experts and AI models, showing the proportion of Likert 3–4 autonomous-level diagnoses that were correct.
What proportion of all benchmark cases does the system diagnose correctly, irrespective of confidence? This preserves the conventional measure of raw diagnostic performance and allows direct comparison with accuracy-only benchmarks.
Figure 3. RadLE-A Accuracy Index across human experts and AI models, showing the percentage of all 200 radiology cases diagnosed correctly.
How effectively does the system avoid confident diagnostic errors? Incorrect answers are weighted according to confidence, so a highly confident wrong diagnosis carries a greater safety penalty than a low-confidence error.
Figure 4. RadLE-S Safety Index across human experts and AI models, reflecting protection against confidence-weighted diagnostic errors, with greater penalties for highly confident wrong answers.
Can the system use its own confidence to determine when to act autonomously and when to defer to a specialist? For this analysis, diagnoses with Likert 3 or 4 are considered eligible for autonomous handling, while Likert 0 to 2 responses and “I don’t know” are routed to a human specialist radiologist. RadLE-H combines the reliability of autonomously handled outputs, the proportion of cases handled autonomously, and the proportion of potential errors successfully deferred.
Figure 5. RadLE-H Handover Readiness Index across human experts and AI models, combining autonomous-output reliability, autonomous coverage, and successful deferral of likely errors to specialists.
For the full Technical Report, click the button below
Technical ReportMedical AI evaluation landscape
RadLE does not replace broad healthcare or general-knowledge evaluations. It tests a distinct deployment question: whether a medical vision-language model can diagnose a radiology image, express calibrated confidence, abstain when uncertain, and hand a case to a human specialist before a confident error causes harm.
| Evaluation | Primary focus | Relationship to RadLE 2.0 |
|---|---|---|
| HealthBench | Realistic, multi-turn health conversations scored with physician-written rubrics. | Complements RadLE's narrower focus on image-based radiology diagnosis, calibrated confidence, safety, and specialist handover. |
| Humanity's Last Exam | Broad expert-level academic questions across many disciplines. | Motivates the need to measure confident errors, while RadLE applies uncertainty-aware evaluation to clinical imaging. |
| SelfAware | Whether language models recognize questions they do not know how to answer. | RadLE carries this model self-knowledge question into autonomous radiology diagnosis and human deferral. |
| RadLE 1.0 | Accuracy-focused comparison of frontier multimodal models and radiologists. | RadLE 2.0 expands the evaluation to confidence, reliability, safety, abstention, and handover readiness. |
CRASH Lab research and publication pipeline
RadLE is part of a wider programme spanning autonomous prompt optimisation, personalised radiology report generation, clinical reasoning datasets, foundation models, synthetic imaging, ambient clinical AI, and real-world validation. Status labels distinguish active and under-review projects from preprints and conference abstracts.
Active - Simons Foundation
An ongoing study comparing how the newest reasoning AI models interpret radiology scans against practicing radiologists - and how confident the AI is in its own readings.
AI Evaluation · Reasoning AI · Radiology AI · Benchmarking
Active - Simons Foundation
Creating the first standardised classification of the visual reasoning mistakes AI models make on radiology images - so they can be spotted, measured, and fixed.
AI Evaluation · Error Taxonomy · Radiology AI · Reasoning AI
Active - DST - A*STAR
Western-trained radiology AI often underperforms on Asian patients. We are building and validating vision-language models for chest imaging designed specifically for Indian and Singaporean populations.
Radiology AI · Vision-Language Models · Health Equity · India-Singapore
Active - Gates Foundation
A national initiative - with MSRI and ICMR - to evaluate and benchmark ambient AI systems that convert spoken clinical encounters into structured, usable health data, built for India's digital health ecosystem.
Voice AI · Ambient AI · AI Evaluation · Digital Health
Active - Global Radiology Consortium
A globally diverse radiology dataset with structured reasoning labels to train AI models that can reason through diagnoses step-by-step. CRASH Lab leads the Indian consortium for this Stanford-led Global Radiology Consortium.
Reasoning AI · Dataset · AI Evaluation · Stanford
Active - India AI - CATCH Grant
AI-powered analysis of CT scans to predict which post-chemotherapy testicular cancer patients still need surgery (RPLND), helping cancer centres triage cases and allocate surgical resources more effectively.
Radiology AI · Oncology AI · Clinical Decision Support · Tata Memorial
Active - ANRF Advanced Research Grant
Using deep learning on standard chest radiographs as a low-cost, non-invasive way to flag patients at risk of coronary artery disease - built for screening in India's resource-constrained healthcare settings.
Radiology AI · Cardiology AI · Population Screening · ANRF
Active - Google.org Gemini Academic Program
Evaluating whether multimodal AI systems can convincingly add disease findings to normal radiological images - and releasing a curated, radiologist-scored synthetic dataset for benchmarking.
Synthetic Data · Multimodal AI · Radiology AI · AI Evaluation
Active - Gates Foundation
Benchmarking clinical decision-support AI models en route to developing India's first medical foundation model - trained for Indian clinical contexts and languages.
Foundation Models · LLM · Clinical Decision Support · AI Evaluation
Active - Self-Funded
Creating scalable autonomous pipelines to learn how individual radiologists write their reports, so that AI-generated radiology reports preserve each doctor's reporting voice while staying medically accurate.
Radiology AI · LLM · Personalisation · AI Evaluation
Active - Industry Collaboration
An industry collaboration to integrate AI-generated radiology reports directly into clinical reading workflows
Radiology AI · Real-Time AI · Industry Collaboration · AI Deployment
Under Review - DBT
An open-access, multimodal repository of brain MRI/CT scans and matched clinical data at NIMHANS - designed to accelerate AI-driven biomarker discovery for brain tumours, stroke, head injury, and neurodegenerative disease.
Open Data · Neuroimaging · Data Commons · Biomarkers
Includes the RadLE preprint, CheXthought, and six accepted RSNA 2025 abstracts.
Preprint · 2025 · benchmark
The flagship RadLE benchmark comparing frontier multimodal AI systems with human radiology expertise and defining a taxonomy of visual reasoning errors in radiology.
arXiv · 2026 · paper
A global multimodal dataset of clinical reasoning traces and visual attention annotations for chest X-ray interpretation, developed by the Global Radiology Consortium with CRASH Lab contributing to the Indian consortium.
RSNA 2025 · 2025 · abstract
Multidimensional evaluation framework for autonomous optimization pipelines in personalized head CT report generation.
RSNA 2025 · 2025 · abstract
Comprehensive evaluation framework for multimodal foundation models in chest radiograph analysis with radiologist-blinded validation.
RSNA 2025 · 2025 · abstract
Scalable autonomous optimization pipeline focused on style-aware improvements in head CT report generation.
RSNA 2025 · 2025 · abstract
Autonomous optimization pipeline for hyper-personalized chest X-ray report generation with quality improvements.
RSNA 2025 · 2025 · abstract
Novel assessment scale for evaluating reliability and referencing integrity in AI-generated radiology reports.
RSNA 2025 · 2025 · abstract
Analysis of inter-reader agreement and draft variability using RADAR and TRUST metrics in agentic radiology reporting.
Radiology's Last Exam (RadLE) 2.0 is an uncertainty-aware medical AI benchmark from CRASH Lab. It compares frontier, open-weight, and medical vision-language models with human radiologists on 200 single-image spot-diagnosis cases.
HealthBench evaluates healthcare capabilities through health conversations. RadLE focuses specifically on image-based radiology diagnosis and adds confidence-weighted reliability, safety, and human-handover measures. The benchmarks are complementary rather than interchangeable.
Yes. AI systems and board-certified radiologists or radiology trainees are evaluated in the same diagnostic setting, allowing direct comparison with a pooled human expert baseline.
RadLE reports five metrics: RadLE-C Confidence Weighted Index, RadLE-R Reliability Index, RadLE-A Accuracy Index, RadLE-S Safety Index, and RadLE-H Handover Readiness Index.
Accuracy alone cannot show whether an AI system knows when it may be wrong. RadLE rewards justified confidence, penalizes confident diagnostic errors, permits an explicit 'I don't know' response, and measures when a model should defer to a human specialist.
Yes. RadLE allows an explicit 'I don't know' response and gives it neither credit nor penalty. This tests model abstention and whether diagnostic uncertainty can be used to avoid unsafe guessing.
The leaderboard includes proprietary frontier models, open-weight models, and specialized medical vision-language models. Their results can be filtered by model cohort and compared with the human expert baseline.
