01
Introduction
Today, we are releasing Radiology's Last Exam (RadLE) 2.0, an uncertainty-aware benchmark designed to test whether AI systems are ready for autonomous diagnosis in radiology.
Since last year, we have been focusing our research on autonomous diagnosis in healthcare, and trying to understand how to meaningfully evaluate AI agents and benchmark their readiness for real world deployment. Accuracy alone cannot tell us whether an AI system is ready to diagnose patients autonomously. RadLE 2.0 is a composite benchmark designed to evaluate not only whether an AI reaches the correct diagnosis, but whether its confidence in making the diagnosis is justified, reliable and if it knows when to handover to a human specialist. RadLE 2.0 measures this through five scores: the Confidence Weighted Index (Primary Metric), Reliability Index, Accuracy Index, Safety Index, and Handover Readiness Index.
Our patients have been increasingly using frontier AI models (through their chat interfaces) for their health concerns. A red-teaming study in npj Digital Medicine showed that publicly available chatbots often give unsafe answers to patient-posed medical questions. As more patients start uploading their medical records and radiological images on these platforms, and relying on their outputs (often more than their doctors), we urgently need more robust benchmarking systems. Currently, there are only a few independent and neutral benchmarks that exist in the healthcare domain to assess how reliable and safe current frontier AI models are.
We have also been witnessing young clinicians, medical students, and first-year residents during the course of their training, outsourcing their judgment and diagnostic reasoning to AI models with serious discussion now emerging around deskilling and never-skilling in the age of AI; and rightly so. They are unaware how AI models can confidently lead them to a wrong diagnosis in rare cases.
Radiology has already lived through one full cycle of AI overclaiming. In 2016, it was declared that we should stop training radiologists, because deep learning would soon do the job better. Almost a decade later, radiologists are still here and overworked despite AI being integrated almost everywhere in modern radiology workflows.
It is important to note that in critical areas like medicine, we are seeing very important progress in retrieval-augmented generation (RAG) with grounding of answers in updated guidelines and clinical knowledge bases rather than whatever the model remembers internally. But to retrieve safely, a model must first understand that it does not itself know the answer. Kalai et al. recently argued in Nature that evaluation metrics that reward only accuracy, in turn encourage models for guessing rather than admitting uncertainty. A model which is unaware that it does not know the answer to a question will confidently make up a wrong answer. Previous publications like Kadavath et al. and the SelfAware benchmark formalized this problem of whether models are aware of what they don't know, but no such benchmark has existed in medicine. This capability, we believe, is the very foundation of autonomous AI agents in medicine and an agent that cannot recognize the limits of its own knowledge, cannot be trusted to act on it. RadLE 2.0 is our attempt to build exactly that.
02
From RadLE 1.0 to RadLE 2.0
Last year, we built and released the first version of Radiology's Last Exam (RadLE) 1.0. It was our first attempt at testing frontier AI models and presented it as a cutting edge paper at RSNA 2025. What we found in September 2025 was simple and stark: expert radiologists scored around 83%, while the best Frontier AI model scored only around 30%. However within 3 months, Gemini 3.0 Pro had outperformed radiology trainees. In 1.0, we focused on accuracy. But as we went deeper, we realized that accuracy alone is not the right benchmark for autonomous AI agents in radiology.
Today, we are releasing a larger benchmark of frontier AI models as well as the leading open-source and medical vision-language models, for diagnosing single-image spot-diagnosis cases in radiology. In RadLE 2.0, both human readers and AI models may either provide a diagnosis with a five-point confidence score from 0 to 4 or explicitly respond, “I don't know."
For our primary metric, the RadLE-C (Confidence Weighted Index), correct diagnoses receive confidence-weighted positive points, while incorrect diagnoses receive confidence-weighted penalties. An “I don't know" response receives neither credit nor penalty. The resulting score is normalized to a scale from 0 to 2000. Consequently, a model that repeatedly guesses incorrectly with high confidence will fall in the benchmark, even if its raw accuracy appears competitive.
We also release four secondary metrics, each designed to answer a different question about readiness for autonomous diagnosis:
RadLE-R (Reliability Index): Among diagnoses given with high-level confidence (Likert 3 or 4) how often is the system correct? This measures whether the outputs that a model presents most confidently can actually be trusted.
RadLE-A (Accuracy Index): What proportion of all benchmark cases does the system diagnose correctly, irrespective of confidence? This preserves the conventional measure of raw diagnostic performance and allows direct comparison with accuracy-only benchmarks.
RadLE-S (Safety Index): How effectively does the system avoid confident diagnostic errors? Incorrect answers are weighted according to confidence, so a highly confident wrong diagnosis carries a greater safety penalty than a low-confidence error.
RadLE-H (Handover Readiness Index): Can the system use its own confidence to determine when to act autonomously and when to defer to a specialist? For this analysis, diagnoses with Likert 3 or 4 are considered eligible for autonomous handling, while Likert 0 to 2 responses and “I don't know" are routed to a human specialist radiologist. RadLE-H combines the reliability of autonomously handled outputs, the proportion of cases handled autonomously, and the proportion of potential errors successfully deferred.
04
Prior Studies and the Need for an Updated Benchmark
If benchmarks mostly reward accuracy, and do not give meaningful credit for admitting uncertainty, then we are actually incentivizing models to guess. In medicine, especially as we are paving the way for autonomous systems, this gap raises significant patient-safety concerns. The results from Humanity's Last Exam proves that models can be confidently wrong, which ultimately has a very high cost in medicine.
A growing number of studies in healthcare already demonstrate this problem. Benchmarking studies of LLM confidence on clinical questions have shown that even better-performing models often have very little separation between the confidence they give for correct answers and for incorrect ones, and work on calibration error in medical QA confirms that stated confidence rarely matches actual correctness. Similar miscalibration has now been shown in multimodal medical VQA, where it can directly lead to misdiagnosis. But almost all of this work has been on text-based questions or general medical VQA. RadLE 2.0 is our first benchmark to test the readiness of vision language models (VLMs) for autonomous radiology diagnosis and asks whether their confidence meaningfully distinguishes answers that may be trusted from those that should be deferred.
That distinction becomes visible when the model outputs are decomposed rather than reduced to a single accuracy number. The following figures show the complete response profile across 200 cases: the number of correct diagnoses, incorrect diagnoses, and “I don't know" responses, together with the confidence assigned to each answer.
Among proprietary frontier models, several models produced substantial numbers of high-confidence errors, meaning that their confidence could not consistently be interpreted as evidence that the diagnosis was trustworthy. The human expert baseline achieved a more favourable balance between correct diagnoses and confident errors, although human performance was also imperfect.
The pattern becomes even more pronounced when we examine open-weight and medical vision-language models.
Among open-weight and medical VLMs, the gap from human experts widens substantially. Several systems attempted nearly every case while producing very few correct diagnoses. More importantly, many of their incorrect outputs were delivered with moderate or high confidence.
05
Why a benchmark for Autonomous Healthcare AI Agents was Urgently Needed
A lot of discussion has happened in the last couple of months with respect to autonomous AI agents and how frontier models are outperforming specialized medical models today. There have been seminal papers coming out recently with researchers trying to prove that frontier models are becoming significantly better. Last month, in the same week, two papers in Nature made the case for MIRA, an autonomous EHR agent that outperformed physicians in diagnostic accuracy in simulations on real patient cases, and AMIE which was non-inferior to primary care physicians in management reasoning across 100 multi-visit scenarios. Alongside this, a highly discussed paper by Vishwanath et al. in Nature Medicine showed that general-purpose frontier LLMs can outperform specialized clinical AI tools on medical benchmarks. We have seen the same trend ourselves and agree that these models are indeed getting better. However, if we are really moving towards autonomous healthcare agents, then those agents must know when they are unsure of a diagnosis and refer to a human specialist.
MDs, CEOs, and investors of billion-dollar companies have been going on record recently and highlighting irresponsibly that AI models are now better than 99% doctors. Many of these claims come from anecdotal examples, controlled benchmarks, and simulated settings. The counterweight is becoming clearer and a recent JAMA Network Open study evaluated 21 off-the-shelf LLMs across sequential stages of clinical reasoning. The study found that, despite progress, current models remain limited in early diagnostic reasoning and are not ready for unsupervised clinical-grade deployment.
The human body is very unique. Every single person has their own unique imaging signature, so it is often unlikely that one will have the exact same image for two patients for the exact same disease (only one of which the model may have been exposed to during its training). Hence, it is incredibly important for the owners of these companies to understand and accept that AI models are not yet ready for autonomous deployments and diagnosis. Which is why we have created RadLE 2.0 to demonstrate where the gap lies in medicine today.
We do not want to claim that radiologists and trainees are perfect and always correct, but neither are frontier AI models. We have tested both radiologists and AI models in exactly the same setup, asking the exact same question.
The results of our benchmark demonstrate the need for strict regulations and mandatory disclosures by AI models to ensure that unreliable AI models do not cause harm to patients. Because if people start completely relying and trusting these models, it is for certain going to lead to harmful consequences for the community. This would be because of the blind trust they put in these AI chatbots, and the false sense of security which may disincentivize them from visiting a doctor, thinking they are absolutely fine because their "AI Doctor" is telling them so.
06
What next?
RadLE 2.0 is not intended to be a one-time benchmark. We see it as the beginning of a continuously evolving evaluation framework for autonomous AI diagnosis in medicine. As new frontier and medical vision-language models become publicly available, we intend to evaluate all of them under similar protocol, allowing the community to compare systems fairly over time using a consistent, transparent and most importantly, neutral benchmark.
This technical report provides an early look at our methodology and findings. We will be evaluating additional frontier models that are expected to be released over the coming weeks. Once these experiments are complete, we will release the full RadLE 2.0 paper, including the benchmark design, scoring methodology, participating radiologists and residents, statistical analyses, token usage, inference costs, confidence calibration analyses, and detailed error taxonomy.
More importantly, our long-term goal is not simply to build another leaderboard, but to develop a family of uncertainty-aware benchmarks that can evaluate whether AI systems are truly ready for autonomous decision-making across healthcare systems.
We look forward to working with medical AI companies, frontier AI labs, and institutions developing and deploying AI agents to help develop and deploy AI agents safely and ensure that claims of expert-level performance are accompanied by rigorous, independent evaluation.
For collaborations, please contact [email protected].
This work was made possible by the entire RadLE team co-led by Suvrankar Datta, Divya Buchireddygari, and Hakikat Bir Singh Bhatti together with our contributing radiologists and research collaborators viz. Anjali Agrawal, Anurag Agrawal, Arjun Kalyanpur, Chirag Mahajan, Devyani Singh, Dhanush Jayanna, Manish Jha, Nishtha Mahajan, Prathamesh Tadage, Prerna Priyadarshini, Rahul John Joseph, Rajesh Vanagundi, Shravan Reddy, Shreyas Reddy K, Shyam Kumar, Siddharth Reddy, Siddharth Valecha, Snigdhaa Rajvanshi, Suyash Gunjal, Swarna Radhakrishnan, Unnathi Nayak, Upasana Karnwal, Vikas H P, and Yash Jakhotia.
We are grateful to the Simons Foundation, the Koita Foundation, and various other funding partners for their support. Further details are available at crashlab.in.
07
Additional Reading Materials
- 1.Bean et al., Reliability of LLMs as medical assistants for the general public, Nature Medicine
- 2.Draelos et al., Large language models provide unsafe answers to patient-posed medical questions, npj Digital Medicine
- 3.Ke et al., AI-induced never-skilling in medical education, Nature Medicine
- 4.Choudhury and Chaudhry, Large Language Models and User Trust, JMIR
- 5.Tejani et al., Understanding and Mitigating Bias in Imaging Artificial Intelligence, RadioGraphics
- 6.Abo El-Enen et al., A survey on RAG models in healthcare, Neural Computing and Applications
- 7.Kadavath et al., Language Models (Mostly) Know What They Know, arXiv
- 8.Yin et al., Do Large Language Models Know What They Don’t Know?, ACL Findings / SelfAware
- 9.Kalai et al., Evaluating large language models for accuracy incentivizes hallucinations, Nature
- 10.OpenAI, Why language models hallucinate
- 11.Datta et al., Radiology's Last Exam (RadLE), arXiv
- 12.Phan et al., Humanity's Last Exam, Nature
- 13.Omar et al., Benchmarking the Confidence of Large Language Models in Answering Clinical Questions, JMIR Medical Informatics
- 14.Boie et al., Calibration of Self-Reported Confidence and Accuracy of LLMs in Medical Question Answering, Journal of Medical Systems
- 15.Vishwanath et al., General-purpose LLMs outperform specialized clinical AI tools on medical benchmarks, Nature Medicine
- 16.Ferber et al., Towards autonomous medical artificial intelligence agents, Nature / MIRA
- 17.Liévin et al., Towards Conversational AI for Disease Management, Nature / AMIE
- 18.Rao et al., Large Language Model Performance and Clinical Reasoning Tasks, JAMA Network Open