Diagnostic AI: accuracy is only part of the decision
By ChatGPT Health. Educational synthesis prepared with AI assistance. Sources checked September 28, 2026. Clinically reviewed by Harvey Castro, MD, MBA, on September 28, 2026. Approved as educational content. This page is not a clinical protocol or a claim of independent clinical peer review.
A diagnostic AI result must be interpreted in its intended population and workflow. Performance in a test dataset cannot answer every question about care.
Specify what the system actually does
Distinguish prioritizing an image, identifying a finding, suggesting a diagnosis and making an autonomous decision. These actions have different consequences. Specify the reference standard used in the evaluation and whether the clinical team had access to information the model did not.
Look beyond a single score
Sensitivity describes how often a system identifies the condition among those who have it. Specificity describes how often it correctly identifies its absence. The chance that a positive result represents disease also depends on the population being tested. Ask about calibration, missing data, excluded cases and the consequence of an error. A high average score can conceal important subgroup differences.
A bounded example
The pivotal autonomous diabetic retinopathy study examined a specific system in primary care with a defined target condition and workflow. Its reported sensitivity and specificity should not be generalized to unrelated diseases or devices. Use the original report to inspect eligibility, handling of images and referral requirements. Pivotal autonomous retinopathy trial.
A practical review conversation
Before relying on a tool, ask:
- Was the system tested outside the development setting?
- Does the local population match the evaluated population?
- Which uncertain or ungradable inputs must be referred for human assessment?
- Can the clinician disagree, and is the alternative workflow usable?
- Does deployment improve decisions or patient outcomes, or only test performance?
For patients
Ask what role AI played and who reviewed the result. A test is one part of clinical assessment. A product can have a legitimate narrow use while still being unsuitable for another question. Treatment decisions require the broader patient context.
Sources and further reading
Continue learning
Editorial standards and corrections · Suggest a correction. Please do not send identifiable patient information.