Evidence over explanations: put medical AI to the test – npj Artificial Intelligence

A friction in the deployment of artificial intelligence (AI) in medicine is the mismatch between how modern systems are built and how medicine draws reliable inferences. Clinical research advances by articulated steps, mirroring the steps of the canonical scientific method: posing questions, proposing hypotheses, designing experiments, analyzing results, and communicating successes and failures with bounded claims. Contemporary medical AI — spanning imaging, clinical prediction/risk models, decision support, and, increasingly, generative systems such as large language models (LLMs) — compresses this sequence. It optimizes end-to-end performance and offers post-hoc explanations after the fact, creating what we define as the “epistemic gap” between what a system does and what we can justifiably conclude about why it works, for whom, and under which conditions¹.

As the terminology in this area is inconsistent across communities, we use intrinsic (ante-hoc) interpretability to denote models whose decision logic is transparent by design, like the mapping from inputs to outputs can be directly inspected at the level of the model form and post-hoc explainability to denote methods applied after training to produce explanatory artifacts about an otherwise opaque model’s behavior, like saliency/attribution maps, feature importance, counterfactuals, and surrogate models².

The limits of explanation are well documented. Strong critiques have argued that post-hoc explanations can distract from a more fundamental requirement: if decisions are consequential, use models whose workings are intrinsically interpretable or, at a minimum, whose claims can be put on trial through rigorous tests². Health-care-specific analyses have highlighted the false hope of current XAI practices, noting that popular saliency and attribution methods can fail basic sanity checks, change under small input perturbations, or remain stable when the model is randomized^3.4. In radiology, quantitative evaluations in mammography confirm that saliency maps can be inconsistent and poorly calibrated to clinically relevant locations⁵, while broader discussions emphasize that explanations are heterogeneous objects—how, why, and when to explain are distinct questions that technical toolkits often conflate⁶. Systematic reviews suggest that despite widespread invocation, XAI is rarely used in a way that meaningfully supports clinical decision-making or safety claims³ although they do not delegitimize explanation, they restrict what explanatory artifacts can credibly guarantee^7,8,9.

What must be guaranteed in medicine is not introspective access to a model’s internal states but grounds for believing that its outputs rest on meaningful signals and will remain reliable when the environment changes. Two dimensions are central.

First, causal alignment. Are predictions driven by features plausibly related to disease biology or clinically relevant intermediates? In imaging, this means distinguishing decision rules based on lesion morphology or tissue texture from rules keyed to scanner logos, image borders, or view markers. The literature offers cautionary tales: high internal accuracy on one hospital’s chest radiographs collapsed on external data because models exploited site-specific confounders rather than pathology¹⁰. In breast imaging, the acquisition signatures, vendor differences, and demographic imbalances can serve as shortcuts that inflate in-sample metrics while undermining transportability^5,11. Because shortcut signals in imaging can be site- and workflow-specific, external validation is necessary but often insufficient on its own; it should be paired with explicit shortcut tests that manipulate suspected nuisance cues¹². Causal alignment can be tested without peering inside the network through intervention-style probes. In tabular data this may involve manipulating or withholding candidate predictors and re-testing on held-out cohorts. In high-dimensional imaging, features need not be discrete variables: they can be spatial regions, acquisition and processing signatures, borders/overlays/markers, or latent concepts (e.g., organ or lesion compartments). Accordingly, tests can include targeted occlusion or ROI-restriction (e.g., lesion or organ masks), removal of view markers/borders, counterfactual swaps of background/context, and stress tests that preserve pathology while perturbing nuisance factors (scanner/vendor/site texture or preprocessing). Negative controls remain central: if predictions change when only irrelevant image factors are altered, the model is not causally aligned.

Second, invariance. Do performance and clinically relevant aspects of behavior remain stable across plausible shifts. Like scanner models, acquisition protocols, sites, and demographic subgroups? Shortcut learning is a known failure mode whereby models rely on correlations that maximize training performance but are brittle under distribution shift¹³. In practice, invariance is assessed by stratified external validation designed to stress the model along clinically salient axes. Instability is not automatically disqualifying, but it obliges either scope restriction (narrow intended use) or targeted remediation (re-training, re-calibration).

Generative LLMs provide a clear worked example of the same governance problem in a different modality. They are vulnerable to confident, unsupported statements and fabricated premises. Retrieval-augmented generation (RAG) can constrain outputs to cited sources and conservative refusal behaviors can reduce speculative answers, but these mitigations do not create causal understanding¹⁴. Here, too, governance should emphasize tests. For example: Does the system consistently ground each clinical recommendation in evidence? Can it be induced to invent references or contraindications? Do guardrails hold when prompts contain adversarial mixtures of correct and incorrect statements? The goal is not to philosophically dissolve opacity; it is to expose claims to disciplined interrogation.

The objection most often raised is pragmatic: even if testability is the right principle, who will run the tests, and with what data? One answer is to bring development closer to where care occurs, with what we call “Institutional AI”, a hospital-embedded pipelines that offer a practical route to testability. Local data reflect the population, scanners, protocols, and reporting culture of deployment. When models are trained or calibrated on such data and validated prospectively, ecological validity improves and calibration error typically narrows^15,16. Yet local does not mean unbiased. Site-specific labeling conventions, measurement errors, and under-representation of rare subgroups can be amplified, and portability across a regional network can degrade^13,15. The remedy is not to retreat from local development but to pair it with environment-stratified external audits. Internal fit and external generalization must be treated as co-equal design constraints.

Before deployment or update, institutions should define the minimum acceptable discrimination (with uncertainty), calibration on the local case-mix and on external cohorts, and invariance across scanners, sites, and key subgroups. When explanations are used for user-facing tasks (for example, lesion localization), tests must quantify explanation fidelity and stability rather than assuming saliency maps are accurate by default. After deployment, drift surveillance should monitor both input data and performance relative to preregistered baselines, with prespecified rollback or recalibration triggers. Crucially, local development must be linked to external, environment-stratified audits that can surface shortcut behavior and subgroup harms early.

This program re-embeds the logic of the scientific method into the lifecycle of medical AI. Hypotheses become explicit claims about information pathways and clinician interaction¹⁷. Experiments become preregistered quantitative and human-factors evaluations that a system must pass to progress, aligned with established protocol and trial reporting guidance (e.g., SPIRIT-AI/CONSORT-AI where applicable)^18,19. Conclusions become bounded intended uses with declared failure modes. Communication becomes the publication of full results, including degraded external performance and negative ablation studies, aligned with established reporting standards—TRIPOD for prediction models and the CONSORT-AI and SPIRIT-AI extensions for interventional trials^18,19,20,21. Transparency remains desirable, but testability is the stronger guarantee when systems resist full interpretation.

There are trade-offs. Institutional pipelines require data engineering, MLOps, statistical audit, and governance, which risks confining their feasibility to well-resourced centers: a real inequity that must be addressed²². Two mitigations are feasible. First, standards and tooling can be shared. The tests like provenance schemas, preregistration templates, invariance batteries, and drift dashboards, can be packaged and disseminated as open protocols, lowering the fixed costs for smaller hospitals. Second, networks of centers can coordinate distributed validation: a system locally calibrated on one site can be audited across a consortium to quantify generalization and equity before widespread use^15,16.

Overall, where interpretable models can meet clinical performance requirements, they should be preferred. But medicine often confronts problems where fully transparent-by-design models are not yet available at clinically acceptable performance or breadth of use—particularly in free-text understanding, many high-dimensional imaging tasks, and multi-modal fusion. In these domains, the practical choice is often not between full transparency and reckless opacity, but between restricting modeling to transparent forms or partial/indirect interpretability aids (with narrower scope or degraded performance), and using higher-capacity opaque models under a testability-first regime with preregistered acceptance thresholds, shortcut-focused stress tests, and continuous monitoring. Testability provides a principled middle path. It does not ask clinicians to accept stories that might be unfaithful representations of what a model is doing; it asks institutions to gather empirical evidence that a system’s behavior is causally sound and robust for the intended population and use.

A conceptual point frequently overlooked in literature is that not all non-human decision criteria are spurious^23,24. Medicine is already comfortable with signals that clinicians cannot directly perceive but accept because they can be validated and mechanistically situated: a distinction articulated in the black-box decision debate and echoed across the broader explainability ethics literature^23,24. In imaging AI, models may identify higher-order parenchymal patterns that correlate with stromal biology without being reducible to existing lexicons¹³. The challenge is to separate such non-human-but-causally-valid signals from genuinely spurious shortcuts. That separation cannot be achieved by saliency maps that decorate predictions after the fact; it requires the causal and invariance tests described above^3,13.

Language models merit a similar reframing. Calls for explainability often reduce to better confidence measures or traceable citations, both useful but insufficient. A clinically acceptable system should be able to demonstrate, under adversarial evaluation, that fabricated citation events and unsupported factual claims are below prespecified acceptability thresholds for the intended use, that the model reliably abstains when evidence is absent or insufficient, and that retrieval constraints (when used) measurably bind outputs. RAG architectures help¹⁴, but the guarantee rests on how they are tested and monitored in the clinical setting, not on architectural labels. Because hallucinations cannot be eliminated entirely, institutions should treat acceptability as a managed error budget: define task-specific metrics (e.g., citation fidelity, unsupported-claim rate, abstention calibration), set go/no-go thresholds and rollback triggers, and restrict higher-risk clinical decision support uses to regimes where the residual error budget is extremely small and abstention is frequent. For example, tolerable error rates are very different for drafting administrative text, summarizing retrieved guideline passages with verifiable citations, and generating patient- or clinician-facing recommendations: only the latter demands near-zero tolerance for source fabrication and should default to ‘no answer’ unless the evidence base is explicitly retrieved and consistent.

Medicine does not need mysticism about machine understanding, nor does it need to reduce every model to a set of human concepts to be safe. It needs disciplined claims, adversarial tests, and bounded conclusions. Interpretable models should be used where they suffice. Where they do not, opaque models can be responsibly deployed if, and only if, they pass tests that establish causal alignment and invariance for the stated context of use. Institutional AI offers a practical framework to do so. The result is not perfect transparency, but something closer to medicine’s traditional strengths: explicit hypotheses, disciplined experiments, robust analyses, and full communication, including informative failures. That is how the epistemic gap narrows: putting models on trial, not by asking them to tell better stories.