Clinical prediction models developed using artificial intelligence and traditional regression techniques can achieve virtually identical overall discrimination while generating sharply conflicting risk estimates for individual patients, according to an extensive diagnostic evaluation published in Diagnostic and Prognostic Research. Evaluating diagnostic data from thousands of individuals suspected of having deep vein thrombosis, an investigative team examined how different algorithmic frameworks translate standard clinical indicators into diagnostic probabilities. The researchers demonstrated that crude summary metrics such as the area under the receiver operating characteristic curve frequently conceal profound discrepancies in patient-level probability outputs. Even when multiple algorithms appear equally effective at separating positive cases from negative cases across a broader study cohort, the specific probability assigned to an individual patient fluctuates substantially depending entirely on the mathematical structure of the chosen predictive architecture. These observed discrepancies indicate that selecting a prediction model based exclusively on aggregate discriminative performance may generate conflicting clinical recommendations at the bedside.
In routine medical practice, clinicians regularly combine multiple demographic, historical, and physiological variables to evaluate an individual’s probability of having an underlying condition or developing an outcome in the future. Clinical prediction models formalize this process by applying mathematical equations that link multiple predictive variables to estimated disease risks. Historically, medical researchers have relied on conventional regression approaches such as unpenalized multivariable logistic regression and Cox proportional hazards models to derive these probability estimates. In recent years, however, machine learning algorithms and artificial intelligence methodology have experienced a notable surge in popularity across biomedical prognostics and diagnostics. The widespread availability of automated, off-the-shelf software tools provides model developers with ample choices to deploy, compare, and select from a diverse spectrum of computational algorithms within identical clinical datasets.
Previous investigations within low-dimensional settings—defined as analytical environments where the number of candidate predictors remains relatively small compared to the total volume of observed individuals—have suggested that modern artificial intelligence algorithms can occasionally outperform classical regression techniques, particularly regarding discrimination measures such as the area under the receiver operating characteristic curve. However, comparative studies repeatedly demonstrate that differences in discriminative performance between complex machine learning models and standard multivariable regression models are often small or practically negligible. When developers observe that an unpenalized logistic regression model, a random forest ensemble, and an artificial neural network achieve nearly indistinguishable discriminative indices on a given clinical dataset, they frequently infer that the underlying models are essentially exchangeable.
To examine whether superficially equivalent models generate concordant clinical guidance, the researchers analyzed an individual participant dataset compiled from eight prospective diagnostic investigations involving patients presenting with leg symptoms suggestive of deep vein thrombosis. Deep vein thrombosis represents a dangerous vascular pathology wherein an undetected blood clot within the deep venous system can mobilize, travel toward the pulmonary arterial circulation, and trigger a life-threatening pulmonary embolism if left untreated. Across the eight incorporated cohort studies, the analytical sample comprised 6,087 suspected individuals, of whom 1,146 patients—constituting nineteen percent of the total cohort—received a verified diagnosis of deep vein thrombosis confirmed by objective compression ultrasonography of the leg as the reference standard. The study cohort exhibited a mean age of fifty-nine years with a standard deviation of seventeen years, and women represented sixty-two percent of all participants. To prevent extraneous variation arising from alternative missing-data handling strategies, the investigators restricted their analysis to the primary imputed dataset.
The investigative protocol evaluated a fixed panel of six preselected clinical predictors that have been established across earlier deep vein thrombosis diagnostic literature. These predictive features encompassed chronological age treated as a continuous metric, patient sex recorded categorically as male or female, qualitative d-dimer assay results categorized as positive or negative, documented personal history of deep vein thrombosis, active malignancy, and the presence of an alternative medical diagnosis deemed as likely as or more likely than deep vein thrombosis. The researchers deliberately avoided data-driven predictor selection, stepwise elimination, or automated feature pruning, ensuring that all six established prognostic variables were uniformly forced into each mathematical model. By preserving an identical predictor set and a singular binary outcome across every framework, the experimental design isolated the intrinsic mathematical properties of the modeling architectures as the sole driver of variation in downstream probability estimations.
Using the caret meta-package within the R statistical computing environment, the research team constructed five distinct model architectures across the shared clinical cohort: unpenalized logistic regression, ridge-penalized logistic regression, random forests, support vector machines, and feedforward artificial neural networks. In both unpenalized and ridge logistic regression architectures, the six clinical predictors entered the equation as strictly linear terms without interaction effects, with unpenalized regression optimizing the standard binomial likelihood and ridge regression incorporating regularization to optimize a penalized binomial likelihood. In contrast, the random forest, support vector machine, and neural network algorithms possessed the structural flexibility to capture non-linear relationships and intricate feature interactions automatically. Random forest models comprised one thousand individual decision trees trained on bootstrapped subsets, generating probabilities through majority voting. The support vector machine constructed a separating hyperplane with maximum margin and utilized a secondary logistic regression model trained on algorithm outputs to yield probability estimates, while the neural network utilized a single hidden layer and a sigmoid output activation function.
Algorithm hyperparameters were systematically tuned using ten-fold cross-validation specifically structured to optimize the area under the receiver operating characteristic curve. A formal sample size calculation performed using the pmsampsize package indicated that developing a reliable logistic regression prediction model within this clinical context required a minimum cohort of at least 266 individuals encompassing 48 confirmed events, corresponding to approximately 7.98 events per candidate predictor parameter assuming a shrinkage factor of 0.90, an anticipated concordance index of 0.81, and an expected outcome prevalence of eighteen percent. Because the actual study cohort contained 6,087 individuals and 1,146 verified events, the sample comfortably satisfied conventional sample size thresholds, providing ample statistical power to fit multivariable models without encountering severe overfitting artifacts stemming from data scarcity.
When the researchers assessed overall discriminative capacity, the five modeling techniques achieved remarkably similar summary metrics, with area under the curve point estimates ranging tightly between 0.81 and 0.84 across the entire patient cohort. Unpenalized logistic regression, ridge logistic regression, and the artificial neural network achieved identical discrimination point estimates of 0.84, with identical ninety-five percent confidence intervals extending from 0.83 to 0.85. The support vector machine achieved an area under the curve of 0.82 with a ninety-five percent confidence interval of 0.81 to 0.83, while the random forest model exhibited the lowest discrimination point estimate at 0.81, accompanied by a ninety-five percent confidence interval spanning 0.80 to 0.82. Viewed solely through the lens of receiver operating characteristic analyses, these aggregate findings would typically lead model developers to conclude that all five predictive algorithms delivered essentially equivalent diagnostic accuracy across the patient population.
In marked contrast to their harmonious discriminative capacities, the five algorithms diverged substantially when evaluated for calibration, which reflects the numerical correspondence between predicted probabilities and observed event rates across the risk spectrum. Calibration curves revealed that unpenalized logistic regression and the artificial neural network maintained good calibration across their complete prediction ranges, closely tracking the ideal diagonal reference line. Ridge logistic regression displayed acceptable calibration but exhibited slight underestimation among higher risk strata. Conversely, the random forest and support vector machine models exhibited pronounced miscalibration patterns: the random forest consistently overestimated probabilities across extensive sections of the prediction range, whereas the support vector machine consistently underestimated predicted probabilities. Furthermore, the overall span of generated probabilities differed notably across architectures, with random forest estimates dispersing across the full spectrum from zero to one, whereas neural network predictions occupied a compressed range from zero to 0.70.
Because diagnostic decision-making in deep vein thrombosis hinges on actionable probability cutoffs rather than abstract ranking scores, the researchers evaluated patient classification at a fixed probability threshold of 0.02. This low decision threshold is routinely mandated in acute thrombotic workflows where clinicians must prioritize diagnostic sensitivity to prevent fatal pulmonary emboli. Despite their nearly identical discrimination indices, the models yielded starkly divergent classification metrics at this uniform operational threshold. The random forest model produced balanced sensitivity and specificity values of 0.81 each, resulting in the highest positive predictive value of 0.50 among all evaluated algorithms, whereas the neural network achieved a sensitivity of 0.82 and a specificity of 0.75. Direct pairwise comparisons across the 6,087 patients revealed that estimated probabilities for identical individuals fluctuated dramatically between different modeling approaches. For the individual with ID 195, the support vector machine calculated an estimated probability of 0.10, whereas the random forest estimated 0.00. Similarly dramatic disparities emerged for patient ID 1142, whose estimated probability was calculated as 0.36 by unpenalized logistic regression, 0.37 by the neural network, but merely 0.03 by the random forest ensemble. Investigation of predictor influence via Individual Conditional Expectation plots demonstrated that d-dimer assay results and the presence of an alternative diagnosis were the most influential predictors across all five architectures. The study authors noted that these individual-level discrepancies illuminate epistemic uncertainty arising from structural modeling choices, echoing findings from previous cardiovascular studies where individual risk estimates varied widely between QRISK3, random forest, and neural network algorithms. Acknowledging that automated tuning procedures within caret can adversely impact calibration, the researchers emphasized that developers must move beyond summary discrimination by rigorously assessing calibration and prediction stability under reporting frameworks like TRIPOD-AI and PROBAST-AI.
Subject of Research: Discrepancies in individual risk predictions among artificial intelligence and regression models with similar discriminative performance in deep vein thrombosis
Article Title: Prediction models developed using artificial intelligence: similar predictive performances with highly varying predictions for individuals – an illustration in deep vein thrombosis
Article References: Yusufujiang, M., Navarro, C. L. A., Damen, J. A., Takada, T., Geersing, G.-J., Hooft, L., Schuit, E., Moons, K. G., de Jong, V. M., & van Smeden, M. (2026). Prediction models developed using artificial intelligence: similar predictive performances with highly varying predictions for individuals – an illustration in deep vein thrombosis. Diagnostic and Prognostic Research, 10(1), Article 1. https://doi.org/10.1186/s41512-025-00216-5
Image Credits: AI Generated
DOI: 10.1186/s41512-025-00216-5
Keywords: artificial intelligence, machine learning, clinical prediction models, deep vein thrombosis, model calibration, logistic regression, neural networks, random forests, support vector machines, diagnostic performance, Prediction, models
