Introduction

Coronavirus disease 2019 (COVID-19), caused by the severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2), was initially identified at the end of 2019 and subsequently spread internationally as a global pandemic. Individuals who were infected with SARS-CoV-2 can exhibit a broad spectrum of disease severity, ranging from asymptomatic or mild infection to critical illness with multi-organ failure1. By the end of November 2023, over 770 million confirmed cases and over 6.9 million deaths had been reported worldwide2. Initially, COVID-19 was recognized as a simple respiratory tract infection. However, it is now acknowledged as a systemic disease that profoundly affects various systems, including the hematopoietic and immune systems3.

Artificial intelligence has been widely applied in various aspects of human life. In recent years, many scholars have applied machine learning in medicine, particularly in pursuing medical event prediction accuracy. Early research mainly used traditional statistical analysis methods, employing univariate analysis to obtain single risk factors, resulting in relatively limited predictive accuracy. The introduction of machine learning technology can effectively eliminate interference factors, improve prediction accuracy, assist healthcare professionals in promptly identifying high-risk patients and increase the accuracy of severity prediction. Many scholars have developed machine learning models to predict the severity and hospitalization mortality of COVID-19 patients, helping to identify those at high risk of mortality4,5,6. Some of the models exhibited bias in the stratification of subgroups with varying mortality risks. Sharifi-Kia developed a machine learning-based mortality prediction model for COVID-19 patients with a history of smoking in the Iranian population7. Zhuang established a model to differentiate between severe and non-severe COVID-19 patients among the elderly8. In addition to erlderly, having underlying health was also a risk factor for COVID-19-related mortality.

Patients with chronic kidney disease (CKD) were more susceptible to developing severe COVID-19 and had a higher risk of mortality. Compared to the general population, CKD patients were prone to be affected by COVID-199,10. Additionally, compared to individuals with other chronic diseases such as diabetes, coronary heart disease, hypertension, and asthma, CKD patients have a considerably higher risk of mortality due to COVID-19 attacks11,12,13,14. Therefore, timely identification and accurate assessment of disease severity of COVID-19-affected CKD patients facilitates proper clinical decisions. Unfortunately, COVID-19 infection prediction models specifically tailored for the unique population of patients with CKD are not available currently. Therefore, we aimed to develop a machine learning-based prediction model for COVID-19 patients with CKD.

In this study, we retrospectively analyzed clinical laboratory data of CKD patients with COVID-19. The objective was to compare multiple machine learning algorithms and identify the optimal method for predicting the likelihood of COVID-19 mortality in CKD patients. Additionally, we used limited data as the validation dataset to assess the model’s practicality using the same predictive variables and outcome endpoints. We aimed to construct a predicting model that could provide personalized early treatment recommendations for CKD patients at a higher risk of mortality when they contract COVID-19. Our results demonstrated that the final selected model was a valuable tool to assist physicians in their clinical decision-making process.

Methods

Research participants

The clinical data were collected from CKD patients hospitalized in The Second Hospital of Dalian Medical University due to COVID-19 from December 2022 to January 2023. Inclusion criteria: (1) age ≥ 50; (2) meeting the CKD diagnostic criteria; (3) all were first-infected with COVID-19. Exclusion criteria: (1) acute kidney injury at the time of admission; (2) combined with concurrent malignant tumors; (3) post-transplant patients. According to the principles of the propensity score matching (PSM) method, 89 patients of death and 84 patients of survival were included, matched 1:1 based on age, sex, history of alcohol consumption and history of smoking. (PSM is a statistical method used in observational studies to enhance the comparability of clinical indicators among research subjects, thereby reducing bias from confounding factors. PSM utilizes covariates (such as age and gender) to estimate the propensity score for each patient, followed by matching between the mortality group and the survival group to ensure similarity in baseline characteristics. Ultimately, by comparing the matched groups, researchers can more accurately assess the factors influencing survival rates.) The flowchart for the study procedure was shown in Fig. 1.

Fig. 1
figure 1

The flowchart of the study procedure.

Full size image

Definitions of CKD and COVID-19

The diagnosis and staging of CKD were referred to the well-recognized criteria15,16. CKD was defined by an estimated glomerular filtration rate (eGFR) < 60 mL/min/1.73 m2 or an albumin-to-creatinine ratio (ACR) ≥ 30 mg/g, sustained for more than three months17,18. eGFR was calculated using the Chronic Kidney Disease Epidemiology Collaboration equation(CKD-EPI) 202119. CKD was classified into five stages based on the revised Kidney Disease: Improving Global Outcomes (KDIGO) staging criteria20. COVID-19 was confirmed through virus detection by real-time reverse transcriptase-polymerase chain reaction (RT-PCR) analysis of nasal or pharyngeal swab samples combined with chest computerized tomography if necessary21.

Clinical and laboratory data

Their demographics, clinical features, and laboratory routine test data at admission were retrieved from the lab information system. The clinical test data included routine complete blood count analysis including hemoglobin, leukocytes, platelets, absolute neutrophil, and lymphocyte counts. Serum biochemical tests (including renal and liver functions), D-dimer, Cardiac Troponin I, C-reactive protein, Glucose and indicators in arterial blood gas analysis were also collected22.

This retrospective study has obtained approval from the Ethics Committee of the Second Affiliated Hospital of Dalian Medical University. Informed consent was waived by the Ethics Committee of the Second Affiliated Hospital of Dalian Medical University. This study was conducted according to the guidelines outlined in the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) for reporting observational studies23. The endpoint event was the occurrence of death in the hospital. During the data collection and classification process, all the data were de-identified.

Feature selection

We utilized the least absolute shrinkage and selection operator (LASSO) logistic regression analysis to rank the importance of the potential risk factors. Lasso regression functioned as a data dimension-reducing tool by rejecting parameters with lower calculated coefficients24,25. It was useful for important variable selection and reducing the number of variables for model construction24. In LASSO regression, the beta coefficients of variables that not strongly associated with the outcomes would approach zero. These variables will be automatically removed. The degree of variable-reducing in Lasso regression was decided by the lambda value. A large lambda value imposed a strong penalty on the model with more variables, leading to left a model with a relatively smaller number of variables. To set a rational cut-off value of lambda, the ten-fold cross-validation procedure was applied to minimize the mean squared error (MSE) of lambda26. By selecting the optimal lambda value corresponding to the minimal MSE of 0.066 (Fig. 2A), 13 variables were retained. The confidence of the parameter selection in the LASSO model was confirmed by using 10-fold cross-validation, which yielded a minimum error (Fig. 2B).

Fig. 2
figure 2

The selection of potential risk factors by LASSO. (A) Plot of the LASSO coefficients. Each curve in the graph represented a coefficient change trajectory for an independent variable. The vertical axis represented the values of the coefficients. The bottom horizontal axis represented the log(λ), and the top horizontal axis represented the number of variables with non-zero coefficients. (B) Cross-validation error curve. The vertical axis represented Binomial Deviance, which could be understood as the magnitude of the model’s error. The bottom horizontal axis represented log(λ) as in (A), and the top horizontal axis showed the number of variables with non-zero coefficients in the model. The aim was to select a model with fewer variable features and minimal error. The left dotted line indicated the boundary of the acceptable errors and the right one indicated the boundary of acceptable feature numbers.

Full size image

Machine learning (ML) algorithms and model establishment

We employed five algorithms to construct the prediction models, including XGBoost, logistic regression, light gradient-boosting machine (LightGBM), random forest and AdaBoost. The features confirmed by LASSO regression were fed to machine learning algorithms to predict the occurrence of death. For each modeling process, we implemented five-fold cross-validation. The training set was randomly divided into five groups, with four groups selected as the training groups to train the model, and the left one served as the internal validation set to evaluate model performance.

Model evaluation and optimization

For each model, the training-validation process was repeated 5 times to evaluate the robustness of the model. Each model’s performance was evaluated by receiver operating characteristic (ROC) curve, calibration curve and decision curve analysis (DCA). The areas under the ROC curves (AUCs) were utilized to measure the discrimination ability of the corresponding prediction models. Calibration curves were conducted to assess how much the model-predicted events were closer to the actual events. Decision curve analysis (DCA) was used to evaluate the utility and net benefit of each model for decision-making. To avoid model overfitting, once the best model was determined, we optimized variable selection based on the feature importance of the optimal model. Ultimately, the top eight ranked variables were selected for constructing the final model. The Shapley Additive exPlanations (SHAP) analysis was subsequently employed to illustrate the significance of each retained parameter. Additionally, we compared the diagnostic performance of the predictive model with the traditional risk assessment tool, APACHE II.

Statistical analysis

All statistical analyses were conducted using R software for Windows (Version 3.6.3, https://www.r-project.org/) and the Beckman Coulter DxAI platform (https://www.xsmartanalysis.com/beckman/login/). Categorical variables were presented as frequencies with percentages, and continuous variables were presented as mean ± standard deviation (SD) or median with interquartile range (IQR). Continuous data were tested for normality using the Kolmogorov-Smirnov test. The Mann-Whitney U test and t-test were adopted when appropriate to compare the means between continuous variables. Categorical variables were cross-tabulated to examine the independence among them. The χ2 test or Fisher’s exact test was used as necessary. All the reported p-values were two-tailed tested. P < 0.05 was regarded as statistically significant.

Results

Baseline characteristics

A total of 173 CKD patients with COVID-19, including 89 dead patients and 84 discharged patients, were included in the model construction process. Their demographic characteristics and clinical features are listed in Table S1. Arameters showing significant differences between the non-survival and survival groups were mostly the biochemical items and the blood cell counting parameters (Table S1).

Model construction and validation

The potential risk factors obtained from LASSO regression were input into each ML algorithm (XGBoost, logistic regression, LightGBM, random forest, AdaBoost). The validation results of the five models are presented in Table 1. The ROC curves in Fig. 3A showed that the LightGBM had an outstanding discriminative ability, achieving the highest AUC score among the five models. The forest plot indicated that the AUC of LightGBM was significantly higher than the other four models (Fig. 3B). In the calibration plots of the five models, LightGBM achieved the lowest score of 0.168 among the studied models, The Brier score, which measures the difference between the predicted and actual probabilities, ranging from 0 to 1, suggested higher consistency with lower score (Fig. 3C). The decision curves of the five models, as shown in Fig. 3D, positioned LightGBM almost consistently above the treat-none line”. (DCA evaluated the net benefit of using a model for clinical decision-making compared to no treatment. If the model curve was above the baseline, it indicated that using the model could provide higher net benefits). Collectively, based on the comprehensive evaluation using multiple statistical methods, LightGBM exhibited excellent discrimination ability in predicting the risk for COVID-19-related CKD death.

To avoid model overfitting, the top 8 features with the highest importance in the LightGBM model were selected for final refinement. The ROC curve in Fig. 3E displayed that the LightGBM model showed a good discriminative ability (AUC = 0.833, sensitivity = 0.952, specificity = 0.714) with the optimal threshold of 53.70%. The F1 score was 0.889, indicating that the LightGBM model was highly accurate27. This conclusion was further supported by the calibration curve, suggesting that the risk predicted by the model was close to the actual risk (Fig. 3F).

Table 1 Construction and performance of the five ML models.
Full size table
Fig. 3
figure 3

Evaluation of the 5 ML models. (A) ROC curve for the five models. (B) Forest plot of the AUCs of the five models. (C) The calibration plots of the five models. (D) The decision curve of the five models. (E) ROC for the LightGBM model. (F) The calibration plot of LightGBM model.

Full size image

Model interpretation

Figure 4 illustrates the ROC curves and AUC values for these biomarkers that show significant values in predicting severe COVID-19 among CKD patients. Among them, the prealbumin demonstrated the highest efficiency (AUC = 0.737), followed by neutrophil percent and count (AUC = 0.710 and 0.700). Through the statistical evaluation of AUC, the individual performance of each biomarker fell short when compared to the LightGBM machine learning model (Table 2).

Fig. 4
figure 4

ROC curves of different biomarkers in predicting COVID-19 mortality in CKD patients.

Full size image
Table 2 AUC difference of each biomarker and the LightGBM model.
Full size table

We utilized SHAP scores and summary plots to visualize the significance of each feature in model prediction and interpretability. According to the SHAP analysis of the LightGBM model, serum prealbumin level was one of the main clinical parameters activating the model predictions, other relevant parameters included NP, RI, P50, CO2, Glucose, NC and UA. The summary plot displayed the top eight features ranked in descending order based on their contribution to the model (Fig. 5A). PA negatively correlated to the death event. RI, P50, CO2 and Glucose were positively linked to death event. NC and UA did not show a significant correlation (Fig. 5B).

Fig. 5
figure 5

Contribution of the 8 parameters for LightGBM model. (A) The relative contribution of each parameter. (B) Relationship of each parameter to the death event. On the y-axis, parameters were arranged in descending order of importance. On the x-axis, each patient was represented by a distinct dot, and the x-axis position of each dot reflected the corresponding feature’s SHAP value for that particular row. Higher values were colored in red, and lower values were colored in blue. The red SHAP values to the right of the zero point moved the model towards predictions of death, while the blue SHAP values to the left of the zero point moved the model away from predictions of death. For example, with prealbumin levels being red on the left side, lower prealbumin levels drove the prediction toward a death outcome. Similarly, with Percentage Neutrophils Percentage, RI in Arterial Blood, P50, CO2 and GLU being red on the right side, they drove the prediction towards a death outcome. The influence of Neutrophils and UA on outcome prediction could not be determined due to the lack of a clear correlation between their observed and predicted values. P50 half-saturation pressure of oxygen, CO2 carbon dioxide in serum, Glu glucose, UA uric acid.

Full size image

Comparison of diagnostic performance between the LightGBM model and the traditional risk assessment tool APACHE II

Acute Physiology and Chronic Health Evaluation (APACHE) II was one of the most widely used risk scoring systems, particularly recognized as an effective clinical tool for predicting the mortality of patients with COVID-19. It has demonstrated excellent performance in predicting mortality among COVID-19 patients in the ICU28,29,30,31.

We evaluated the diagnostic accuracy of the LightGBM model in comparison to APACHE II, employing various evaluation metrics (Table 3). The AUC for APACHE II was 0.662. It was lower than the LightGBM model’s AUC of 0.833. Figure 6 illustrated the ROC curves of various models in the studied populations. Apparently, the performance of the LightGBM model was superior to APACHE II.

Table 3 Performance comparison of APACHE II and the LightGBM model.
Full size table
Fig. 6
figure 6

ROC of LightGBM model and APACHE II.

Full size image

External validation

To further validate the prediction model, a cohort of 46 patients (including 19 CKD patients who died from COVID-19) was collected from February to December 2023, who met the same inclusion criteria as previously tested. Limited by the finite number of specimens in the validation set, the LightGBM model showed the values: AUC = 0.675, sensitivity = 0.706, and specificity = 0.655 (Fig. 7). The prediction accuracy was 63% with a positive likelihood ratio of 1.7, indicating that the prediction results were acceptable (Table 4).

Fig. 7
figure 7

ROC for the validation of the LightGBM model.

Full size image
Table 4 The validation results of the LightGBM model.
Full size table

Discussion

COVID-19 has spread globally, resulting in a pandemic and placing significant strain on healthcare systems, highlighting the need for predictive risk models to prioritize patients and facilitate early intervention. Individuals suffering from CKD have an elevated risk of contracting COVID-19 and are more prone to developing severe illness. The mortality rate for CKD patients is as high as 50.6%, which surpasses that of the general population22. Recent investigations have revealed unexpected adverse clinical outcomes and elevated mortality rates among COVID-19 patients with CKD22,32,33. Therefore, to develop early prognostic tools to stratify the risk of poor outcomes is of utmost importance.

In this study, we have successfully developed a prediction model for COVID-19-related mortality in patients with CKD, demonstrating high discrimination capabilities with impressively high AUC values of 0.833, F1 score of 0.889, as well as a remarkable sensitivity and accuracy of 0.952 and 0.743. The calibration plot demonstrated strong consistency between the predicted and actual observed outcomes, as indicated by the low Brier score. This proved that our prediction model could accurately forecast the mortality risk of CKD patients who contracted COVID-19 in real clinical practice. DCA is utilized to evaluate the clinical utility and advantages of the predictive model. The results indicated that the model had significant clinical benefits, potentially reducing unnecessary interventions and improving the accuracy of mortality risk prediction. This early prediction model enabled clinicians to rapidly identify CKD patients with a higher risk of COVID-19-related mortality, providing crucial insights into predicting disease severity and patient survival rates. It was determined that a prediction probability threshold of 53.70% yielded optimal results for risk stratification. If the probability exceeded this threshold, it indicated that patients faced a higher risk of morality. Therefore, proactive targeted interventions can be implemented in advance to effectively manage and mitigate the risks faced by these high-risk patients.

Some studies have reported that “the aggregate index of systemic inflammation (AISI), (neutrophils * monocytes * platelets/lymphocytes)”34, “the systemic immune inflammation index (SII) (platelet counts * neutrophil counts/lymphocyte counts)”22 and so on were able to indicate disease mortality in COVID-19 patients with CKD. In this study, 8 clinical parameters were proved to be closely related to mortality, and their visual representation was done through SHAP analysis. It showed that lower levels of PA, higher NP, RI in arterial blood, P50, CO2, and Glu pushed the model’s prediction toward death. Reports have indicated that SARS-CoV-2 infection leads to a significant decrease in serum PA levels35. It was thought to be the result of the activated inflammation response and the excessively released cytokines. Lower serum PA concentrations were significantly associated with the severity and mortality rate of COVID-1936. CKD patients suffer from protein loss through the kidneys, and COVID-19 further exacerbates the decrease in serum PA, which worsens the malnutrition status of CKD patients. Both lowered PA and malnutrition were negative prognostic factors for COVID-1937. Neutrophils act as the primary cellular defense against infections, and play a crucial role in safeguarding the airway. For severe SARS-CoV-2 infection, there was a notable involvement of neutrophils within the lungs to protect the airway epithelium38. Elevated levels of immature neutrophils were a distinctive feature of severe COVID-19 infection and were associated with aggravating systemic inflammation3,39,40. The initial neutrophil count offered valuable insights for evaluating the severity of COVID-19 in hospitalized patients39. The contribution of neutrophils to the LightGBM model could be ascribed to the presence of a pro-inflammatory state accompanied by impaired innate and adaptive immunity due to the uremic environment which was often found in CKD patients. Impaired renal function could increase the vulnerability of neutrophils to hyper-inflammation and cytokine storm caused by SARS-CoV-2 infection, resulting in severe illness and a higher risk of mortality13. This conclusion could be proved by what had been observed in patients with type 2 diabetes, affected concurrently by COVID-19. Increased white blood cell counts and lymphopenia in COVID-19 patients were frequently encountered41. As for CKD patients who unfortunately succumbed to COVID-19, elevated leukocyte and neutrophil counts, along with decreased lymphocytes were also evident34,42. Notably, nearly all COVID-19 patients experienced an elevation in neutrophil count and a reduction in lymphocytes during severe periods, leading to an increased percentage of neutrophils43. The oxygen tension at which 50% of hemoglobin is saturated with oxygen, is known as P50. Some researchers have pointed out that the median P50 value in patients who died showed a more pronounced right shift compared to survivors, indicating an impaired oxygen binding capacity44. Patients with a higher mortality risk tended to have a higher P50 value. Additionally, a lower bicarbonate level also contributed to a higher mortality rate in COVID-19 patients45. Generally, measurements of total CO2 using ion-selective electrodes or enzymatic methods typically yield similar results to those of venous HCO3- concentrations46. Recent investigations have revealed a correlation between elevated blood glucose levels and severe COVID-19 infection. Increased blood glucose was very common in COVID-19 patients and was an indicator of worse outcomes even for non-diabetes patients47. Intensive monitoring of blood glucose levels facilitated the pursuit of a good prognosis for patients with COVID-1948.

A notable advantage of our study was the prospective validation of the developed prediction model, which distinguishes it from other studies. Furthermore, our model utilized laboratory test results in those patients obtained at initial admission, which did not increase the burden on patients and was easy to implement. Most importantly, the previous studies primarily focused on clinical features such as age, gender, and symptoms, using traditional statistical analysis methods. Nevertheless, our research focused on molecular biomarkers that reflect disease states and incorporated more advanced techniques such as machine learning algorithms (XGBoost, logistic regression, LightGBM, random forest, AdaBoost), greatly improving the predictive accuracy of our models. Our model can serve as an excellent decision support tool for clinicians, allowing them to accurately assess a patient’s prognosis, adjust treatment plans in real-time, and prevent and manage complications in advance, thereby improving patient survival optimizing healthcare resource utilization, and ultimately improving patient outcomes.

Limitations

However, it was crucial to acknowledge that there were somethe limitations in our study. Firstly, it was a single-center retrospective study with a relatively small sample size, which might restrict the generalizability of our findings. Second, the study population was limited to Chinese patients, so it remains unknown whether the model can be effectively extended to other populations. These limitations highlight the need for further research and validation in larger multicenter studies involving diverse patient populations. To assess the extrapolation and generalization of our model, larger sample sizes and independent datasets are needed for comprehensive validation.

In further research, we will expand the participation by incorporating data from additional sources and collecting sufficient external validation datasets to enhance the model’s applicability, robustness and reliability. Furthermore, we will optimize the model based on these findings, rectify its deficiencies, and continuously improve its predictive capabilities. Our study provided promising insights into prediction model development for COVID-19-related mortality in patients with CKD, and further research efforts are required to optimize its utility for clinical decision-making.

Conclusions

To our knowledge, this is the first attempt to create a predictive model to assess the mortality of CKD patients with COVID-19. We developed and calibrated five different prediction models and evaluated their performance using various statistical metrics such as ROC curve analysis, the Brier score, and decision curve analysis. Based on these evaluations, we identified the best-performing model with robust performance and practical utility. By providing personalized predictions for COVID-19-related mortality in CKD patients, this model enabled tailored risk assessment for each individual, making it an essential tool for screening those who were at elevated risk of COVID-19 death based on the findings of our study. By implementing this predictive approach, healthcare providers could prioritize interventions, allocate resources efficiently, and deliver timely and targeted care to those who need it the most.