Secondary Logo

Journal Logo


A new risk stratification score for patients with suspected cardiac chest pain in emergency departments, based on machine learning

Mao, Hai-Feng1; Chen, Xiao-Hui1; Li, Yun-Mei1; Zhang, Si-Yuan2; Mo, Jun-Rong1; Li, Min1; Lin, Pei-Yi1; Rainer, Timothy H.3; Graham, Colin A.3; Jiang, Hui-Lin1

Editor(s): Ji, Yuan-Yuan

Author Information
doi: 10.1097/CM9.0000000000000725

To the Editor: Chest pain is one of the most common complaints for patients attending emergency departments (EDs) globally. It is important to accurately stratify risk of possible acute coronary syndrome (ACS) for these patients.[1] Several risk stratification scores such as thrombolysis in myocardial infarction (TIMI), global registry for acute coronary events (GRACE), Banach and HEART are helpful.[2] Previous research in our setting compared these four scores and found that the HEART score, with a C-statistic of 0.731, was the best for predicting 7-day major adverse cardiac events (MACE).[3]

The purpose of this study was to develop risk stratification prediction models for 7-day MACE in patients with chest pain, utilizing machine learning algorithms such as eXtreme Gradient Boosting (XGBoost), Support Vector Machine (SVM), and Logistic regression (LR).

This is a retrospective observational cohort study based on data from a prospective observational study. Patients with suspected cardiac chest pain attending EDs at the Prince of Wales Hospital (PWH) and Second Affiliated Hospital of Guangzhou Medical University (AHGZMU) were recruited consecutively and followed up at the PWH between May 2012 and March 2013 and the AHGZMU between March 2012 and August 2013. Based on the order of time attending the ED, data from 583 (70.0%) patients recruited from the two hospitals were used as training material to develop the classification models. Data from 250 (30.0%) patients were used to evaluate the prognostic performance of the classification models.

Patients ≥18 years old presenting to the ED with a chief complaint of chest pain or discomfort of possible cardiac origin were included in the study. Patients were excluded if they were non-Chinese or there was a clear non-cardiac cause of chest pain. Those with confirmed ST-segments elevation myocardial infarction (STEMI) were also excluded as they did not have undifferentiated chest pain.

Data regarding subsequent visits to ED, hospital readmission for evaluation of chest pain and all cardiac procedures carried out were obtained from Clinical Management System (CMS) in PWH and Health Insurance Information Management System (HIIMS) in AHGZMU and confirmed by phone interviews at 7-day follow-up after the initial presentation.

Categorical variables were compared using a Chi-square analysis, while continuous variables were compared with independent t tests. Prognostic performances of the three machine learning models and HEART scores were compared with receiver operating characteristic curve (ROC) analysis. All analyses were carried out using SPSS v 20.0 (SPSS Inc, IL, USA), MedCalc v18.11.3 (MedCalc Software, Mariakerke, Belgium). For the development of machine learning models (XGBoost, SVM and LR), all data were stored in csv format and read by pandas library of Python.

Patients were recruited between 17 March 2012 and 14 August 2013. There were 1274 eligible patients, of whom 418 were excluded due to unwillingness to join the study (n = 111), missing the onset time (n = 16), inability to give consent (n = 79) or being a non-cardiac chest pain patient (n = 212). This left 856 cases for inclusion in the study. Of these, 833 completed 7-day follow-up (Supplementary Figure 1,

Figure 1 shows prognostic performances of XGBoost, SVM, LR, and HEART scores, based on test data. The area under curve (AUC) of XGBoost was significantly larger (0.822, 95% confidence interval [CI]: 0.769 to 0.868) than the other models: SVM (0.649, 95% CI: 0.586 to 0.708), LR (0.667, 95% CI: 0.605 to 0.725), HEART score (0.702, 95% CI: 0.641 to 0.758); P = 0.002, 0.001, 0.098, respectively.

Figure 1
Figure 1:
Receiver operating characteristic (ROC) curves of XGBoost, SVM, LR and HEART for predicting 7-day MACE of patients with chest pain, based on test data. LR: Logistic regression; SVM: Support Vector Machine; XGBoost: eXtreme Gradient Boosting.

In this study, utilizing machine learning algorithms, we found that XGBoost produced the most discriminating model for predicting 7-day MACE in people presenting to the ED with chest pain. The three most important features generated and ranked by XGboost algorithm were troponin, gender, and creatinine.

XGBoost is one type of decision tree algorithm which has been used in medical research. In this study, XGBoost demonstrated better predictive values than other models, including SVM, LR, and HEART score. This may be because XGBoost can summarize rules from medical data automatically and efficiently, meaning it can conduct a more comprehensive analysis which includes all variables from raw data. The lower 95% CI of XGBoost did not overlap with the mean AUC of HEART, suggesting that AUC of XGBoost was significantly higher than HEART.

In our study, all key variables, including symptoms, signs and blood biomarkers of patients with chest pain, could be obtained in 2 hours, meaning machine learning models such as XGBoost could be conducted in ED. This could be beneficial for emergency staff when predicting clinical outcomes and making decisions about triage classification.[4]

The strengths of this study are that a new risk stratification model for chest pain has been developed by XGBoost algorithm and that its prognostic performance was better than for other models, including SVM, LR and HEART. In conclusion, the machine learning model XGBoost may be a better prognostic tool for predicting 7-day MACE following chest pain than SVM, LR, and HEART score.

Declaration of patient consent

Ethical approval was obtained from the joint Chinese University of Hong Kong - New Territories East Cluster Clinical Research Ethics Committee in Hong Kong as well as the Clinical Research Ethics Committee of the Second Affiliated Hospital of Guangzhou Medical University. Written informed consent was obtained from each patient or their relative. Patients were informed that they could withdraw from the study at any time. All patients and their guardians gave consent for the images and other clinical information to be reported in a journal. Patients and their guardians understood that the names and initials would not be published and that due efforts would be made to conceal identity, but that anonymity cannot be guaranteed.


This study was supported by grants from the Scientific Research Project of the Guangzhou Education Bureau (No. 1201610645) and the Key Medical Disciplines and Specialties Program of Guangzhou.

Conflicts of interest



1. Stopyra JP, Miller CD, Hiestand BC, Lefebvre CW, Nicks BA, Cline DM, et al. Chest pain risk stratification: a comparison of the 2-hour accelerated diagnostic protocol (ADAPT) and the HEART pathway. Crit Pathw Cardiol 2016; 15:46–49. doi: 10.1097/HPC.0000000000000072.
2. Chang AM, Fischman DL, Hollander JE. Evaluation of chest pain and acute coronary syndromes. Cardiol Clin 2018; 36:1–12. doi: 10.1016/j.ccl.2017.08.001.
3. Chen X-H, Jiang H-L, Li Y-M, Chan CPY, Mo J-R, Tian C-W, et al. Prognostic values of 4 risk scores in Chinese patients with chest pain: prospective 2-centre cohort study. Medicine 2016; 95:e4778doi: 10.1097/MD.0000000000004778.
4. Goto T, Camargo CA, Faridi MK, Freishtat RJ, Hasegawa K. Machine learning-based prediction of clinical outcomes for children during emergency department triage. JAMA Netw Open 2019; 2:e186937doi: 10.1001/jamanetworkopen.2018.6937.

Supplemental Digital Content

Copyright © 2019 The Chinese Medical Association, produced by Wolters Kluwer, Inc. under the CC-BY-NC-ND license.