Prognostic value of combined clinical and myocardial perfusion imaging data using machine learning

Study population

2689 consecutive patients who were referred for clinically indicated exercise or pharmacologic stress MPI at Sacred Heart Medical Center, between January 2010 and December 2011, were included. The study was approved by the institutional review board including a waiver for informed consent. After excluding 70 patients with early revascularization within 90 days, 2,619 patients were included for further analysis.

Clinical data

Clinical data were derived from patients’ medical records and included age, gender, and risk factors. Risk factors recorded were hypertension, diabetes mellitus, dyslipidemia, and smoking (defined as current smoking or cessation within 3 months of testing), and family history of premature clinical coronary artery disease (CAD). Chest pain presence and type and shortness of breath were assessed by the stress testing physician.

MPI and stress protocols

Rest/stress 1-day ^99mTc-sestamibi imaging was performed using a high-efficiency solid-state SPECT scanner (D-SPECT, Spectrum-Dynamics, Haifa, Israel) (11). Weight-adjusted doses of 353 ± 151 MBq (9.5 ± 4.1 mCi) for rest and 1252 ± 196 MBq (34 ± 5.3 mCi) for stress (recommended by vendor), were employed (12), equivalent to a total average effective dose of 10.7 mSv based on the latest ICRP 103 estimates (13). Patients underwent symptom-limited Bruce protocol exercise testing (38%) or pharmacologic stress (62%, regadenoson 0.4mg) with injection at peak stress. Rest image acquisition was performed supine with 6–10 min acquisition time, based on patient BMI. Upright and supine stress imaging (4–6 min) began 15–30 minutes after stress.

Transaxial images were generated from list mode data maximum likelihood expectation maximization reconstruction (11). No attenuation or scatter correction was applied. Images were automatically reoriented into short-axis and vertical and horizontal long-axis slices with Quantitative Perfusion SPECT (QPS)/Quantitative Gated SPECT (QGS) software (Cedars-Sinai Medical Center, Los Angeles, CA).

Visual perfusion analysis

The visual analysis was done by multiple physicians who were aware of patient clinical information and quantitative assessment at the time of the study. Reader scan interpretation (MD-diagnosis) was scored as 0 = normal, 1 = equivocal, 2 = probably abnormal, 3 = abnormal, or 4 = definitely abnormal. A 3-step scale probability of CAD was also reported (0 = low, 1 = intermediate, 2 = high).

Automated quantification

All image datasets were de-identified, transferred to Cedars-Sinai and quality control checked by a single experienced core laboratory technologist without knowledge of clinical data. Automatically generated myocardial contours by QPS/QGS software were evaluated and when necessary, contours were adjusted to correspond to the myocardium. Upright and supine images were quantified as previously described (14). The quantitative perfusion variable employed was automatic total perfusion deficit (TPD), which reflects a combination of defect extent and severity, producing stress, rest, and ischemic (stress – rest) TPD values. Ejection fraction (EF), systolic and diastolic volumes at stress and rest were quantified separately for each acquisition using standard QGS software with 8 frames per cardiac cycle. Transient ischemic dilation (TID) was computed as previously described (15). Counts in the left ventricle were obtained by planar projections of the left ventricular region defined during the first step of data reconstruction (16).

Outcome and follow-up data collection

The endpoint was MACE, comprised of all-cause mortality, non-fatal myocardial infarction (MI), unstable angina, or late coronary revascularization (percutaneous coronary intervention or coronary artery bypass grafting). All-cause mortality was determined from the Social Security Death Index and combined with MACE events obtained by using the hospital electronic medical records including all clinics, cardiology group and hospital visits. Non-fatal MI was defined based on the criteria of hospital admission for chest pain, elevated cardiac enzyme levels and typical changes on the ECG (17). The first event in each patient was used as the outcome. Patients with early revascularization ≤90 days after MPI were excluded.

Machine learning

Figure 1 illustrates the machine learning pathway, which involved automated variable selection by information gain ratio ranking and model building with a boosted ensemble algorithm, both performed into a stratified 10-fold cross validation procedure, as reported in our previous work (8). ML techniques were implemented in the open-source Waikato Environment for Knowledge Analysis (WEKA) platform (3.8.0) (18).

Variable Selection

25 imaging data variables, 17 stress test variables, and 28 clinical variables were available for variable selection by information gain ratio (18). Information gain ratio offers a measure of the effectiveness of a variable in classifying the training data. Only variables resulting in information gain ratio > 0 were subsequently used in model building (Figure 2B).

Model building

Predictive classifiers for MACE scoring were developed by an ensemble (“boosting”) LogitBoost algorithm. The principle behind ML ensemble boosting is to combine the prediction of simple classifiers with weak performances (19) to create a single strong classifier. These weak predictions are then combined in an ensemble (weighted majority voting) to derive an overall classifier–the ‘ML score’.

Cross validation

The performance and general error estimation of the entire ML process (variable selection and LogitBoost) were assessed using stratified 10-fold cross validation (Figure 1), which is currently the preferred validation technique in machine learning (18). The main advantages of this technique, compared to the conventional split-sample approach, are: i) it reduces the variance in prediction error; ii) it maximizes the use of data for both training and validation, without overfitting or overlap between test and validation data; and iii) it guards against testing hypotheses suggested by arbitrarily split data (20).

Statistical analysis

Using receiver-operating characteristic (ROC) analysis and pairwise comparisons according to Delong et al (21) the predictive accuracy for MACE was compared between i) ML with all available data (ML-combined), ii) ML with only imaging data (ML-imaging), iii) 5-point scale visual diagnosis (MD-diagnosis), and iv) automated quantitative imaging analysis (stress TPD and ischemic TPD). Brier score and Pearson correlation were computed between predicted and observed MACE events (22). For all analyses, MACE-free patients were censored to their follow-up date. To define the low-risk limit for MACE prediction by ML-combined, we used clinical diagnosis = 0 which is considered as definitely normal scans, as a well-established low-risk limit. Then, low-risk cutoffs for ML-combined and TPD were calculated for approximately the same population percentile as for MD-diagnosis = 0 (87^th percentile). Subsequently, improvement in risk classification using ML-combined compared to MD-diagnosis was assessed with a 5-category reclassification. Statistical calculations were performed using R software version 3.3.1 and PredictABEL package for the reclassification.

Study population and outcome

Table 1 shows the baseline clinical characteristics of the studied population. When the first event per patient was considered, there were 239 (9.1%) 3-year MACE events, with 150 (5.7%) all-cause deaths, 11 (0.4%) non-fatal MIs, 24 (0.9%) unstable anginas, and 54 (2.1%) late target revascularizations. The observed annual MACE rate was 3%.

Hemodynamic and MPI results

Table 2 shows hemodynamic and stress results separately for pharmacological stress and for exercise stress. The frequency of exercise stress was lower among patients with MACE compared to those without MACE (9% in MACE vs. 41% in no MACE; P < 0.0001). Table 3 shows quantitative and visual MPI results. For the quantitative evaluation of perfusion and function, 9.8% of myocardial contours were corrected by the core lab technologist.

Variable selection

Figure 2A shows the average information gain ratio within 10-fold cross validation. On average, 22 imaging data, 8 stress test and 17 clinical variables were selected. All perfusion and functional variables from MPI had information gain ratio > 0, including left ventricular counts and injected dose. Top 9 selected variables were all imaging data variables.

MACE prediction by individual variables

Figure 2B shows the area under ROC curve (AUC) for the prediction of MACE by each individual variable. Stress TPD, stress heart rate, ischemic TPD, stress systolic blood pressure, rest TPD, and age were the best individual predictors. When compared to the information gain ratio in Figure 2A, there are some variables for which individual AUCs are predictive, yet they do not offer incremental information gain for predicting MACE (white bars). Furthermore, the variables with highest AUCs do not always have the highest information gain.

MACE prediction by combined variables

3-year MACE prediction was significantly higher for ML-combined than ML-imaging (AUC: 0.81 [0.78–0.83] vs 0.78 [0.75–0.81]; P < 0.01). ML-combined also had higher AUC compared to the AUCs of automated stress TPD and automated ischemic TPD (Figure 3), and compared to the AUCs for probability of CAD (0.64 [0.61–0.66]) or MD-diagnosis (0.65 [0.62–0.68]), as reported by the physician (all P < 0.001). When stress test variables were added to image variables for ML integration, AUC did not change significantly (AUC: 0.79 [0.76–0.82] vs. 0.78 [0.75–0.81]; P = 0.4).

The Brier score for ML-combined prediction of MACE was 0.07, indicating good calibration between ML scores (estimated predicted risk) and observed 3-year risk. The plot of observed vs. predicted MACE events over percentiles of ML-combined risk is shown in Figure 3. High correlation of ML-combined predicted vs. observed MACE was found (r = 0.97, P < 0.0001).

Risk re-categorization

To allow categorical comparison, a low-risk ML-combined score (<0.15) was determined as the cutoff defining the same percentile as visual MD-diagnosis = 0 (87th percentile). This percentile also approximately corresponded to the stress TPD threshold of < 5% (14). For patients within the 95–100th percentile of ML-combined score, 19% (25/131) of patients had normal MD-diagnosis and 10% (13/131) had stress TPD < 5% (Figure 4). Finally, a 5-category risk reclassification of 26% was significant for ML-combined scores with incremental 5% risk as compared to 5-category MD diagnosis (P < 0.001, Table 4), with 30.5% improved identification of patients with MACE and −5% decreased identification of MACE-free patients (all P < 0.001).

Implications

The ability to optimally assess risk in individual patients remains a major challenge in cardiology. With MPI, visual image analysis itself is subjective, and the overall risk assessment, incorporating clinical, stress test, and imaging results, is highly variable, based on physician knowledge and experience, and limited by the complexity of appropriately assigning weight to individual factors. The presented ML score provides an automated precise and objective risk estimate combining imaging, clinical, and stress testing variables. The same optimal method for risk computation would be readily available to all imaging centers, including less experienced centers. The practical implementation will depend on the ability to interface the MPI reporting workstation with electronic patient records, in order to access the clinical variables. Such a tool could be perhaps interfaced with large registry data (e.g., the ImageGuide registry of the American Society of Nuclear Cardiology (25)), which will collect clinical variables similar to those used in this study. The implementation will depend on the availability of the interface to the electronic health records.

Limitations

This was a single center study and further multi-center and external validation of the derived risk score will be required. Future work should include the definition of the optimal ML threshold, to validate prospective practical clinical implementation. The sample size was modest and follow-up was only 3-years; however, all results were significant. Although training data was always separated from test data within 10-fold cross validation, it is not yet known how well such ML score can extrapolate between different centers, patient populations and follow-up time. Although we included key perfusion and function imaging variables in this study, the list was not exhaustive. The derived ML score is generic and can be applied to both pharmacologic and stress protocols, since ML technique uses the information about the type of test internally. However, further evaluation of ML risk stratification for MACE prediction in specific sub-populations, for example in patients with suspected disease, patients with early revascularization, or patients undergoing adenosine protocols, may be appropriate in multicenter studies. Risk reclassification metrics have limitations such as dependence on the choice of cutoff values of the continuous probability risk score. It is likely that more appropriate threshold selection in future studies may optimize the reclassification patterns for specific clinical risks. Alternatively, the MACE risk score without any categories could be also used clinically to indicate the probability of events for a given patient. Finally, we selected a LogitBoost approach for automatic ML variables integration, as in our previous work (8) – but the LogitBoost approach used here is one of many possible ML approaches to combine multiple variables for prediction. It is possible that different approaches such as deep learning may provide more optimal risk score derivation. However, a larger multi-center dataset is required to evaluate possible advantages of other ML approaches.