Classification of breast cancer in ultrasound imaging using a generic deep learning analysis software: a pilot study

Study population

The cantonal ethics committee of Zürich, Switzerland approved this retrospective study and waived the need for an informed consent for patients from the year 2014.

All patients undergoing breast ultrasound in 2014 in our hospital were reviewed for malignant or benign lesions. The breast ultrasound examination at our institution is highly standardized: All examinations are performed on a Logiq E9 Ultrasound Station with a 9L linear probe (GE Healthcare, Chicago, IL). The depth extends beyond the lesion of interest and the focus point is set on the lesion. For large lesions, more than one focus point may be used. For this study, only the B-images were used (i.e. no colour-Doppler or elastography data). Exclusion criteria were applied by Breast Imaging Reporting and Data System (BI-RADS) scores:²⁰ As a first step, we excluded all patients with normal breast ultrasound (BI-RADS 1) as well as all patients with lesions classified as clearly benign, except for patients with prior breast-conserving surgical treatment (BI-RADS 2 excluding scars). As a second step, all patients with neither radiological follow up of at least 24 months (breast ultrasound, mammography or breast MR) nor histopathologically proven lesion were excluded. We chose the rather conservative timeframe of 24 months to ensure the absence of even low-grade malignancies in the depicted lesions at the time of examination.

Deep learning analysis

For the image analysis we used an industrial grade image analysis software (ViDi Suite v. 2.0; ViDi Systems Inc, Villaz-Saint-Pierre, Switzerland). The software uses state-of-the-art deep learning algorithms²¹ to identify and categorize anomalies in image data. It is currently used in various industries for quality inspection e.g. in defect detection of metal surfaces, real time traffic analysis or appearance-based product identification. Though it is currently not approved for the routine clinical use, it has recently shown promising results for detecting malignancies in a dual-centre mammography study.²² Deep learning or deep neural networks differ from conventional “shallow” neural networks. Deep neural networks contain three or more hidden layers not directly connected to the output neurons, which enables them to solve much more complex problems.²³ All computations were performed on a GeForce GTX 1080 graphics processor unit. All malignant lesions were labelled and contoured by two investigators in consensus (ASB and AB) for supervised training. A randomly chosen subset of the images (n = 445, 70%) was used for the training of the software, and the remaining cases (n = 192) were used to validate the resulting model in the training process. The probabilistic heatmaps generated by the software were used by the investigators to qualitatively assess the suspicious features as detected by the neural network; they were not shown during the readout.

Human readout

The validation images (n = 192) were presented in random order to two radiologists (Reader 1, SG, PGY-3 resident in diagnostic radiology and Reader 2, MMa, 3 years of experience in breast imaging) who were blinded to the clinical information as well as the study background or design. Additionally, a 4th year medical student (Reader 3, ES) was given the training images. The student had no prior clinical or research experience in breast or ultrasound imaging and did not receive specific instructions. The images were placed in two separate folders, one for benign and one for malignant lesions, and available to study once for a freely chosen amount of time (n = 445). It was expected that the student, like the software, learns solely from the data. Subsequently, the student rated the validation cases in the same manner as the radiologists. All images were rated on a 5-point Likert-type scale for malignancy (roughly corresponding to the BI-RADS classification with 5 meaning >98% probability of breast cancer). The time needed for the complete readout was noted, and for the medical student the training was timed as well.

Statistical analysis

The statistical analysis was performed in R v.ion 3.3.1 (R Foundation for Statistical Computing, Vienna, Austria). Continuous variables were expressed as median and interquartile range, categorical variables as counts or percentages. Due to obvious differences in readout times between computer and humans, statistical testing was omitted. Interreader agreement was assessed pair-wise with Lin’s concordance correlation coefficient. To analyse the diagnostic performance, receiver operating characteristic analysis was performed for the computer test and validation data and the human readers. Diagnostic accuracy was expressed as the area under the receiver operating characteristic curve (AUC) and compared with DeLong’s non-parametric test. The optimal cut-off (Youden Index) was determined and the resulting specificity, sensitivity, positive predictive value and negative predictive value were calculated. Sensitivities and specificites were compared using the McNemar test. A p value < 0.05 was considered indicative of significant differences. All tests were two-tailed.

Study cohort

A total of (n) 3432 examinations were reviewed. Exclusion criteria were applied as defined and shown in Figure 1. From the remaining 657 patients, the ultrasound images saved in the examination were searched for the most representative image of the described BI-RADS 3–6 lesion or scar (BI-RADS 2), respectively. During this step, another 25 patients had to be excluded due to lack of a suitable image. Two images of two different lesions were used in five patients: in two cases because of a bilateral malignancy and in three cases because of two independent benign lesions, which met the criteria described. The final population was comprised of 632 patients with one image for each of the 637 lesions. The eligible study cohort contains 82 patients with a malignancy or borderline lesion (84 lesions) and 550 patients without malignancies (553 benign lesions or scars).

The mean age of this cohort (n = 632) was 53 ± 15 years (range 15–91 years). Of the 82 patients with malignancy, the most common histopathological diagnosis was invasive ductal carcinoma in 52 cases. Two of these patients had bilateral disease. Further histological subtypes are reported in Table 1. Of the 550 patients with benign lesions, three were included twice because of two different, biopsy-proven lesions, resulting in 553 images of benign lesions or scars included. 176 lesions in 173 patients were histopathologically proven, whereas for the other 377 no histopathological diagnosis was available, but only an inconspicuous (unremarkable) follow-up of at least 24 months. Table 2 shows the diagnoses of all benign lesions. In cases with no histopathological diagnosis available, the most probable diagnosis is listed, in general the first differential diagnosis of the initially examining radiologist. Of all included patients, 295 (46.7%) had undergone prior treatment of the breast, such as surgery, radiation therapy, or a combination therapy. The percentage of patients with prior treatment was smaller in the malignancy group (n = 17, 21.0%) than in the one with benign lesions (n = 278, 50.5%). The reason for this difference is the inclusion of post-surgical scars, which turned out to be benign in almost all the cases (n = 209), while relapses of a prior breast carcinoma occurred in only 7 of the included cases (exclusively in patients with invasive ductal carcinomas). Benign lesions were slightly but significantly smaller than malignant lesions [12 mm (8–17 mm) vs 14 mm (10–22 mm); p < 0.001], which held true for all lesions with benign imaging characteristics and only follow-up as reference standard [8 mm (6–12 mm), p < 0.001] excluding scars [16 mm (12–19 mm), p = 0.66].

Timing and interreader agreement

Training times for training set (445 images) were 7 min for the neural network [0.94 s image^–1 (sec im^–1)] and 48 min for the medical student (6.5 s im^–1). Readout times for the validation set (192 images) were 28 min (Reader 1, 8.8 s im^–1), 22 min (Reader 2, 6.9 s im^–1) and 25 min (Reader 3, 7.8 s im^–1). Processing time of the neural network for the test set was 3.7 s (0.0193 ± 0.0011 s im^–1). Interreader agreement between the human readers was best between the two more experienced readers 1 and 2 (0.56; [95% CI: 0.45–0.67]) and worst between the two less experienced readers 2 and 3 [0.35; (0.22–0.46)]. Interreader agreement between the neural network and the human readers was best between the software and the reader with intermediate experience [Reader 2, CCC = 0.49; (0.38–0.59)]. The full pair-wise comparison is given in Table 3.

Diagnostic performance

The neural network’s accuracy on the training set (n = 445) was AUC = 0.96 [95% CI (0.92–0.99)]. The performance on the validation set (n = 192) was AUC = 0.84 (0.75–0.93). Specificity and sensitivity were 80.4 and 84.2%, respectively. Diagnostic accuracy measured by AUC was not significantly different between the human readers (Reader 1: AUC = 0.89, Reader 2: AUC = 0.89 and Reader 3: AUC = 0.79) and the neural network (p = 0.45–0.47), as depicted in the Receiver operating characteristic curve in Figure 2. As visualized in Table 4, between the human readers, there was a significant trend of better performance with increased experience, especially for the specificity (89.0, 82.7 and 72.8%), but also for the AUC (0.89, 0.89 and 0.79) and for the sensitivity (84.2, 84.2 and 73.7%).

Features on the heatmaps

The neural network rated post-operative changes more often as malignant than the human readers (Figure 3), especially in cases with large areas of acoustic shadowing. The differentiation of other features such as size or texture might have led to the few false negatives as illustrated in Figure 4, which shows a quite well defined, small carcinoma with partly well defined granular internal texture. On the other hand, the neural network was excellent and in several cases superior to the readers in correctly classifying small benign lesions (Figure 5) and voluminous malignant lesions (Figure 6). Interestingly, the neural network classified the only lymphoma and the only male patient (Figure 6) correctly in contrast to the two more experienced readers. Notable are also the cases where the neural network classified benign lesions of patients with no prior surgery correctly, even if the images might have had some aspects of post-operative changes (Figures 7 and 8).

Advances in knowledge

• DLS for industrial quality control can detect anomalies in breast ultrasound with high diagnostic accuracy (AUC 0.84), comparable to radiologists (AUC 0.88).

• The software learns faster and better than a medical student with no prior experience in breast imaging (AUC 0.79) and its reading is most similar to a radiology resident (CCC = 0.49).

• The speed of the software in the order of milliseconds per image would allow real-time analysis during an ultrasound examination.