Figures
Abstract
To retrospectively determine the association between breast lesion morphology and malignancy and to determine the optimal value of lesion enhancement (HU, Hounsfield units) to improve the diagnostic accuracy of breast cancer in patients with incidental breast lesions (IBLs). A total of 97 patients with 102 IBLs detected from July 2009 to December 2012 were enrolled in this study. Two radiologists analyzed CT images for the presence of malignancy based on the morphology of the lesions alone and in combination with an enhancement value (HU) analysis. There were 36 malignant and 66 benign IBLs. When the morphology and enhancement values were combined, the sensitivity, specificity, and accuracy were 92%, 97%, and 95%, respectively, for reader 1 and 89%, 94%, and 92%, respectively, for reader 2. The addition of HU values led to correct changes in the diagnosis; specifically, the accuracy of the diagnosis of reader 1 and reader 2 improved by 6.9% and 11.8%, respectively. The addition of the enhancement value (HU) to the CT morphology improved the diagnostic accuracy in the differentiation of malignant from benign IBLs by using the region of interest (ROI) to measure the HU within the most suspicious part of the lesion.
Citation: Lin Y-P, Hsu H-H, Ko K-H, Chu C-M, Chou Y-C, Chang W-C, et al. (2016) Differentiation of Malignant and Benign Incidental Breast Lesions Detected by Chest Multidetector-Row Computed Tomography: Added Value of Quantitative Enhancement Analysis. PLoS ONE 11(4): e0154569. https://doi.org/10.1371/journal.pone.0154569
Editor: Olga Y. Gorlova, Geisel School of Medicine at Dartmouth College, UNITED STATES
Received: August 3, 2015; Accepted: April 15, 2016; Published: April 29, 2016
Copyright: © 2016 Lin et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: HHH received the funding partly from Tri-Service General Hospital (TSGH-C105-044 and TSGHIRB No.1-103-05-036) and Ministry of Science and Technology, R. O. C. (MOST 104-2314-B-016-025). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Multidetector-row computed tomography (MDCT) is increasingly being used in the diagnosis and surveillance of chest disease. Occasionally, incidental breast lesions (IBLs) are encountered during a MDCT examination [1–7]. Computed tomography (CT) has been shown to have high diagnostic efficacy in the evaluation of breast tumours. In CT scans of known breast tumours, the lesion morphology and enhancement pattern can be used to differentiate benign from malignant lesions [8–13]. Moreover, investigators in previous studies explored the mean CT values (HU, Hounsfield units) of the lesions in patients with suspected breast cancer [14]. For example, Lindfors and Prionas et al. aimed to quantify the enhancement value of lesions identified by dedicated breast CT [15, 16]. Although the importance of morphology and the CT enhancement pattern in differentiating breast tumours has been reported in the previous literature, there has been no study focusing on the relationship between the enhancement values (HU) and malignancy of IBLs.
The aims of the present study were two-fold: (1) to determine the association between lesion morphology and breast cancer and (2) to determine the optimal value of lesion enhancement (HU) to improve the diagnostic accuracy of breast cancer in patients with IBLs.
Materials and Methods
Patients and lesions
This retrospective study was approved by the institutional review board of Tri-Service General Hospital. Informed consent for the diagnostic procedure was waived. A total of 13,651 patients, 123 of whom were identified to have IBLs, underwent contrast-enhanced chest CT scans at our hospital from July 2009 to December 2012. Of those 123 patients, 26 were excluded due to a history of breast cancer and/or insufficient diagnostic information. Finally, a total of 97 female patients were included in this retrospective study (mean age: 55.3 years, range: 27–91 years).
Of the 97 patients, the indications for chest CT included staging or a follow-up study for other malignancies (n = 21; non-small cell lung cancer, thyroid cancer, cervical cancer, gastric cancer, oesophageal cancer, renal cell carcinoma), benign neoplasms (n = 4; goitre, thymoma), abnormal chest radiographs (n = 35), CT pulmonary angiography (n = 14) and other pathologies (n = 23) (Table 1). Ninety-three patients had one breast lesion, three patients had two lesions, and one patient had three lesions. Therefore, a total of 102 IBLs were analyzed in our study.
NSCLC non-small cell lung cancer, COPD chronic obstructive pulmonary disease.
CT technique
The patients underwent a 64-detector row CT (Brilliance, Philips Medical Systems, Cleveland, Ohio) scan of the chest in the supine position. The standard protocols in our hospital include unenhanced MDCT from the lung apices through the adrenal glands with the following parameters: slice thickness of 2.5 mm, reconstruction interval of 2.5 mm, rotation speed of 0.75 sec, pitch of 1.05–1.25, and 120 kVp; the effective tube current × time product ranged from 150–200 mAs. Subsequently, contrast-enhanced MDCT was performed with intravenous administration of 100 mL Iopromide (Ultravist-300, Schering, Berlin, Germany) or Iohexol (Omnipaque 300, General Electric Health Care, Princeton, NJ) using a mechanical power injector at a rate of 2.0–3.0 mL/sec. The scanning was executed 50 sec after the injection of the nonionic contrast medium with technical parameters identical to those used for unenhanced CT. For patients suspected to have an aortic dissection or pulmonary embolism, the contrast medium was administered at a rate of 3.0–4.0 mL/sec. The scanning was triggered after a bolus of contrast medium reached the descending aorta or pulmonary artery. The images were obtained by employing a standard soft-tissue algorithm (window width, 350 HU; level, 40 HU) and a retrospective lung algorithm (window width, 1000 HU; level, -700 HU).
CT imaging interpretation
To assess the benefits of adding the CT enhancement value (HU) in the differentiation of malignancy from benign lesions, the diagnostic performance of morphology analysis alone was compared with that of morphology combined with CT enhancement analysis (both axial and coronal images). An enhancing breast lesion meant that the lesion had higher attenuation compared with the normal breast glandular tissue on contrast-enhanced CT. For the evaluation of the CT morphology alone, the CT images were retrospectively reviewed by two radiologists (H.H.H. and Y.P.L., with 25 and 3 years of experience, respectively). The radiologists were blinded to the imaging reports, clinical history, and final pathology of the lesion, and they independently interpreted the findings of the 102 IBLs. In cases of discordance, the rating was discussed until an agreement was reached. The results of the independent rating and consensus rating were all documented for the analysis. Due to the lack of a formal lexicon for CT breast imaging, a consensus analysis of morphology was achieved by using the Breast Imaging and Reporting Data System (Bi-RADS) lexicons of MR imaging-detected breast lesions as a reference [17].
The CT characteristics of each lesion were analyzed as follows: size or extent (greatest axial diameter), lesion type (mass or non-mass), shape (oval, round, lobulated, irregular), margin (circumscribed, indistinct, spiculated), the presence of calcifications or abnormal axillary nodes, and lesion conspicuity (visual enhancement). The non-mass lesions, also known as non-mass enhancement (focal, linear, ductal, segmental, and regional), indicated that the lesion appeared as an area of enhancement without an associated mass (Fig 1), as defined in the BI-RADS MR lexicon [17]. The diameter of a mass, as a measure of its size, was assessed on axial images with electronic callipers. The size of a non-mass lesion was delineated as the extent of enhancement. The axillary nodes were considered to be abnormal if they were larger than 1 cm in the maximum transverse dimension, and if there was presence of a lobulated cortical margin or absence of a fatty hilum. With regard to the visual enhancement of the breast lesions, contrast-enhanced CT images were evaluated first. If the lesion could not be identified, the unenhanced CT images were examined, and with the added information from that evaluation, the contrast-enhanced CT images were interpreted. The visual enhancement was evaluated by the two radiologists as a qualitative assessment, where ‘good’ indicated a more conspicuous lesion on enhanced CT images compared with unenhanced CT images; otherwise, the visual enhancement was described as ‘poor’.
Coronal nonenhanced (a) and contrast-enhanced (b) CT scans showed an ovoid, well-defined fibroadenoma (arrow, 41 HU in nonenhanced CT and 47 HU in contrast-enhanced CT,ΔHU = 3) in right breast and multifocal nonmass enhancement (arrowheads, 39 HU in nonenhanced CT and 119 HU in contrast-enhanced CT, ΔHU = 77) in left breast. Histopathlogy revealed diagnosis of invasive ductal carcinoma.
To evaluate the combination of the CT imaging morphology with a quantitative enhancement analysis, the same radiologists who executed the first evaluation again decided whether each lesion was benign or malignant. To prevent recall bias, the evaluation of the combined CT imaging morphology and lesion enhancement (HU) was carried out six weeks after the evaluation of the CT morphology alone and in a different order from that of the first evaluation. Both readers were blinded to the initial imaging reports, clinical history, and final lesion pathology, and they independently analyzed the quantitative value of the enhancement (HU) of the IBLs. The enhancement values (HU) were assessed by placing a manually drawn region of interest (ROI) in the most suspicious part of the lesion on contrast-enhanced images. Sites of calcification were carefully avoided because they had relatively high HU values. To correctly draw the ROIs in the lesion, we set the contrast-enhanced image and a corresponding non-contrast image side-by-side to ensure that the images were at the same level. All ROIs were drawn as large as possible within the lesion. We used the adipose tissue density to normalize the lesion density (HU) and account for variations between image acquisitions. The lesion enhancement (ΔHU) was calculated as the difference between the normalized lesion density in the pre- and post-contrast images. The data were evaluated using the following equation: ΔHU = (HU post L-HU post A)- (HU pre L- HU pre A) [16], where L and A represent the lesion attenuation and adipose tissue attenuation, respectively, measured in the pre-contrast (pre) and post-contrast (post) image sets.
Statistical analysis
We entered the data into a computerized spreadsheet (Excel, Microsoft, Redmond, WA, USA). All analyses were performed using a commercially available software program (SPSS, version 19.0, SPSS Inc., Chicago, Illinois). The categorical variables related to malignancy were calculated by theχ2 test, and the odds ratios (ORs) with 95% confidence intervals (CIs) were determined. Moreover, the predictive abilities of the variables were compared using an area under the curve (AUC) analysis [18]. Statistically significant variables associated with malignancy identified in the univariate analysis (P < 0.05) were chosen for inclusion in a multivariate analysis using multiple logistic regression models to distinguish variables of independent statistical significance. The HU cut-off values in malignant breast lesions were determined using receiver operating characteristic (ROC) curve analysis. We used the kappa values to assess the interobserver agreement between the two readers regarding the decision of whether the lesion was benign or malignant. A kappa value of 0.00–0.20 indicated slight agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, substantial agreement; and 0.81–1.00, almost perfect agreement [19]. Sensitivity and specificity were compared by McNemar statistics.
Results
Histological findings
A summary of the diagnoses of the IBLs is shown in Table 2. Thirty-six of these lesions were subsequently proven to be malignant, and 66 lesions were considered to be benign. For the 36 malignant lesions, all were proven to be malignant by biopsy and pathological diagnosis and included invasive ductal carcinoma (n = 23), invasive lobular carcinoma (n = 7), and ductal carcinoma in situ (n = 6). The 66 benign lesions were further followed by sonography or CT and were confirmed to have a satisfactorily benign appearance. The follow-up period was 1.5 years (n = 5), three years (n = 32), or five years (n = 29).
Associations between malignancy and imaging characteristics
The patient and lesion variables associated with malignancy are listed in Table 3. The lesion size, type, shape, margins, and presence of abnormal axillary lymph nodes predicted malignancy (P < 0.05 for each). Based on the area under the ROC curve (AUC), greater discrimination was provided by the shape (AUC = 0.84) and margin (AUC = 0.90). The age of the patient and presence of calcification were not significantly associated with malignancy.
Note.—Unless otherwise indicated, data are numbers of lesions, with percentages in parentheses.
The usefulness of CT in differentiating benign and malignant lesions
The cut-off CT values for differentiating benign and malignant IBLs and the AUCs are shown in Table 4. The differential enhancement between benign (14.1 HU ± 1.9, standard error) and malignant (46.8 HU ± 3.3) lesions was significant (P < 0.001) (Fig 1). The following results are clinically applicable and showed high diagnostic accuracy. The values useful for differentiating malignant from benign lesions are a value of more than 57 HU on contrast-enhanced CT images (sensitivity, 100%; specificity, 83%; accuracy, 89%) and a difference between the normalized lesion density on pre- and post-contrast CT images (ΔHU) of more than 33 HU (sensitivity, 83%; specificity, 95%; accuracy, 93%) (Fig 2); the ROC curve analyses yielded AUCs of 0.92 (95% CI, 0.87–0.98) and 0.93 (95% CI, 0.87–0.98), respectively, for these values. Based on the AUC values, the greatest discrimination was provided by the ΔHU.
Axial nonenhanced (a) and contrast-enhanced (b) CT scans showed an irregular enhancing lesion (arrow, 35 HU in nonenhanced CT and 79 HU in contrast-enhanced CT, ΔHU = 40) with indistinct margin in left breast. Histopathlogy revealed diagnosis of invasive ductal carcinoma. Fluid cytology of the pleural effusion was proven to be malignant.
Note.—Numbers in parentheses are raw data; numbers in brackets are 95% confidence intervals. Percentages were rounded. CIs confidence intervals, AUC area under the receiver operating characteristic curve, HU Hounsfield units.
Based on the multivariate analysis results shown in Table 5, the relationships of the margin and ΔHU between malignant and benign lesions were significant (P = 0.008 and P < 0.001). The lesion size and shape did not differ significantly according to the multiple logistic regression analysis.
Note.—Numbers in parentheses are number of malignant lesions out of total number of lesions.
The sensitivity, specificity, and accuracy for the discrimination of malignant breast lesions by the CT morphology alone and by the combination of morphology and the difference between the normalized lesion density in pre- and post-contrast CT images (ΔHU) are listed in Table 6. When the CT morphology was used alone, the sensitivity, specificity, and accuracy of reader 2 (72%, 85%, 80%) were lower than those of reader 1 (81%, 92%, 88%). With the additional information provided by the quantitative ΔHU, the accuracy was improved for both reader 1 (added value = 6.9%, seven of 102 lesions) and reader 2 (added value = 11.8%, twelve of 102 lesions). In our study, there were specific improvements in the diagnostic accuracy of the readers, including the reader with less experience, following the addition of CT values. The sensitivity and specificity of reader 2 was significantly different (P = 0.014 and P = 0.031). The kappa values showed substantial to almost perfect agreement (k = 0.696 when diagnosed by morphology alone; k = 0.935 when diagnosed using morphology in combination with the ΔHU) in whether IBLs were malignant or benign based on CT imaging.
Note.—Numbers in parentheses are raw data. Percentages were rounded.
Discussion
Our study showed that contrast-enhanced breast CT offers a promising quantitative technique with which to predict malignancy in IBLs identified with routine chest CT. The quantitative findings suggested that there is improved specificity for contrast-enhanced chest CT in the detection of malignant IBLs. The diagnostic accuracy was improved by adding a quantitative enhancement analysis to the assessment of qualitative morphology features, including improved accuracy for a less-experienced radiologist.
Our results also support the findings of other investigators indicating that lesion morphology (qualitative features) is valuable from a diagnostic standpoint. In our study, the morphology analysis of the IBLs revealed that the features with the highest positive predictive value (PPV) for malignancy are non-circumscribed margins (PPV, 74%) and an irregular or lobulated shape (PPV, 50%). These findings are consistent with the previous literature [5, 9]. In addition, we found low PPV values for malignancy with features such as an oval or round shape (PPV, 6%) and circumscribed margins (PPV, 8%) (Table 3). These findings are also consistent with the results reported by Moyle et al. [5] and Inoue et al. [9]. The chi-square test revealed a significant relationship between the margins or shape of the lesion and malignancy (P < 0.001). In general, our data are concordant with the previous literature in concluding that non-circumscribed margins and an irregular or lobulated shape are more likely to be associated with malignancy.
In the present study, malignant lesions were enhanced approximately 38.9 HU more than benign lesions [16]. In our study, the mean difference was 32.7 HU. Although the HU value is affected by various factors, such as the machine used, type of contrast medium, presence of calcification, and imaging acquisition time, these cut-off values can be of practical value for guiding judgement at an institution [14, 16]. Based on the AUC, excellent discrimination was achieved when ΔHU≥33 (AUC = 0.922) (Table 4). Additionally, the IBLs that had an HU value >57 on contrast-enhanced CT had a higher rate of malignancy (AUC = 0.917). We noticed that a few breast lesions had macro-calcifications, which may have been partially included in the ROI because the lesion was too small, which could have led to a higher HU value. Such errors can be corrected by calculating the ΔHU to reduce the partial volume effect. However, some malignant lesions have micro-calcifications (Fig 3), which are difficult to exclude when drawing the ROI. This may be the reason why the HU values on non-contrast-enhanced CT images higher than 32 HU had some power (AUC = 0.717) to detect malignancy.
Axial nonenhanced (a) and contrast-enhanced (b) CT scans showed tiny calcifications (arrow) in left breast. The area adjacent to the calcifications showed enhancement (arrowheads, 82 HU in nonenhanced CT and 116 HU in contrast-enhanced CT, ΔHU = 34). Histopathlogy revealed diagnosis of ductal carcinoma in situ.
In our study, the CT protocols were not optimized for breast pathology; instead, the study included routine chest CT and protocols to detect thoracic aneurysms or pulmonary emboli, which had scan times with an approximately 50-sec delay and a less than 30-sec delay, respectively. A study by Miyake et al. [14] used a protocol including a scan at 30 sec (early phase) and two minutes (delayed phase) after contrast injection; the authors noted that the best cut-off point for differentiating benign from malignant lesions was 60 HU in the early phase, which is similar to our result of 57 HU. Miyake et al. [14] also classified enhancement patterns into five types. Types 3, 4, and 5, which were all associated with rapid initial growth, had high PPVs for malignancy, with sustained, stable, and decreasing HU, respectively. These high PPVs may explain why—even though our patients did not undergo a dedicated breast CT protocol and the dynamic pattern was not available—the cut-off HU value was still clinically applicable; the most striking difference between malignant and benign lesions is the HU value in the early phase, not the delayed phase. The scanning time of our protocols resembled that of the early arterial phase.
Nineteen of the IBLs showed non-mass enhancement. Seventeen of them were proven to be malignant, and two were found to be fibrocystic changes. Baltzer [20] found that non-mass lesions were the major cause of false-positive magnetic resonance imaging (MRI) findings. In our study, the 17 malignant non-mass lesions all had values of more than 57 HU on contrast-enhanced CT and a ΔHU >33. In contrast, the two benign non-mass lesions both had enhancement lower than the cut-off values. Recognizing the enhancement pattern may be a reliable method for differentiating malignant and benign non-mass lesions. By paying attention to the breast in chest MDCT, non-mass lesions can be detected and treated at an earlier stage.
Our study had limitations. First, the assessment of morphology was subjective and had potential bias, as the readers had different experiences in reading breast images. Second, this was a retrospective study. There was still a possibility of selection bias, even though we recruited all patients who fulfilled the inclusion criterion. Third, the number of lesions (n = 102) included in our study was small. Randomized multicentre trials with larger sample sizes are needed to establish the relationship between CT enhancement values and malignancy. Fourth, pathological results were not available in all cases. Nevertheless, we trusted that imaging follow-up criteria were adequate to assume the benign nature of the breast lesions.
In conclusion, the addition of a quantitative CT enhancement value to the qualitative morphology of IBLs improves the diagnostic accuracy of the differentiation between benign and malignant lesions, even by less-experienced radiologists. We have shown that malignant lesions show significantly higher attenuation on contrast-enhanced CT in the early phase. Without a dedicated dynamic breast CT, we were still able to identify enhancing breast lesions and to use the cut-off value in combination with morphology to detect breast cancer in the early stage.
Supporting Information
S1 Data. Clinical data and CT measurements in this study.
https://doi.org/10.1371/journal.pone.0154569.s001
(XLSX)
Acknowledgments
The authors thank department of radiology of Tri-Service General Hospital for the support of the clinical data. HHH received the funding partly from Tri-Service General Hospital (TSGH-C105-044 and TSGHIRB No.1-103-05-036) and Ministry of Science and Technology, R.O.C. (MOST 104-2314-B-016-025). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Author Contributions
Conceived and designed the experiments: HHH YPL. Performed the experiments: HHH YPL THC. Analyzed the data: CMC YCC. Contributed reagents/materials/analysis tools: KHK WCC. Wrote the paper: YPL HHH.
References
- 1. Harish MG, Konda SD, Macmahon H, Newstead GM. Breast lesions incidentally detected with CT: what the general radiologist needs to know. RadioGraphics. 2007;27:S37–S51. pmid:18180233
- 2. Shojaku H, Seto H, Iwai H, Kitazawa S, Fukushima W, Saito K. Detection of incidental breast tumors by noncontrast spiral computed tomography of the chest. Radiation medicine. 2008;26(6):362–7. pmid:18677611.
- 3. Porter G, Steel J, Paisley K, Watkins R, Holgate C. Incidental breast masses detected by computed tomography: are any imaging features predictive of malignancy? Clinical radiology. 2009;64(5):529–33. pmid:19348850.
- 4. Hussain A, Gordon-Dixon A, Almusawy H, Sinha P, Desai A. The incidence and outcome of incidental breast lesions detected by computed tomography. Annals of the Royal College of Surgeons of England. 2010;92(2):124–6. pmid:19995489; PubMed Central PMCID: PMC3025223.
- 5. Moyle P, Sonoda L, Britton P, Sinnatamby R. Incidental breast lesions detected on CT: what is their significance? The British journal of radiology. 2010;83(987):233–40. pmid:19546179; PubMed Central PMCID: PMC3473543.
- 6. Yi JG, Kim SJ, Marom EM, Park JH, Jung SI, Lee MW. Chest CT of incidental breast lesions. J Thorac Imaging. 2008;23:148–55. pmid:18520577
- 7. Lin WC, Hsu HH, Li CS, Yu JC, Hsu GC, Yu CP, et al. Incidentally detected enhancing breast lesions on chest computed tomography. Korean journal of radiology: official journal of the Korean Radiological Society. 2011;12(1):44–51. pmid:21228939; PubMed Central PMCID: PMC3017883.
- 8. Chang CH, Sibala JL, Fritz SL, Dwyer SJ III, Templeton AW, Lin F, et al. Computed tomography in detection and diagnosis of breast cancer. Cancer. 1980;46:939–46. pmid:7397672
- 9. Inoue M, Sano T, Watai R, Ashikaga R, Ueda K, Watatani M, et al. Dynamic multidetector CT of breast tumors: diagnostic features and comparison with conventional techniques. AJR. 2003;181:679–86. pmid:12933459
- 10. Nishino M, Hayakawa K, Yamamoto A, Nakajima Y, Morimoto T, Mukaihara S, et al. Multiple enhancing lesions detected on dynamic helical computed tomography–mammography. J Comput Assist Tomogr. 2003;75:771–8.
- 11. Boone JM, Kwan AL, Yang K, Burkett GW, Lindfors KK, Nelson TR. Computed tomography for imaging the breast. Journal of mammary gland biology and neoplasia. 2006;11(2):103–11. pmid:17053979.
- 12. Perrone A, Lo Mele L, Sassi S, Marini M, Testaverde L, Izzo L, et al. MDCT of the breast. AJR American journal of roentgenology. 2008;190(6):1644–51. pmid:18492919.
- 13. Uematsu T, Yuen S, Kasami M, Uchida Y. Comparison of magnetic resonance imaging, multidetector row computed tomography, ultrasonography, and mammography for tumor extension of breast cancer. Breast cancer research and treatment. 2008;112(3):461–74. pmid:18193352.
- 14. Miyake K, Hayakawa K, Nishino M, Nakamura Y, Morimoto T, Urata Y, et al. Benign or malignant? differentiating breast lesions with computed tomography attenuation values on dynamic computed tomography mammography. J Comput Assist Tomogr. 2005;29:772–9. pmid:16272850
- 15. Lindfors KK, Boone JM, Nelson TR, Yang K, Kwan AL, Miller DF. Dedicated breast CT: initial clinical experience. Radiology. 2008;246:725–33. pmid:18195383
- 16. Prionas ND, Lindfors KK, Ray S, Huang SY, Beckett LA, Monsky WL, et al. Contrast-enhanced dedicated breast CT: initial clinical experience. Radiology. 2010;256:714–23. pmid:20720067
- 17.
Morris EA, Comstock C, Lee C, Lehman CD, Ikeda DM, Newsteed GM. ACR BI-RADS® Magnestic Resonance Imaging. In: ACR BI-RADS® Atlas, Breast Imaging Reporting and Data System Reston, VA: American College of Radiology. 2013.
- 18. Hanley JA, McNeil BJ. The Meaning under Characteristic a Receiver and Use of the AreaOperating (ROC) Curve. Radiology. 1982;143(1):29–36. pmid:7063747
- 19. Landis JR, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977;33(2):363–74. pmid:884196
- 20. Baltzer PA, Benndorf M, Dietzel M, Gajda M, Runnebaum IB, Kaiser WA. False-positive findings at contrast-enhanced breast MRI: a BI-RADS descriptor study. AJR American journal of roentgenology. 2010;194(6):1658–63. pmid:20489110.