In 2017, Wang and colleagues trained various known CNN models to detect 8 abnormal patterns (atelectasis, cardiomegaly, effusion, infiltration, mass, nodules, pneumonia, and pneumothorax) on chest radiographs and achieved accuracy ranging from 0.56 to 0.78.46 Another study by Cicero and colleagues reported that, in a retrospective analysis of 35,038 chest radiographs from a single medical center using CNN (GoogLeNet), they were able to obtain MAP classification accuracy of 0.88.11 MAP detection in CXR with deep learning technology is still an area of active ongoing research, and different methodologies are being tested and validated, and overall accuracy will likely improve.
Unlike chest radiographs, chest CT provides cross-sectional images, allowing for direct 3-dimensional visualization of anatomic structures. Chest CT has a much higher sensitivity and lower interreader variability for detection of lung abnormalities and is frequently utilized in the diagnosis and follow-up of most pulmonary diseases. In addition, enhanced clinical availability, decreased cost, reduced radiation dose, and overall technical improvements of CT machines have resulted in a progressive increase in numbers of CT examinations performed each year. Therefore, an effective CAD system for chest CT interpretation would promote overall workflow for radiologists, by reducing the time required to read each CT exam and enhance reading accuracy.
Accurate nodule detection on chest CT has become a recent point of emphasis for efficient lung cancer screening. Despite advances in cancer treatment and screening programs, most lung cancer patients are still initially diagnosed at an advanced stage of the disease, which is associated with a <20% 5-year survival.47
Since the National Lung Screening Trial (NLST) announced a significant improvement (20%) in lung cancer mortality in high-risk populations when screened with low-dose chest CT (LDCT),48 LDCT for cancer screening has been widely accepted.49 Potentially this will lead to an increased number of LDCT, which will require expert analysis from a radiologist for the detection and classification of nodules into either benign or malignant diagnoses.
A CAD system could aid radiologists in both detection and classification of lung nodules (Fig. 6). Although traditional CAD systems have provided solid results, they often consist of complex pipelines of algorithms that depend heavily on manual human input such as preprocessing, segmentation, feature extraction, and model training, potentially hindering their performance.50 Application of deep learning technology, on the other hand, can potentially remove innate challenges in traditional CAD systems by providing seamless feature identification and classification and removing the need for complex human-led feature extraction pipelines.
In 2011, the Lung Image Database Consortium (LIDC) database, containing 1018 cases of thoracic CT scans and image annotations by 4 thoracic radiologists, was released and has motivated deep learning researchers to develop CAD systems for chest CT nodule detection and classification.51 CNNs are the most commonly utilized deep learning technology for lung nodule detection on CT images, and they achieve good nodule detection sensitivity while maintaining an acceptable false-positive rate. The first report of a CAD system with deep learning technology for lung nodule detection on CT was that of Hua and colleagues in 2015, achieving a sensitivity of 73% and a specificity of 80%, which was superior to any other available conventional CAD system.52 In 2016, Setio and colleagues trained CNN to detect pulmonary nodules and achieved 85.4% sensitivity with only one false-positive lesion per scan.53 Studies that are more recent have shown the ability of CNNs to boost nodule detection sensitivity on CT to a higher level (95%) but were associated with a wide range (1.17 to 22.4) of false-positive rates.54–56
Classification of detected lung nodules is also a potential area that could benefit from the use of CAD systems. CT characteristics of a lung nodule, mainly nodule type and size, are closely associated with the likelihood of malignancy. These CT features are important determinants for planning treatment and follow-up strategy. However, there is considerable observer variability in the classification of pulmonary nodules among radiologists, and this can lead to redundant follow-up examinations, unnecessary invasive procedures, or neglected malignancy.57 In 2017, Ciompi and colleagues introduced a deep learning system that achieved good performance for nodule-type classification based on lung-RADS system and was even within the interobserver variability among 4 experienced human readers.58 Furthermore, one study found that nodule classification accuracy of the CAD system was improved by combining deep residual learning, curriculum learning, and transfer learning.59 Other studies using different CNN models have achieved a classification accuracy as high as 87.1%.60,61
High-resolution CT is currently the diagnostic imaging tool of choice for the diagnosis and evaluation of ILDs. However, ILDs have similar appearance on CT, and CT readings are prone to high interobserver and intraobserver variability.63 Therefore, automatic identification and classification of different ILD patterns on chest CT may be helpful even for experienced chest radiologists, and application of deep learning technology could play an eminent role in developing such CAD systems. Segmentation of the lung with ILD could be enhanced by semantic segmentation with CNN. In 2016, deep learning technology with CNN showed an accuracy of 85% for classifying 6 different ILD patterns in a data set of 14,696 image patches.64 In 2017, Kim and colleagues compared shallow and deep learning methods on classifying 6 ILD patterns on CT; they found that deep learning methods showed significantly better accuracy, and that accuracy was further increased with the addition of more convolution layers65 (Fig. 7). More recently, a new CNN method achieved an ILD pattern classification accuracy of 87.9% using the holistic input of the entire CT data set.66 Moreover, CAD methodology demonstrated a prognostic ability of lung function decline using quantifiable ILD on CT studies.67
In 2017, Harrison and colleadues developed a deep model called progressive holistically nested networks (P-HNNs) and reported that their P-HNNs model showed significant improvements in lung segmentation performance compared with previous segmentation approaches.68 As for lobar segmentation, traditional methods are semiautomatic at best and largely rely on airway or vessel anatomy to delineate the lobar borders, with only few exceptions.69 To address these problems, a deep learning method for lobe segmentation was introduced in 2017, and this method achieved high accuracy without reliance on prior airway or vessel segmentations, even when tested in lungs that had an underlying disease70 (Fig. 8).
Aside from lung tissue segmentation, robust and reliable airway segmentation is also essential for quantitative evaluation of various diseases involving the airways, such as chronic obstructive pulmonary disease (Fig. 9). A large number of prior methods have common limitations—they are substantially influenced by morphologic changes in airway trees and by measurement errors, such as airway leaks that are most prevalent at smaller (or more peripheral) airways.71 In fact, 15 different traditional algorithms were evaluated at an airway segmentation challenge in 2009 (EXACT 09), and precise delineation of a small bronchus without airway measurement leaks remained a common unsolved problem from this challenge.72 In 2017, a deep learning method was developed and tested using a data set from EXACT 09, and it was found that CNN significantly decreased airway leaks during segmentation process, resulting in higher sensitivity and specificity, compared with all the other algorithms that participated in the EXACT 09 challenge.73 In another study, even with incompletely annotated data, 3D deep FCNs demonstrated considerable improvements in airway segmentation while maintaining acceptable quantity of airway leaks.74
The reconstruction kernel is one of the most important technical parameters that determine the trade-off between spatial resolution and image noise in CT.75 As the selection of kernel affects the quantitative analysis,76 CT images with different reconstruction kernels are necessary for various diagnostic or quantitative purposes. To overcome the limitation of difficulty in saving the raw data before reconstruction with various kernels, postprocessing techniques have been developed to permit interconversion among CT images obtained with different kernels. Kim et al77 recently demonstrated that CNNs could be taught differences between high-resolution and low-resolution images (residual images), and then they could be used to accurately and rapidly convert low-resolution images to high-resolution images. This approach is also applicable for interconverting CT images obtained using different kernels (Fig. 10).
Radiomics and prediction of patient outcomes (a.k.a.“deep survival”) are also active areas of research for the application of deep learning technology. Radiomics, which has gained substantial interest from researchers around the globe, involves the high-throughput extraction of quantitative features from medical images to develop reliable models to predict genomic information, clinical outcomes, and survival.78 Extraction of radiomics features is a critical process in radiomics research, and the majority of previous studies use hand-crafted features, which are limited by current medical knowledge and human observation. In contrast, CNN and transfer learning can be incorporated into radiomics models to extract more diverse features (deep features), which are free from prerequisite medical knowledge and biases. In this context, Lao and colleagues extracted 98,304 deep features (this would qualify as an example of overfitting of the data) from images of glioblastoma multiforme and found 6 deep features that could predict overall survival with a concordance index of 0.71.79
Second, CAD can help in the diagnosis of diseases such as ILD and generate a preliminary quantitative report based on CAD results. This CAD report is repeatable with the same results and has no “intrareader” variability. CAD combined with big data technology may retrieve similar images or diagnosis when radiologists require them during interpretation of CT. It can also help to reduce the reading time.
However, there are still many challenges to overcome. Currently, training deep learning algorithms requires large, strongly labeled, and anonymized image data sets. These data sets are very challenging to acquire. Although some abnormalities such as pneumothorax and malpositioned lines/tubes can be based on imaging findings alone, most diseases require clinical documentation and/or pathologic confirmation. Ambiguous or overlapping radiographic terms such as “consolidation” and “infiltrate” should not be used as surrogates for pneumonia to label training cases. This has been recognized as a limitation of some publicly available data sets. National organizations (eg, ACR, RSNA) and subspecialty radiology societies can play important roles in defining appropriate tasks for deep learning algorithms, as well as assisting in making publicly available strongly labeled training data sets and validation data sets.
Furthermore, the challenges with regard to the ethical and legal aspects of data sharing and patient privacy are also paramount. There are severe monetary penalties (ie, fines) in the United States84 for any medical facility that allows compromise of personal health information/images. In the United States, the Health Insurance Portability and Accountability Act (HIPPA) governs any use of a patient’s health information; as such, it is of critical importance that these imaging and medical data that are used for training, testing, and validation of deep learning methods are fully anonymized and comply with this law. New data protection laws have also been introduced throughout Europe. As deep learning requires an enormous amount of high-quality data, the laws governing the safe handling of medical images and medical record data need to be followed. New technology, such as Blockchain, may be helpful in guaranteeing secure data sharing.
1. LeCun Y, Bengio Y, Hinton G. Deep learning
. Nature. 2015;521:436–444.
2. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst. 2012;25:1097–1105.
3. Hinton G, Deng L, Yu D, et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Proc Mag. 2012;29:82–97.
4. Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning
algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316:2402–2410.
5. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–118.
6. Ehteshami Bejnordi B, Veta M, Johannes van Diest P, et al. Diagnostic assessment of deep learning
algorithms for detection of lymph node metastases in women with breast cancer. JAMA. 2017;318:2199–2210.
7. Ting DSW, Cheung CY, Lim G, et al. Development and validation of a deep learning
system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA. 2017;318:2211–2223.
8. McAdams HP, Samei E, Dobbins J III, et al. Recent advances in chest radiography
. Radiology. 2006;241:663–683.
9. Schalekamp S, van Ginneken B, Koedam E, et al. Computer-aided detection improves detection of pulmonary nodules in chest radiographs beyond the support by bone-suppressed images. Radiology. 2014;272:252–261.
10. Liang M, Tang W, Xu DM, et al. Low-dose CT screening for lung cancer: computer-aided detection of missed lung cancers. Radiology. 2016;281:279–288.
11. Cicero M, Bilbily A, Colak E, et al. Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs. Invest Radiol. 2017;52:281–287.
12. Lakhani P, Sundaram B. Deep learning
at chest radiography
: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology. 2017;284:574–582.
13. Gonzalez G, Ash SY, Vegas-Sanchez-Ferrero G, et al. Disease staging and prognosis in smokers using deep learning
in chest computed tomography
. Am J Respir Crit Care Med. 2018;197:193–203.
14. Robert C. Machine Learning
, A Probabilistic Perspective. Abington, UK: Taylor & Francis; 2014.
15. Hubel DH, Wiesel TN. Receptive fields and functional architecture of monkey striate cortex. J Physiol. 1968;195:215–243.
16. Dutta JK, Liu J, Kurup U, et al. Effective building block design for deep convolutional neural networks using search. arXiv preprint arXiv:2018;1801.08577:1–8.
17. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:2014;1409.1556:1–14.
18. Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition, 2015:1–9.
19. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016:770–778.
20. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, 2015:3431–3440.
21. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention, Springer. 2015:3431–3440.
22. He K, Gkioxari G, Dollár P, et al. Mask r-cnn. Computer Vision (ICCV), 2017 IEEE International Conference. 2017: 2980–2988.
23. Lin T-Y, Maire M, Belongie S, et al. Microsoft coco: common objects in context. European Conference on computer vision, Springer. 2014:740–755.
24. Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, 2014:580–587.
25. Girshick R. Fast r-cnn. arXiv preprint arXiv:2015;1504.08083:1–9.
26. Ren S, He K, Girshick R, et al. Faster r-cnn: towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst. 2015;28:91–99.
27. Redmon J, Divvala S, Girshick R, et al. You only look once: unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 779–788.
28. Huang Y, Liu Z, He L, et al. Radiomics signature: a potential biomarker for the prediction of disease-free survival in early-stage (I or II) non–small cell lung cancer. Radiology. 2016;281:947–957.
29. Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets. Advances in neural information processing systems, 2014:2672–2680.
30. Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:2015;1511.06434:1–16.
31. Reed S, Akata Z, Yan X, et al. Generative adversarial text to image synthesis. arXiv preprint arXiv:2016;1605.0539:1–10.
32. Bousmalis K, Silberman N, Dohan D, et al. Unsupervised pixel-level domain adaptation with generative adversarial networks. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.7.
33. Tzeng E, Hoffman J, Saenko K, et al. Adversarial discriminative domain adaptation. Computer Vision and Pattern Recognition (CVPR), 2017:4.
34. Ledig C, Theis L, Huszár F, et al. Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint.2016;1609.04802:1–19.
35. Sønderby CK, Caballero J, Theis L, et al. Amortised map inference for image super-resolution. arXiv preprint arXiv:1610.04490.2016;1610.04490:1–17.
36. Kamel SI, Levin DC, Parker L, et al. Utilization trends in noncardiac thoracic imaging, 2002-2014. J Am Coll Radiol. 2017;14:337–342.
37. Finigan JH, Kern JA. Lung cancer screening: past, present and future. Clin Chest Med. 2013;34:365–371.
38. Quekel LG, Kessels AG, Goei R, et al. Miss rate of lung cancer on the chest radiograph in clinical practice. Chest. 1999;115:720–724.
39. Li F, Engelmann R, Armato SG III, et al. Computer-aided nodule detection system: results in an unselected series of consecutive chest radiographs. Acad Radiol. 2015;22:475–480.
40. van Beek EJ, Mullan B, Thompson B. Evaluation of a real-time interactive pulmonary nodule analysis system on chest digital radiographic images: a prospective study. Acad Radiol. 2008;15:571–575.
41. Wang C, Elazab A, Wu J, et al. Lung nodule classification using deep feature fusion in chest radiography
. Comput Med Imaging Graph. 2017;57:10–18.
42. Pesce E, Ypsilantis P-P, Withey S, et al. Learning to detect chest radiographs containing lung nodules using visual attention networks. arXiv:2017;1712.00996:1–23.
43. Miller C, Lonnroth K, Sotgiu G, et al. The long and winding road of chest radiography
for tuberculosis detection. Eur Respir J. 2017;49:1700364.
44. Melendez J, Sanchez CI, Philipsen RH, et al. An automated tuberculosis screening strategy combining X-ray-based computer-aided detection and clinical information. Sci Rep. 2016;6:25265.
45. Pande T, Cohen C, Pai M, et al. Computer-aided detection of pulmonary tuberculosis on digital chest radiographs: a systematic review. Int J Tuberc Lung Dis. 2016;20:1226–1230.
46. Wang X, Peng Y, Lu L, et al. ChestX-Ray8: Hospital-scale chest x-Ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. arXiv:2017;1705.02315:1–19.
47. Siegel RL, Miller KD, Jemal A. Cancer statistics, 2018. CA Cancer J Clin. 2018;68:7–30.
48. National Lung Screening Trial Research Team, Aberle DR, Adams AM, Berg CD, et al. Reduced lung-cancer mortality with low-dose computed tomographic screening. N Engl J Med. 2011;365:395–409.
49. Oudkerk M, Devaraj A, Vliegenthart R, et al. European position statement on lung cancer screening. Lancet Oncol. 2017;18:e754–e766.
50. Goo JM. Computer-aided detection of lung nodules on chest CT: issues to be solved before clinical use. Korean J Radiol. 2005;6:62–63.
51. Armato SG III, McLennan G, Bidaut L, et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. Med Phys. 2011;38:915–931.
52. Hua KL, Hsu CH, Hidayati SC, et al. Computer-aided classification of lung nodules on computed tomography
images via deep learning
technique. Onco Targets Ther. 2015;8:2015–2022.
53. Setio AAA, Ciompi F, Litjens G, et al. Pulmonary nodule detection in CT images: false positive reduction using multi-view convolutional networks. IEEE Trans Med Imaging. 2016;35:1160–1169.
54. Hamidian S, Sahiner B, Petrick N, et al. 3D Convolutional neural network for automatic detection of lung nodules in chest CT. Proc SPIE Int Soc Opt Eng. 2017:10134.
55. Jiang H, Ma H, Qian W, et al. An automatic detection system of lung nodule based on multi-group patch-based deep learning
network. IEEE J Biomed Health Inform. 2017;22:1227–1237.
56. Masood A, Sheng B, Li P, et al. Computer-assisted decision support system in pulmonary cancer detection and stage classification on CT images. J Biomed Inform. 2018;79:117–128.
57. van Riel SJ, Sanchez CI, Bankier AA, et al. Observer variability for classification of pulmonary nodules on low-dose CT images and its effect on nodule management. Radiology. 2015;277:863–871.
58. Ciompi F, Chung K, van Riel SJ, et al. Towards automatic pulmonary nodule management in lung cancer screening with deep learning
. Sci Rep. 2017;7:46479.
59. Nibali A, He Z, Wollersheim D. Pulmonary nodule classification with deep residual networks. Int J Comput Assist Radiol Surg. 2017;12:1799–1808.
60. Zhao X, Liu L, Qi S, et al. Agile convolutional neural network for pulmonary nodule classification using CT images. Int J Comput Assist Radiol Surg. 2018;13:585–595.
61. Shen W, Zhou M, Yang F, et al. Multi-crop Convolutional Neural Networks for lung nodule malignancy suspiciousness classification. Pattern Recognit. 2017;61:663–673.
62. Harari S, Caminati A, Madotto F, et al. Epidemiology, survival, incidence and prevalence of idiopathic pulmonary fibrosis in the USA and Canada. Eur Respir J. 2017;49:1602384.
63. Watadani T, Sakai F, Johkoh T, et al. Interobserver variability in the CT assessment of honeycombing in the lungs. Radiology. 2013;266:936–944.
64. Anthimopoulos M, Christodoulidis S, Ebner L, et al. Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imaging. 2016;35:1207–1216.
65. Kim GB, Jung KH, Lee Y, et al. Comparison of shallow and deep learning
methods on classifying the regional pattern of diffuse lung disease. J Digit Imaging. 2017;31:415–424.
66. Gao M, Bagci U, Lu L, et al. Holistic classification of CT attenuation patterns for interstitial lung diseases via deep convolutional neural networks. Comput Methods Biomech Biomed Eng Imaging Vis. 2018;6:1–6.
67. Humphries SM, Yagihashi K, Huckleberry J, et al. Idiopathic pulmonary fibrosis: data-driven textural analysis of extent of fibrosis at baseline and 15-month follow-up. Radiology. 2017;285:270–278.
68. Harrison AP, Xu Z, George K, et al. Progressive and Multi-path Holistically Nested Neural Networks for Pathological Lung Segmentation from CT Images. Cham: Springer International Publishing; 2017:621–629.
69. Ross JC, San Jose Estepar R, Kindlmann G, et al. Automatic lung lobe segmentation using particles, thin plate splines, and maximum a posteriori estimation. Med Image Comput Comput Assist Interv. 2010;13:163–171.
70. George K, Harrison AP, Jin D, et al. Pathological Pulmonary Lobe Segmentation from CT Images Using Progressive Holistically Nested Neural Networks and Random Walker. Cham: Springer International Publishing; 2017:195–203.
71. Pu J, Gu S, Liu S, et al. CT based computerized identification and analysis of human airways: a review. Med Phys. 2012;39:2603–2616.
72. Lo P, van Ginneken B, Reinhardt JM, et al. Extraction of airways from CT (EXACT'09). IEEE Trans Med Imaging. 2012;31:2093–2107.
73. Charbonnier JP, Rikxoort EMV, Setio AAA, et al. Improving airway segmentation in computed tomography
using leak detection with convolutional networks. Med Image Anal. 2017;36:52–60.
74. Jin D, Xu Z, Harrison AP, et al. 3D Convolutional Neural Networks with Graph Refinement for Airway Segmentation Using Incomplete Data Labels. Cham: Springer International Publishing; 2017:141–149.
75. Schaller S, Wildberger JE, Raupach R, et al. Spatial domain filtering for fast modification of the tradeoff between image sharpness and pixel noise in computed tomography
. IEEE Trans Med Imaging. 2003;22:846–853.
76. Boedeker KL, McNitt-Gray MF, Rogers SR, et al. Emphysema: effect of reconstruction algorithm on CT imaging measures. Radiology. 2004;232:295–301.
77. Kim J, Kwon Lee J, Mu Lee K. Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 1646–1654.
78. Aerts HJ, Velazquez ER, Leijenaar RT, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun. 2014;5:4006.
79. Lao J, Chen Y, Li ZC, et al. A deep learning
-based radiomics model for prediction of survival in glioblastoma multiforme. Sci Rep. 2017;7:10353.
80. Paul R, Hawkins SH, Schabath MB, et al. Predicting malignant nodules by fusing deep features with classical radiomics features. J Med Imaging (Bellingham). 2018;5:011021.
81. Vovk U, Pernus F, Likar B. A review of methods for correction of intensity inhomogeneity in MRI. IEEE Trans Med Imaging. 2007;26:405–421.
82. Prevedello LM, Erdal BS, Ryu JL, et al. Automated critical test findings identification and online notification system using artificial intelligence in imaging. Radiology. 2017;285:923–931.
83. Suk HI, Shen D. Deep learning
-based feature representation for AD/MCI classification. Med Image Comput Comput Assist Interv. 2013;16:583–590.
84. Lee SI, Krishnaraj A, Chatterji M, et al. When does a radiologist’s recommendation for follow-up result in high-cost imaging? Radiology. 2012;262:544–549.