Artificial intelligence (AI) is “the capability of a machine to initiate intelligent human behaviour.” The concept of deep learning (DL) in AI is based on learning data representations rather than on task-specific algorithms. The best-known advance in DL is its performance in the ImageNet competition; one of its goals is to assign each image to one of 1000 predefined categories. A DL-based algorithm that first appeared in 2012 dramatically reduced the error rate from 0.258 (http://image-net.org/challenges/LSVRC/2011/results) to 0.153 (http://image-net.org/challenges/LSVRC/2012/results.html). As the performance of DL algorithms for image classification has improved, and for many tasks, it is now considered comparable or superior to human performance.1,2
Artificial intelligence techniques are useful in the field of radiology because images, whose pixel values can be quantified, are the primary source data that can inform and train some AI algorithms. As image analysis has been the primary focus of DL, its value to radiology is obvious. The development of algorithms for radiology has shown some inertia because of the time needed for acquisition of the appropriate expertise in the medical imaging community as well as limited availability of large medical imaging data sets. However, the last 2 to 3 years have seen remarkable productivity in the field. It is now acknowledged that DL will play a significant role in radiology and radiologists must be familiarized with DL.
We review the premises and promises of DL and present its difference from machine learning. Besides computer-aided diagnosis and segmentation,3,4 DL may facilitate a reduction in the image noise, thereby improving the image quality. We describe the advantages and disadvantages of current computed tomography (CT) image reconstruction methods and discuss the potential value of a new DL-based CT image noise reduction technique designated DL reconstruction (DLR).
An Introduction to DL
Most AI currently used in the clinical setting is categorized as machine leaning (ML). In the field of diagnostic radiology, ML has been used for computer-aided diagnosis.5 Although ML is not a new technology, advances in the computer infrastructure have led to its wide acceptance. The use of support vector machines, a traditional feature-based approach, is now considered classical ML6 and it continues to be used7 because it is easy to interpret the output. However, traditional ML techniques are limited in their ability to process natural data in their raw form. The construction of a pattern-recognition or ML system required careful engineering and considerable domain expertise to design a feature extractor that transforms raw data, for example, the pixel values of an image, into a suitable internal representation or feature vector from which the learning subsystem, often a classifier, could detect or classify patterns in the input.8
Deep learning is a subset of ML in AI; its networks are capable of learning unsupervised from unstructured or unlabeled data. In general, DL consists of massive multilayer networks of artificial neurons. In the presence of large amounts of labeled and unlabeled data, it can automatically discover useful features, that is, representations of input data needed for tasks, such as lesion detection and classification.8,9 As DL automatically learns useful representations of the data, handcrafted features are not required; this is the key difference between DL and traditional ML techniques (Supplemental Figure 1, http://links.lww.com/RCT/A87). Several types of DL approaches have been developed for different purposes, such as object detection and segmentation on images, speech recognition, genotype/phenotype detection, and disease classification. Some DL algorithms are stacked autoencoders, deep Boltzmann machines, deep neural networks, and convolutional neural networks (CNNs). The deep convolutional neural networks (DCNN) method, which has its roots in the neocognitron,10 is commonly used in image recognition.11,12
Representations learned from one data set can be useful even when they are applied to a different set of data. Although this property known as transfer learning13,14 is not unique to DL, the large training data requirements of DL make it particularly useful when relevant data for a particular task are scarce. For instance, in clinical imaging, a DL system can be trained on a large number of natural images or on images acquired with different modalities to learn accurate feature representations that allow it to “see.” The pretrained system can then use these representations to produce an encoding of a clinical image that is used for classification.15,16 Systems using transfer learning often outperform state-of-the-art methods that are based on traditional handcrafted features that were developed for many years with a great deal of expertise.17
Noise Reduction With DL to Improve the Image Quality
Impressive improvements based on DL have drawn the attention of scientists engaged in the field of computational medical imaging. In addition to its usefulness for computer-aided diagnosis and segmentation,3,4 AI has attracted interest for its potential to improve the quality of medical images by reducing the image noise.18
NOISE REDUCTION USING DL
On digital images, image noise reduction, known as image denoising, is important and image processing often requires denoising as a preprocessing step. In the course of denoising, the noise component must be removed without degrading the true signal component. Deep learning applications that reduce the image noise are available for diagnostic imaging.19 For CT studies, radiation exposure can be reduced and for magnetic resonance (MR) studies, the acquisition time and the image noise can be reduced. Chen et al20 who developed a DL-based image noise reduction technique acquired the teaching data set from routine CT studies. Their DCNN was trained with virtual low-dose CT images generated by adding Poisson noise to the raw data of their teaching images. They trained the CNN in 200 cases; their test of 100 cases showed that the signal-to-noise ratio (SNR) was higher with theirs than other noise reduction methods. Du et al21 reported a noise reduction method for low-dose CT using a stacked competitive network composed of several successive competitive blocks that can increase the width of the network and improve the ability of the traditional CNN. They used a publicly available CT image data set available in the Cancer Imaging Archive for teaching data and generated virtual low-dose CT images by adding Poisson noise. The highest SNR and the lowest mean square error were obtained on images processed with stacked competitive network. Kang et al22 proposed a noise reduction method for CT images that incorporates wavelet transform into the DCNN. The input image is wavelet transformed, and the noise in the wavelet domain is removed by the DCNN, and then an output image is obtained with the wavelet recomposition process. By applying their method to quarter-dose abdominal CT scans, they were able to reduce the image noise without degrading visualization of the organ boundary. However, qualitative evaluation by radiologists showed that the texture of the denoised images differed from conventional images. Jiang et al23 applied a DCNN-based noise reduction method to brain MR images. Because the denoising CNN is a noise reduction DCNN for 2-dimensional images, they proposed a multichannel version of denoising CNN for 3-dimensional denoising. They obtained robust denoising performance when training was with images featuring various image noise levels.
NOISE REDUCTION USING DL FOCUSED ON CT IMAGE RECONSTRUCTION
As CT scans that evaluate the whole body in a single session are the primary modality for follow-up examinations and for determining the disease stage, their quality improvement is of utmost importance. Computed tomography image reconstruction is a mathematical process that generates tomographic images from x-ray projection data acquired at many different angles around the patient. For a given radiation dose, it is desirable to reconstruct images with the lowest possible noise without sacrificing image accuracy and spatial resolution.
Current CT Image Reconstruction
There are 2 major reconstruction methods, that is, analytical and iterative reconstruction (IR). The most commonly used analytical reconstruction methods on commercial CT scanners are in the form of filtered back projection (FBP). Because of the relatively low complexity of the underlying linear transformation from the projection space (raw data) to the image space, that is, the back projection, FBP is fast and robust and only requires limited computing power for CT image reconstruction routinely used in the clinical setting. However, at extremely low radiation doses, FBP may produce streak artifacts and a notable increase in the image noise.24
Iterative reconstruction techniques are classified as hybrid and model based. Hybrid IR, a reconstruction method based on FBP, has been used conventionally for CT image reconstruction; the iterative image noise reduction algorithm is combined with the reconstruction process. Because the reconstruction time of hybrid IR and FBP is similar, hybrid IR has widely replaced FBP.25 Unlike hybrid IR, model-based iterative reconstruction (MBIR) is an IR algorithm in the true sense. The correct, reconstructed image is acquired by iterating forward and back projections. By incorporating an optical and a statistical noise model into the iterative process, spatial resolution is improved, and the image noise is effectively eliminated from the reconstructed image.26 The advantage of MBIR is its higher spatial resolution compared with conventional FBP or hybrid IR (Supplemental Figure 2, http://links.lww.com/RCT/A88, lung CT images of a 60-year-old man. Reconstruction was with FBP [2a], hybrid IR [2b], and MBIR [2c]). However, the improved detectability of low-contrast lesions, particularly at low-dose tube flux levels and in larger patients, on MBIR images remains to be demonstrated.27–29 As MBIR involves longer computational time than FBP and hybrid IR, throughput is reduced.30 Consequently, a new generation of image reconstruction techniques is needed to obtain excellent images at low radiation doses and at computational costs as low as for hybrid IR.
Computed Tomography Image Reconstruction Using DL
Studies on replacing the image reconstruction process with DL have been published.19,25 One application is image space–based reconstruction in which CNNs are trained with low-dose CT images to reconstruct routine-dose CT images.20,31 Another approach is to optimize IR algorithms,32 which are generally based on manually designed prior functions that yield low-noise images without the loss of structures.33 Deep learning methods facilitate the implementation of more complex functions that may make it possible to reduce the radiation dose for CT22,33–35 and for sparse-sampling CT.36
Deep Learning Reconstruction
Deep learning reconstruction (Advanced Intelligent Clear-IQ Engine [AiCE], Canon Medical Systems), a DL-based image noise reduction technique, is the first commercialized DLR tool (https://mfl.ssl.cdn.sdlmedia.com/636837173033229994OU.pdf). For the DL-based approach, the training pair consists of hybrid IR and ideal MBIR images acquired at a high-dose setting with iterations exceeding those of commercially available MBIR. Statistical features that differentiate signals from noise and artifacts are “learned” in the training process and then “updated” in the DCNN kernel for subsequent inference use. The trained DCNN module is introduced into the reconstruction flow after hybrid IR (Supplemental Figure 3, http://links.lww.com/RCT/A89. [A] Schematic drawing of DLR; [B] The convolution neural network module introduced in DLR. The residual network architecture applied in this study [convolution, rectified linear unit activation function, and batch normalization layers]. Θ is the set of parameters to be optimized in the neural network, N the total number of training instances in the training process, R the neural network to be optimized, yi the ith element of the training input, and fi the ith element of the training target). As the DCNN kernel is trained on ideal MBIR images, the DLR approach is expected not only to generate images whose quality is comparable to MBIR images but also to reduce the image processing time.
Phantom Images Reconstructed With DLR
As phantom images revealed that DLR is superior to other reconstruction methods with respect to the visibility of low-contrast lesions acquired at a low radiation-dose setting (Supplemental Figure 4, http://links.lww.com/RCT/A90. Phantom CT images acquired at a low radiation dose and reconstructed with FBP [4a], hybrid IR [4b], MBIR [4c], and DLR [4d]. The image noise is reduced on the DLR image and the low-contrast lesion is more clearly visualized on the DLR than the other images), it may improve the detectability of low-contrast lesions that are not readily visualized on MBIR images, especially those obtained at low-dose settings.27–29 Further studies are needed to confirm this hypothesis. At high radiation dose settings, MBIR is superior to DLR with respect to the visibility of high-contrast lesions (Supplemental Figure 5, http://links.lww.com/RCT/A91. Phantom CT images acquired at a high radiation dose and reconstructed with FBP [5a], hybrid IR [5b], MBIR [5c], and DLR [5d]. The image noise is reduced on the DLR image. However, different from the other images, on the MBIR image the edge of the high-contrast lesion is smooth without overshooting/undershooting). The visualization of small vessels is better on MBIR than hybrid IR images because MBIR reduces the image noise while maintaining the image contrast and resolution.26,37,38 Therefore, DLR and MBIR may be complementary methods.39
Clinical Images Reconstructed With DLR
Clinical images of various areas showed that the image noise was consistently lower on DLR than the hybrid IR images; the image quality was better on DLR images and spatial resolution was not degraded (Figs. 1–3). The low-frequency noise on abdominal MBIR images was suppressed by DLR and the image quality was drastically improved (Fig. 1), suggesting that DLR is a robust reconstruction approach that can yield a consistent image quality of various areas.
Deep Learning Reconstruction at Ultrahigh-Resolution Computed Tomography
Ultrahigh-resolution CT (U-HRCT) features a smaller detector element and tube focus size than conventional CT and yields images of higher spatial resolution. However, compared with conventional CT, the image noise is greater on U-HRCT scans because of the relatively insufficient number of incident photons on smaller detectors. Consequently, increased noise may prevent its implementation especially for abdominal examinations.40–42 Akagi et al39 reported that DLR yielded higher-quality abdominal U-HRCT images than hybrid IR or MBIR (Figs. 4, 5), suggesting that it is an essential reconstruction method for U-HRCT.
Deep Learning Reconstruction for CT Screening
The radiation dose for CT screening must be as low as possible. Figure 6 shows lung cancer screening CT images acquired at 1.5 mGy. For lung CT, the quality of the DLR, hybrid IR, and MBIR images was comparable. However, the best image quality was obtained with DLR for abdominal CT because the image noise on the other reconstruction images was very high (Fig. 6). We suggest that DLR may be an essential technique for screening CT studies.
Computed Tomography Image Reconstruction Using DL by Other Vendors
Other vendors have already tried image reconstruction using DL. PixelShine (AlgoMedica PS [https://algomedica.com/]) is a DL-based denoising algorithm that applies DL techniques and ultra large amounts of high- and low-dose data for training. Tian et al43 reported that it improved the image quality of pelvic arterial-phase CT images, significantly reduced the image noise, and improved the SNR and the contrast-to-noise ratio. GE Healthcare also introduced a DL image reconstruction method (TrueFidelity), trained with high radiation dose CT images reconstructed with FBP (Fig. 7) (https://www.gehealthcare.co.uk/products/computed-tomography/revolution-apex). At present, the information regarding this product is limited.44
Because of the high performance of DL with respect to image recognition, its applicability for radiological imaging is increasing and radiologists must become familiar with DL. As this method improves the image quality, it is of diagnostic importance. Especially focused on CT image reconstruction, unlike MBIR, DLR can improve the quality of CT images of various areas even at low-dose radiation settings. Therefore, DLR may represent a breakthrough in image reconstruction field.
1. Chung JS, Senior AW, Vinyals O, et al. Lip reading sentences in the wild. CVPR
2. Taigman Y, Yang M, Ranzato M, et al. Deepface: closing the gap to human-level performance in face verification. Proc IEEE Conf Comput Vis Pattern Recognit
3. Lee JG, Jun S, Cho YW, et al. Deep learning in medical imaging: general overview. Korean J Radiol
4. Shen D, Wu G, Suk HI. Deep learning in medical image analysis. Annu Rev Biomed Eng
5. Giger ML. Machine learning in medical imaging. J Am Coll Radiol
. 2018;15(3 pt B):512–520.
6. Chartrand G, Cheng PM, Vorontsov E, et al. Deep learning: a primer for radiologists. Radiographics
7. Kunimatsu A, Kunimatsu N, Yasaka K, et al. Machine learning-based texture analysis of contrast-enhanced MR imaging to differentiate between glioblastoma and primary central nervous system lymphoma. Magn Reson Med Sci
8. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature
9. Schmidhuber J. Deep learning in neural networks: an overview. Neural Netw
10. Fukushima K, Miyake S. Neocognitron: a new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recognit
11. Akkus Z, Galimzianova A, Hoogi A, et al. Deep learning for brain MRI segmentation: state of the art and future directions. J Digit Imaging
12. Yasaka K, Akai H, Kunimatsu A, et al. Liver fibrosis: deep convolutional neural network for staging by using gadoxetic acid-enhanced hepatobiliary phase MR images. Radiology
13. Pan SJ, Yang Q. A survey on transfer learning. IEEE Trans Knowl Data Eng
14. Weiss K, Khoshgoftaar TM, Wang D. A survey on transfer learning. J Big Data
15. Azizi S, Mousavi P, Yan P, et al. Transfer learning from RF to B-mode temporal enhanced ultrasound features for prostate cancer detection. Int J Comput Assist Radiol Surg
16. Samala RK, Chan HP, Hadjiiski L, et al. Mass detection in digital breast tomosynthesis: deep convolutional neural network with transfer learning from mammography. Med Phys
17. Sahiner B, Pezeshk A, Hadjiiski LM, et al. Deep learning in medical imaging and radiation therapy. Med Phys
18. Goldman LW. Principles of CT: radiation dose and image quality. J Nucl Med Technol
19. Higaki T, Nakamura Y, Tatsugami F, et al. Improvement of image quality at CT and MRI using deep learning. Jpn J Radiol
20. Chen H, Zhang Y, Zhang W, et al. Low-dose CT via convolutional neural network. Biomed Opt Express
21. Du W, Chen H, Wu Z, et al. Stacked competitive networks for noise reduction in low-dose CT. PLoS One
22. Kang E, Min J, Ye JC. A deep convolutional neural network using directional wavelets for low-dose X-ray CT reconstruction. Med Phys
23. Jiang D, Dou W, Vosters L, et al. Denoising of 3D magnetic resonance images with multi-channel residual learning of convolutional neural network. Jpn J Radiol
24. Stiller W. Basics of iterative reconstruction methods in computed tomography: a vendor-independent overview. Eur J Radiol
25. Willemink MJ, Noel PB. The evolution of image reconstruction for CT-from filtered back projection to artificial intelligence. Eur Radiol
26. Kaza RK, Platt JF, Goodsitt MM, et al. Emerging techniques for dose optimization in abdominal CT. Radiographics
27. Nishizawa M, Tanaka H, Watanabe Y, et al. Model-based iterative reconstruction for detection of subtle hypoattenuation in early cerebral infarction: a phantom study. Jpn J Radiol
28. Euler A, Stieltjes B, Szucs-Farkas Z, et al. Impact of model-based iterative reconstruction on low-contrast lesion detection and image quality in abdominal CT: a 12-reader-based comparative phantom study with filtered back projection at different tube voltages. Eur Radiol
29. Racine D, Ba AH, Ott JG, et al. Objective assessment of low contrast detectability in computed tomography with Channelized Hotelling Observer. Phys Med
30. Higaki T, Nakamura Y, Fukumoto W, et al. Clinical application of radiation dose reduction at abdominal CT. Eur J Radiol
31. Wolterink JM, Leiner T, Viergever MA, et al. Generative adversarial networks for noise reduction in low-dose CT. IEEE Trans Med Imaging
32. Kopp FK, Catalano M, Rummeny EJ, et al. Evaluation of a machine learning based model observer for x-ray CT. Proc SPIE
. 2018. https://doi.org/10.1117/12.2293582
33. Wu D, Kim K, El Fakhri G, et al. Iterative low-dose CT reconstruction with priors trained by artificial neural network. IEEE Trans Med Imaging
34. Chen Y, Liu J, Xie L, et al. Discriminative prior - prior image constrained compressed sensing reconstruction for low-dose CT imaging. Sci Rep
35. Yi X, Babyn P. Sharpness-aware low-dose CT denoising using conditional generative adversarial network. J Digit Imaging
36. Lee H, Lee J, Kim H, et al. Deep-neural-network based sinogram synthesis for sparse-view CT image reconstruction. IEEE Transactions on Radiation and Plasma Medical Sciences
37. Higaki T, Tatsugami F, Fujioka C, et al. Visualization of simulated small vessels on computed tomography using a model-based iterative reconstruction technique. Data Brief
38. Hur BY, Lee JM, Joo I, et al. Liver computed tomography with low tube voltage and model-based iterative reconstruction algorithm for hepatic vessel evaluation in living liver donor candidates. J Comput Assist Tomogr
39. Akagi M, Nakamura Y, Higaki T, et al. Deep learning reconstruction improves image quality of abdominal ultra-high-resolution CT. Eur Radiol
40. Kakinuma R, Moriyama N, Muramatsu Y, et al. Ultra-high-resolution computed tomography of the lung: image quality of a prototype scanner. PLoS One
41. Yanagawa M, Hata A, Honda O, et al. Subjective and objective comparisons of image quality between ultra-high-resolution CT and conventional area detector CT in phantoms and cadaveric human lungs. Eur Radiol
42. Nakayama Y, Awai K, Funama Y, et al. Abdominal CT with low tube voltage: preliminary observations about radiation dose, contrast enhancement, image quality, and noise. Radiology
43. Tian SF, Liu AL, Liu JH, et al. Potential value of the PixelShine deep learning algorithm for increasing quality of 70 kVp+ASiR-V reconstruction pelvic arterial phase CT images. Jpn J Radiol
44. Yue M, Tang J, Nett BE, et al. Evaluation of image quality of a deep learning image reconstruction algorithm. Proc SPIE