Deep Learning-Based Optical Coherence Tomography and Optical Coherence Tomography Angiography Image Analysis: An Updated Summary : The Asia-Pacific Journal of Ophthalmology

Secondary Logo

Journal Logo

Review Article

Deep Learning-Based Optical Coherence Tomography and Optical Coherence Tomography Angiography Image Analysis: An Updated Summary

Ran, Anran PhD; Cheung, Carol Y. PhD

Author Information
Asia-Pacific Journal of Ophthalmology 10(3):p 253-260, May-June 2021. | DOI: 10.1097/APO.0000000000000405


Over the past two decades, optical coherence tomography (OCT) has become one of the most important imaging modalities in ophthalmology. It is a non-invasive technology that can generate in vivo structural images by detecting interference signals between the reflected signals from the reference mirror and the backscattering signals from biological tissues.1 OCT visualizes structures of the eye with cross-sectional and three-dimensional (3D) volumetric scans objectively and quantitatively. The advances in OCT technology (ie, from time-domain OCT to spectral/Fourier-domain OCT and swept-source OCT) have largely improved the image quality, axial resolution, acquisition speed, and structural details. With retinal layered segmentation by image processing methods and cross-sectional morphology assessment, it offers quantitative thickness measurement and more detailed pathology evaluation, which provides a more accurate evaluation for retinal features compared with 2D fundus photographs.2,3 Currently, OCT is widely used for retinal disease detection (eg, diabetic macular edema and age-related macular degeneration) and prognosis monitoring (eg, response to anti-vascular endothelial growth factor treatment).4 It has also been applied to assess glaucoma-related structural damage (eg, reduction of retinal nerve fiber layer thickness) in eye clinics soon after its demonstration,5 and progress were made in using it for glaucoma diagnosis and management.6

In addition to structural imaging, OCT has been explored and extended for “dynamic” imaging and en face imaging to map the retinal capillary network and choriocapillaris without the use of exogenous intravenous dye injection, based on the principle of mapping red blood cell movement over time by comparing sequential OCT sagittal-scans at a given cross-section, namely OCT angiography (OCTA). OCTA not only offers high-quality images of retinal and choroidal vessels at capillary-level but also allows quantification of vascular metrics that correlate with clinical disease severity and progression.7 Recently, quantitative OCTA metrics and features (eg, foveal avascular zone area, capillary network density, non-perfusion area) have been defined and explored for assessing retinal diseases such as age-related macular degeneration (AMD), diabetic retinopathy (DR), and retinal vein occlusion (RVO),8,9 and glaucoma.10

Artificial intelligence (AI) has been explosively applied to a wide variety of medical fields in the last decade. Machine learning (ML) is a subset of AI that teaches a computer system to perform a task or predict an outcome without explicitly programmed.11,12 In the development of traditional ML models, meaningful features are manually extracted by highly-trained domain expert. It is challenging to design and generalize such features, especially in medical images with variances in the anatomy and pathologies. On the contrary, deep learning (DL), under the term of ML, specifically uses multiple levels of classification with automatically extracted data features. Besides, with neural networks, DL models can handle data growth better than traditional ML models.13,14 Convolutional neural network (CNN),15 one kind of neural networks, was proposed by Krizhevsky et al15 and gathered much attention for its high performance in image recognition tasks compared to conventional methods. Since then, many models for image recognition tasks have been developed based on DL techniques. Contrary to natural images that are RGB images, OCT/OCTA includes 2D grayscale images and 3D volumes. Besides, OCT/OCTA images often contain quantitative information that is useful to improve neural networks performance. The exact intensity of a pixel, the scale of abnormalities, and their locations in a scan can all be essential cues. Thus, when applying DL in OCT/OCTA images, changing the network architectures developed for natural images and taking differences into account can significantly improve the results and push for clinically viable products.16 In ophthalmology, an imaging-driven specialty, DL was particularly well-suited and promising for disease evaluation.17–20 With the promising results from DL-based image analysis of 2D fundus photography,21–23 efforts quickly expanded toward 3D OCT and OCTA image analysis, given its widespread adoption and integration into the routine management of retinal diseases and glaucoma. Advances in DL have significantly improved the state-of-the-art OCT and OCTA image analysis, including disease detection and prognosis prediction, retinal layers and avascular area segmentation, and image quality assessment. DL-based models can be trained with different kinds of input from conventional OCT, such as 2D en face images, 2D B-scans, and 3D volumetric scans, and OCTA images.9,24

This review summarized recent studies on DL-based OCT and OCTA image analysis models and discussed the potential challenges of DL models’ clinical deployment and future research directions.

Deep Learning–Based Single Common Eye Disease Detection on OCT/OCTA

AMD is one of the leading causes of visual impairment in elderly patients. Drusen, pseudodrusen, neovascularization, fibrovascular scarring, and intraretinal or subretinal macular edema are typical retinal changes in AMD patients, which are detectable in OCT images.25 Many studies have shown the promise of DL in AMD detection from OCT images. Treder et al26 found that with a DL model developed by a pre-trained Inception-v3 network, it is possible to detect AMD in cross-sectional OCT B-scans with a sensitivity of 100%, a specificity of 92.0%, and an accuracy of 96.0%. As accurate differentiation of AMD subtypes is essential for treatment, Motozawa et al27 proposed two DL-based models for differentiating not only AMD and normal, but also wet and dry AMD from OCT 2D B-scans, respectively. The proposed DL models achieved area under the receiver operating characteristics (AUROC) values of 0.995 and 0.991, with sensitivities of 100% and 98.4%, specificities of 98.8% and 91.1%, accuracies of 99.0% and 93.3%, respectively. In the computer science domain, advanced preprocessing methods were effective in improving DL model performance for OCT image analysis. Rong et al28 proposed a surrogate-assisted CNN using different methods, including image denoising, thresholding, and morphological dilation for mask extraction, to generate surrogate images for CNN training. It achieved AUROC values of 0.978 and 0.986 for AMD detection in the internal validation and external testing datasets, respectively.

DR is a leading cause of blindness in the working-age population, and of an estimated 285 million people worldwide with diabetes mellitus (DM), more than one-third have signs of DR.29 DL is at the forefront in DR assessment with OCT or OCTA images. Despite the application of traditional CNN models, advanced techniques were also used to refine the models. For example, Li et al30 proposed a novel DL model called OCTD_Net based on modified DenseNet and ReLayNet. It classified early-stage DR (ie, mild DR), DM without DR, and normal subjects from OCT 2D B-scans, which achieved a sensitivity, specificity, and accuracy of 90.0%, 95.0%, and 92.0%, respectively. Heatmaps generated from their model demonstrated that patients with early-stage DR showed different features around the myoid and ellipsoid zones, inner nuclear layers, and photoreceptor outer segments. Le et al31 used a pre-trained visual geometry group (VGG-16) network to differentiate non-proliferative DR, DM without DR, and normal eyes from OCTA images. The overall AUROC, sensitivity, specificity, and accuracy were 0.965, 83.8%, 90.8%, and 87.3%, respectively. Heisler et al32 used ensemble learning techniques, which created and combined multiple models for performance improvement, to differentiate referable DR from non-referable DR on en face OCT and OCTA images. Though ensemble learning methods improved diagnostic performance, they also greatly increased the computational cost and training time.

RVO is the second most common retinal vascular disease after DR.33 Studies showed that the foveal avascular zone (FAZ) and visual acuity are reportedly inversely correlated in RVO.34 Thus, it is important to detect nonperfusion area (NPA) caused by RVO accurately. Nagasato et al35 found that a DL model could distinguish RVO from normal based on NPA measurement from OCTA images with an AUROC value of 0.986, a sensitivity of 93.7%, and a specificity of 97.3%. More importantly, it took less average time for evaluation than ophthalmologists (176.9 s vs 700.6 s, P value > 0.01). As shown from their heatmaps, the DL model focused on the FAZ area in normal OCTA images and the FAZ area and NPA in RVO OCTA images.

Studies have demonstrated good performance by using DL to interpret OCT optic disc scans for discriminating glaucomatous eyes from normal eyes. For example, Thompson AC et al36 developed a segmentation-free DL algorithm based on OCT 2D circular B-scans, and they found it achieved better performance for detecting glaucomatous structural changes comparing with conventional RNFL thickness parameters (AUROC: DL model vs RNFL thickness = 0.960 vs 0.870). Ran et al37 developed and validated a 3D DL model with volumetric data and achieved comparable performance to 2 specialists with more than 10 years’ experience in glaucoma. The heatmaps generated by class activation map (CAM) showed that the regions with the most discriminative power for the 3D DL model to detect glaucomatous optic neuropathy (GON) were similar to what ophthalmologists usually observe in clinics. The DL-based method was also proven applicable to detect referable glaucoma from OCT volumetric macula scans. It achieved high diagnostic performance in both internal validation (AUROC = 0.880) and 2 external testing (AUROC = 0.780 and 0.950, respectively).38

Deep Learning–Based Multiple Retinal Disease Detection on OCT/OCTA

The above DL algorithms were all proved to be feasible for detecting single eye disease or abnormality. However, DL models capable of detecting specific eye diseases as well as offering referral suggestions would be more applicable and valuable in clinics. Kermany et al39 proved that DL-based OCT B-scan analysis could provide accurate referral suggestions (urgent vs. non-urgent referral) by differentiating images with choroidal neovascularization (CNV) and diabetic macular edema (DME) from images with drusen and normal. The model was developed with VGG-16 pre-trained on the ImageNet dataset and achieved an AUROC value of 0.999, a sensitivity of 97.8%, a specificity of 97.4%, and an accuracy of 96.6%. They also found that a “limited model” trained with only 1000 cross-sectional OCT images was not much inferior to the model trained with a larger dataset including 108,312 cross-sectional images (AUROC 0.988, sensitivity 96.6%, specificity 94.0%, and accuracy 93.4%). More importantly, they released the OCT dataset with ground truth labels for other researchers’ further investigation.40–43 Li et al40 fine-tuned the network with VGG-16 and achieved slightly better performance with an AUROC value of 1, a sensitivity of 97.8%, a specificity of 99.4%, and an accuracy of 98.6%. Another study proposed a multi-task CNN to discriminate five retinal status (ie, CNV, DME, AMD, drusen, and normal), which showed an overall sensitivity of 97.1%, a specificity of 99.3%, and an accuracy of 97.1%.41 Tsuji et al42 proposed a capsule network, which achieved a higher mean accuracy with 99.6% on the released dataset. Zhang et al43 used a holistically-nested edge detection structure based on the VGG-16, which not only provided disease detection results but also extracted detection targets and enhanced data quality.

Another landmark study by De Fauw et al44 proved the clinical applicability of DL technology on interpreting images with higher volume and complexity. It successfully addressed the challenges of detecting retinal diseases on 3D OCT volumetric data. The DL architectures first produced tissue segmentation and then classified the tissue-segmentation maps into different diseases with referral suggestions (ie, urgent, semi-urgent, routine, and observation only). This study showed that the classification given by the DL model was based on device-independent tissue segmentation maps and its accuracy was free from the influence of device types. The referrable suggestions performance was comparable to experts. It was a novel framework to analyse OCT scans for multiple retinal diseases and provide referral suggestions, which could potentially reduce the burden of retinal disease triage.

It is noteworthy that the aforementioned studies were based on supervised DL methods, which requires substantial efforts to label large amount of data manually. New DL techniques, such as weakly supervised learning, semi-supervised learning, and unsupervised learning45 have great potential to deal with this issue. For example, Wang et al46 developed a weakly supervised learning-based model called uncertainty-driven deep multiple instance learning (UD-MIL) to discriminate DME and AMD patients from normal subjects on OCT images, which achieved robust performance in both B-scan levels and volumetric-scan levels.

Deep Learning–Based Rare Retinal Disease Detection on OCT/OCTA

Despite the detection of the above prevalent retinal diseases, the advances in DL technologies for small datasets, such as few-shot learning (FSL),47 also largely improved the feasibility of DL application to rare retinal disease detection. Yoo et al48 proposed the FSL model using a technique called generative adversarial network (GAN),49 which achieved an accuracy of 93.9% for nine kinds of retinal diseases classification. They proved that GAN was not only useful in improving the accuracy of rare retinal disease detection (ie, central serous chorioretinopathy, macular telangiectasia, macular hole, Stargardt disease, and retinitis pigmentosa) from OCT images, but also maintaining the diagnostic performance of prevalent retinal diseases detection (ie, DME, CNV, and drusen).

Deep Learning–Based Retinal Diseases Prognosis Prediction on OCT/OCTA

The introduction of anti-vascular endothelial growth factor (VEGF) has revolutionized the treatment of individuals with DME, CNV, and RVO. Although intravitreal anti-VEGF agents are the most common first line of therapy for these patients, not every patient responds to them. Besides, frequent injections of anti-VEGF agents are costly and burdensome.50,51 Thus, precise treatment response prediction could potentially avoid unnecessary anti-VEGF treatment and reduce clinical burden. A pilot study52 found that a refined VGG-based DL algorithm could predict responsive patients from non-responsive ones with an AUROC of 0.866, and average sensitivity, specificity, and precision of 80.1%, 85.0%, and 85.5%, respectively. Although only OCT images from DME patients were used, this study proved the feasibility of DL applications on disease prognosis prediction.

The level of treatment requirement is also essential for making the clinical decision, such as whether and when to conduct anti-VEGF treatment. A predictive DL algorithm was developed on a longitudinal OCT dataset collected from 350 neovascular AMD (nAMD) patients with 2 years’ follow-up.53 There were three kinds of predictive tasks: 1) treatment requirement score (RQS), 2) multi-class classification task including low, intermediate, and high requirement, 3) binary classification tasks (ie, low requirement vs. the remaining patients, high requirement vs. the remaining patients). The agreement between predicted RQS and the ground truth was moderate, but the model performed well for identifying low or high requirement patients. Although the predictive task was still in need of improvement, this study showed that DL was promising in disease prognosis prediction. A larger dataset with a longer follow-up time was warranted for further model refinement.

Deep Learning–Based Segmentation on OCT/OCTA

Macular fluids, such as the presence of intraretinal cystoid fluid (IRC) and subretinal fluid (SRF), can be used as qualitative OCT biomarkers to assist with treatment decision and prognosis evaluation. Schlegl et al54 proposed a DL algorithm to conduct automated detection, segmentation, and quantification of macular fluids across exudative macular diseases (ie, AMD, DME, and RVO) from two commonly used OCT devices (ie, Cirrus spectral-domain OCT and Spectralis SDOCT). It achieved the mean AUROC values of 0.94 and 0.92 for IRC and SRF detection, respectively. Quantification of fluid was also highly concordant with manual expert assessment (ie, Pearson's correlation coefficient of 0.90 for IRC and 0.96 for SRF). This DL method was further proven to be applicable to quantify the fluid volumes after anti-VEGF therapy in another clinical trial,55 demonstrating that the DL-based approach could offer a precise measurement of nAMD activity and allow identification of response patterns to improve therapeutic management of nAMD and avoid discrepancies between clinicians/investigators potentially.

Pekala et al56 also proposed a DL model using DenseNet together with post-processing based on Gaussian processes for retinal layers segmentation on OCT B-scans from subjects with mild non-proliferative DR. The DL model performed on par with humans and other ML methods.

In recent years, studies found that the analysis of the choroid in the eye was crucial for understanding a range of ocular diseases and physiological processes. However, only a very limited number of commercial OCT instruments provide methods for automatic segmentation of choroidal tissue. DL methods could segment the Bruch's membrane and choroid layer on OCT B-scans and calculate the choroidal thickness map calculation with a lower error rate and higher dice similarity coefficient.57 Kugelman et al58 developed several supervised deep learning models, such as Cifar CNN, Complex CNN, RNN, and U-net, to detect the location of the choroidal boundaries of interest, ie, inner limiting membrane (ILM), retinal pigment epithelium (RPE), and choroid-scleral interface (CSI). Their results suggested that all methods exhibit similar accuracy and good performance on the retinal layer (ILM and RPE) whereas performance on the CSI showed more variability between methods. All these studies demonstrated the potential of DL-based technologies for clinical and research tasks involving OCT choroidal segmentation.

Retinal blood vessels are not only valuable for the evaluation of retinal diseases but also for systemic diseases including hypertension, stroke, and cardiovascular diseases, potentially. A DL algorithm developed by U-net achieved an accuracy of 86.8% for artery and vein segmentation with OCT and OCTA en face images as the input.59 DL-based methods could also conduct microvascular segmentation on the SCP and DCP OCTA en face images with high accuracy and dice similarity coefficient.60 DL-based segmentation method was also applied on measurement and segmentation of FAZ area that was proven to outperform the automated measurements of FAZ by the built-in software.61,62

Deep Learning–Based Visual Function Prediction on OCT/OCTA

The above studies mainly focused on the detection or prediction of structural changes. Accurate assessments of visual functions, such as visual acuity (VA) and retinal sensitivity, were also important for clinical trial endpoints or treatment biomarkers evaluation. It could potentially be predicted from OCT measurements.63 Studies showed VA was correlated with central retinal thickness derived from OCT,64 whereas retinal sensitivity was significantly correlated with outer retinal thickness.65 Advancements in DL can provide promising tools for visual function predictions from structural changes detected on OCT images. Kawczynki et al66 proposed a DL-based regression model to predict the best-corrected visual acuity (BCVA) from OCT images of treatment-naïve nAMD patients. They used volumetric OCT scans as the input to predict the BCVA and classify poorer BCVA from better BCVA. The mean R2 of all visits and all eyes was 0.79. The AUROC values of binary classification of BCVA of < 69 letters, < 59 letter, and ≤ 38 letters were 0.92, 0.95, 0.96 at baseline, and 0.87, 0.89, 0.89 at month 12, respectively. It demonstrated that DL-based OCT image analysis could help predict concurrent and future BCVA in patients with nAMD, which would potentially assist with visual function evaluation during treatment. Kihara et al67 utilized a modified VGG-16 network to predict retinal sensitivity from OCT images of macular telangiectasia type 2 patients. They mapped the microperimetry sensitivities on OCT B-scans and generated en face sensitivity prediction maps. The correlation showed a high degree of agreement between predicted and real retinal sensitivities (Pearson correlation coefficient = 0.78).

Deep Learning–Based Image Quality Assessment of OCT/OCTA

Image quality control beforehand is essential for further accurate disease evaluation on OCT and OCTA images. Ran et al68 developed a DL-based image quality assessment model to differentiate ungradable OCT optic disc volumetric scans from gradable ones. It showed good performance in both internal validation and 2 external testing datasets, which could potentially be incorporated into DL-based OCT analysis model for glaucoma detection. Kauer et al69 proposed an automatic quality assessment network based on CNN to evaluate the quality of OCTA scans and the model achieved an accuracy of 99.5% to classify the image as good, bad, upper, and lower quality. Lauermann et al70 developed a multilayer deep CNN for automated assessment of superficial capillary plexus OCTA image quality (sufficient vs. insufficient). The sensitivity, specificity, and accuracy of the DL model were higher than 90.0%. Other studies utilized DL to reconstruct71 or denoise72 OCTA images, which largely enhanced the image quality and would potentially increase the number of sufficient images, decrease the image interpretation time, and further benefit retinal disease detection.

In addition to general image quality assessment, the detection of a specific artifact is also valuable. Guo et al73 had developed a CNN-based DL model to distinguish the capillary NPA from signal reduction artifacts in 6 × 6 mm2 OCTA. The capillary NPA is a key quantifiable biomarker in the evaluation of DR using OCTA. The proposed DL model achieved good performance for NPA detection across a wide range of DR severity and scan quality. It can potentially address the problem of signal reduction artifacts in the detection and quantification of capillary dropout in the retina using OCTA.


In summary, DL-based models were able to assist diseases classification, prognosis monitoring, functional changes prediction, structural segmentation, and image quality control on OCT and OCTA images. The most commonly used networks were VGG-16, Inception, ResNet, RNN, DenseNet for classification, prediction, and image quality control tasks, whereas U-Net was used for segmentation task. Other techniques, such as transfer learning and few-shot learning were also used in several studies to deal with relatively small training sample size. Major studies were summarized in eTables 1–5 ( Most of the studies were still in the stage of “proof-of-concept” and substantial issues are yet to be dealt with before deploying these DL models in real-time clinics.

First, the training sample size of the existing DL models for OCT or OCTA image analysis was relatively smaller than those for retinal fundus photographs. It potentially reduces the data diversities, results in model overfitting, and decreases the generalizability in unseen OCT/OCTA datasets. Different data augmentation methods, such as rotation, flipping, shifting, and cropping, are useful to enlarge the training sample size. Besides, new explorations in the computer science domain, such as transfer learning, low-shot learning,74 or few-shot learning,48 also show promises in dealing with this issue. Although current DL-based OCT and OCTA models can be further upgraded for rare diseases, researchers should be cautious when using advanced DL technologies such as GAN in medicine as the pseudo images could contain unobservable or unexplainable features, or vice versa, losing some features in the true images.75

Second, OCT images are 3D volumetric data, containing a lot of noises and variances. It requires high-performance preprocessing methods, a series of operations on the initial images, to improve the image quality and produce standardized training input from different devices. Denoising, contrast enhancement, image normalization, segmentation were commonly used methods.76 However, the preprocessing operations may increase the computation time and memories for dealing with 3D images, more explorations are still warranted to tackle to the dilemma. Wang et al's46 study potentially reduced great efforts on sophisticated preprocessing as the weakly supervised learning model did not require preprocessing and was robust to handle speckle noise.

Third, most of the DL models for OCT/OCTA image analysis were usually trained and validated on an internal dataset without proper external testing. However, there are less available open-source datasets of OCT or OCTA than that of retinal fundus photographs,77 and the data sharing among institutions is quite challenging due to issues of patient privacy and large storage requirement. “Model-to-Data” approach, by packaging an easily share DL model and bringing it to the data, was proven a potential solution to the limitation of data exchange among different institutions. It could allow for central construction of a robust DL model, followed by distribution to multiple clinical sites where distributed training can take place.78 Federated learning (FL), a learning paradigm training algorithms collaboratively without exchanging the data itself, has the potential to address the problem of data governance and privacy. The DL model training occurs locally at each participating institution and only model characteristics (eg, parameters and gradients) are transferred. It enables gaining insights collaboratively, for example, in the form of a consensus model, without moving patient data beyond the firewalls of the institutions in which they reside.79

Fourth, different OCT and OCTA devices produce different types of images from various imaging protocols. Most of the existing DL models were trained and validated on one kind of image from one particular OCT or OCTA device, which may have limited capability to generalize on other devices or other types of images. Romo-Bucheli et al80 proposed unsupervised unpaired image translation models based on cycle consistency losses to deal with image variability across 2 kinds of OCT devices. These models improved the generalizability of segmentation models to other unseen OCT images from a different device. Future development to unify all types of OCT scans into one framework in a device-agnostic manner is needed, in addition to consistent upgrade and refinement of DL models.

Fifth, current DL models are relying on AI to detect things overlooked by human eyes, but the “black box” nature reduces their interpretability and reliability. Occlusion tests, activation maps, and salient maps were often used to generate heatmaps and to increase the transparency of DL models. Investigating the “black box” has now become known as explainable AI (XAI), which provides tools to reveal AI decisions. Maloca et al81 proposed a CNN for OCT segmentation task in monkey eyes, which was enhanced with a post hoc XAI technique called Traceable Relevance Explainability (T-REX). Post hoc XAI can highlight and visualize regions of the input data that lead to relevant prediction decisions after the neural network training process. In the future, more efforts are warranted to provide a real-time illustration of DL-based OCT and OCTA analysis results in clinics.

Sixth, although many studies claimed the non-inferior performance of proposed DL models to experienced ophthalmologists or retina specialists, the reports were not fully met with the guidelines82–84 and not comparable among different studies. Thus, standardized randomized control trials (RCT) and result reporting are warranted to provide a more comprehensive insight into the clinical impact of these DL models. For RCT, instead of just comparing the performance between DL-based methods and humans, or emphasizing its superiority, the comparison should be made between the performance of decision-makers with and without the assistance of AI tools.85

Seventh, current DL-based OCT/OCTA image analysis models are mainly aiming for one particular task. In clinics, it is ideal to have a full set and well-integrated DL system for image quality control, disease classification, triage, and prognosis prediction. Nevertheless, extra engineering is required for developing a user-friendly interface and incorporating different models. Besides, prospective validation in real-world settings with consistent upgrade and refinement is necessary to make sure the integrated DL-based OCT/OCTA image analysis system's performance is on par with human expertise.

Eighth, although some DL models performed well in internal validation, there are still present pertinent challenges for real-world application among different settings (ie, domain shift) due to the diversities in devices and imaging protocols, variances in ocular physiological anatomy, imbalance in the data distribution. Currently, domain adaptation techniques, such as synergistic fusion86 and batch normalization,87 are proposed to improve DL model robustness for medical image analysis.

Last but not least, this review mainly focuses on DL-based posterior OCT and OCTA analysis. However, there are other kinds of OCT in ophthalmology, such as anterior segment OCT,24 and intra-operative OCT,88 which are also with large potentials for the application of DL technologies. Besides, studies proved the feasibility of DL-based methods for systemic disease evaluations from fundus photographs.89,90 DL-based OCT image analysis for systemic disease evaluation is yet to be explored. It would surely be a promising research area as some systemic diseases showed structural changes measured by OCT and OCTA.91,92


DL-based OCT and OCTA image analysis models are showing their advantages in eye disease detection, feature segmentation, prognosis prediction, and image quality control. In the future, the explainability and transparency of DL models are paramount in final clinical deployment. Consistent upgrade and refinement of these models are necessary to make sure the models’ performance is on par with human expertise.

Literature Search

We searched databases of PubMed, Medline, Web of Science, Google Scholar, and Scopus for studies published in English up to January 31, 2021, using these keywords: “retinal diseases”, “macular diseases”, “glaucoma”, “optical coherence tomography/OCT”, “optical coherence tomography angiography/OCTA”, “image analysis”, “artificial intelligence/AI”, “deep learning/DL”, “deep neural network”, “convolutional neural network/CNN”, “deep learning techniques”, and “deep learning update”. The final reference list was generated based on relevance to the topics covered in this review.


1. Aumann S, Donner S, Fischer J. Bille JF, et al. Optical coherence tomography (OCT): principle and technical realization. High Resolution Imaging in Microscopy and Ophthalmology: New Frontiers in Biomedical Optics. Cham (CH): Springer; 2019. 59–85.
2. Ong SS, Cummings TJ, Vajzovic L, et al. Comparison of optical coherence tomography with fundus photographs, fluorescein angiography, and histopathologic analysis in assessing coats disease. JAMA Ophthalmol 2019; 137:176–183.
3. Wang YT, Tadarati M, Wolfson Y, et al. Comparison of prevalence of diabetic macular edema based on monocular fundus photography vs optical coherence tomography. JAMA Ophthalmol 2016; 134:222–228.
4. Müller PL, Wolf S, Dolz-Marco R. Bille JF, et al. Ophthalmic diagnostic imaging: retina. High Resolution Imaging in Microscopy and Ophthalmology: New Frontiers in Biomedical Optics. Cham (CH): Springer; 2019. 87–106.
5. Schuman JS, Hee MR, Arya AV, et al. Optical coherence tomography: a new tool for glaucoma diagnosis. Curr Opin Ophthalmol 1995; 6:89–95.
6. Bussel II, Wollstein G, Schuman JS. OCT for glaucoma diagnosis, screening and detection of glaucoma progression. Br J Ophthalmol 2014; 98:15–19.
7. Spaide RF, Fujimoto JG, Waheed NK, et al. Optical coherence tomography angiography. Prog Retin Eye Res 2018; 64:1–55.
8. Chalam KV, Sambhav K. Optical coherence tomography angiography in retinal diseases. J Ophthalmic Vis Res 2016; 11:84–92.
9. Sun ZH, Yang DW, Tang ZQ, et al. Optical coherence tomography angiography in diabetic retinopathy: an updated review. Eye 2021; 35:149–161.
10. Moghimi S, Hou H, Rao H, et al. Optical coherence tomography angiography and glaucoma: a brief review. Asia Pac J Ophthalmol (Phila) 2019; 8:115–125.
11. Simon A, Venkatesan S, Babu DRR. An overview of machine learning and its applications. Int J Elect Sci Eng 2015; 1:3.
12. Shinde PP, Shah S. A review of machine learning and deep learning applications. 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA), 2018, pp. 1–6.
13. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015; 521:436–444.
14. Kulkarni S, Seneviratne N, Baig MS, et al. Artificial intelligence in medicine: where are we now? Acad Radiol 2020; 27:62–70.
15. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Commun Acm 2017; 60:84–90.
16. Raghu M, Zhang CY, Kleinberg J, et al. Transfusion: understanding transfer learning for medical imaging. Adv Neural Inform Proc Syst 2019; 32:32.
17. Ting DSW, Pasquale LR, Peng L, et al. Artificial intelligence and deep learning in ophthalmology. Br J Ophthalmol 2019; 103:167–175.
18. Li JO, Liu H, Ting DSJ, et al. Digital technology, tele-medicine and artificial intelligence in ophthalmology: a global perspective. Prog Retin Eye Res 2020. 100900.
19. Ruamviboonsuk P, Cheung CY, Zhang X, et al. Artificial intelligence in ophthalmology: evolutions in Asia. Asia Pac J Ophthalmol (Phila) 2020; 9:78–84.
20. Cheung CY, Tang F, Ting DSW, et al. Artificial intelligence in diabetic eye disease screening. Asia Pac J Ophthalmol (Phila) 2019; 8:158–164.
21. Ting DSW, Cheung CY, Lim G, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 2017; 318:2211–2223.
22. Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 2016; 316:2402–2410.
23. Burlina PM, Joshi N, Pekala M, et al. Automated grading of age-related macular degeneration from color fundus images using deep convolutional neural networks. JAMA Ophthalmol 2017; 135:1170–1176.
24. Ran AR, Tham CC, Chan PC, et al. Deep learning in glaucoma with optical coherence tomography: a review. Eye 2021; 35:188–201.
25. Lim LS, Mitchell P, Seddon JM, et al. Age-related macular degeneration. Lancet 2012; 379:1728–1738.
26. Treder M, Lauermann JL, Eter N. Automated detection of exudative age-related macular degeneration in spectral domain optical coherence tomography using deep learning. Graefes Arch Clin Exp Ophthalmol 2018; 256:259–265.
27. Motozawa N, An G, Takagi S, et al. Optical coherence tomography-based deep-learning models for classifying normal and age-related macular degeneration and exudative and non-exudative age-related macular degeneration changes. Ophthalmol Ther 2019; 8:527–539.
28. Rong Y, Xiang D, Zhu W, et al. Surrogate-assisted retinal OCT image classification based on convolutional neural networks. IEEE J Biomed Health Inform 2019; 23:253–263.
29. Wong TY, Cheung CMG, Larsen M, et al. Diabetic retinopathy. Nat Rev Dis Primers 2016; 2:16012.
30. Li XC, Shen LL, Shen MX, et al. Deep learning based early stage diabetic retinopathy detection using optical coherence tomography. Neurocomputing 2019; 369:134–144.
31. Le D, Alam M, Yao CK, et al. Transfer learning for automated OCTA detection of diabetic retinopathy. Transl Vis Sci Techn 2020; 9:35.
32. Heisler M, Karst S, Lo JL, et al. Ensemble deep learning for diabetic retinopathy detection using optical coherence tomography angiography. Transl Vis Sci Techn 2020; 9:20.
33. Lee JY, Yoon YH, Kim HK, et al. Baseline characteristics and risk factors of retinal vein occlusion: a study by the Korean RVO Study Group. J Korean Med Sci 2013; 28:136–144.
34. Balaratnasingam C, Inoue M, Ahn S, et al. Visual acuity is correlated with the area of the foveal avascular zone in diabetic retinopathy and retinal vein occlusion. Ophthalmology 2016; 123:2352–2367.
35. Nagasato D, Tabuchi H, Masumoto H, et al. Automated detection of a nonperfusion area caused by retinal vein occlusion in optical coherence tomography angiography images using deep learning. Plos One 2019; 14:e0223965.
36. Thompson AC, Jammal AA, Berchuck SI, et al. Assessment of a segmentation-free deep learning algorithm for diagnosing glaucoma from optical coherence tomography scans. JAMA Ophthalmol 2020; 138:333–339.
37. Ran AR, Cheung CY, Wang X, et al. Detection of glaucomatous optic neuropathy with spectral-domain optical coherence tomography: a retrospective training and validation deep-learning analysis. Lancet Dig Health 2019; 1:e172–e182.
38. Russakoff DB, Mannil SS, Oakley JD, et al. A 3D deep learning system for detecting referable glaucoma using full OCT macular cube scans. Transl Vis Sci Techn 2020; 9:12.
39. Kermany DS, Goldbaum M, Cai W, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 2018; 172:1122–1131.
40. Li F, Chen H, Liu Z, et al. Fully automated detection of retinal disorders by image-based deep learning. Graefes Arch Clin Exp Ophthalmol 2019.
41. Alqudah AM. AOCT-NET: a convolutional network automated classification of multiclass retinal diseases using spectral-domain optical coherence tomography images. Med Biol Eng Comput 2020; 58:41–53.
42. Tsuji T, Hirose Y, Fujimori K, et al. Classification of optical coherence tomography images using a capsule network. BMC Ophthalmol 2020; 20:114.
43. Zhang Q, Liu Z, Li JX, et al. Identifying diabetic macular edema and other retinal diseases by optical coherence tomography image and multiscale deep learning. Diabet Metab Synd Ob 2020; 13:4787–4800.
44. De Fauw J, Ledsam JR, Romera-Paredes B, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med 2018; 24:1342–1350.
45. Hosoda Y, Miyake M, Yamashiro K, et al. Deep phenotype unsupervised machine learning revealed the significance of pachychoroid features in etiology and visual prognosis of age-related macular degeneration. Sci Rep-Uk 2020; 10:18423.
46. Wang X, Tang FY, Chen H, et al. UD-MIL: uncertainty-driven deep multiple instance learning for OCT image classification. IEEE J Biomed Health 2020; 24:3431–3442.
47. Wang YQ, Yao QM, Kwok JT, et al. Generalizing from a few examples: a survey on few-shot learning. Acm Comput Surv 2020; 53:1–34.
48. Yoo TK, Choi JY, Kim HK. Feasibility study to improve deep learning in OCT diagnosis of rare retinal diseases with few-shot classification. Med Biol Eng Comput 2021; 59:401–415.
49. Goodfellow IJ, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets. Adv Neur In 2014; 27:2672–2680.
50. Gonzalez VH, Campbell J, Holekamp NM, et al. Early and long-term responses to anti vascular endothelial growth factor therapy in diabetic macular edema: analysis of protocol I data. Am J Ophthalmol 2016; 172:72–79.
51. Suzuki M, Nagai N, Izumi-Nagai K, et al. Predictive factors for non-response to intravitreal ranibizumab treatment in age-related macular degeneration. Br J Ophthalmol 2014; 98:1186–1191.
52. Rasti R, Allingham MJ, Mettu PS, et al. Deep learning-based single-shot prediction of differential effects of anti-VEGF treatment in patients with diabetic macular edema. Biomed Opt Express 2020; 11:1139–1152.
53. Romo-Bucheli D, Erfurth US, Bogunovic H. End-to-end deep learning model for predicting treatment requirements in neovascular AMD from longitudinal retinal OCT imaging. IEEE J Biomed Health Inform 2020; 24:3456–3465.
54. Schlegl T, Waldstein SM, Bogunovic H, et al. Fully automated detection and quantification of macular fluid in OCT using deep learning. Ophthalmology 2018. 549–558.
55. Schmidt-Erfurth U, Vogl WD, Jampol LM, et al. Application of automated quantification of fluid volumes to Anti-VEGF therapy of neovascular age-related macular degeneration. Ophthalmology 2020; 127:1211–1219.
56. Pekala M, Joshi N, Liu TYA, et al. Deep learning based retinal OCT segmentation. Comput Biol Med 2019; 114:103445.
57. Masood S, Fang RG, Li P, et al. Automatic choroid layer segmentation from optical coherence tomography images using deep learning. Sci Rep-Uk 2019; 9:3058.
58. Kugelman J, Alonso-Caneiro D, Read SA, et al. Automatic choroidal segmentation in OCT images using supervised deep learning methods. Sci Rep-Uk 2019; 9:13298.
59. Alam M, Le D, Son T, et al. AV-Net: deep learning for fully automated artery-vein classification in optical coherence tomography angiography. Biomed Opt Express 2020; 11:5249–5257.
60. Lo J, Heisler M, Vanzan V, et al. Microvasculature segmentation and intercapillary area quantification of the deep vascular complex using transfer learning. Transl Vis Sci Techn 2020; 9:
61. Mirshahi R, Anvari P, Riazi-Esfahani H, et al. Foveal avascular zone segmentation in optical coherence tomography angiography images using a deep learning approach. Sci Rep 2021; 11:1031.
62. Guo M, Zhao M, Cheong AMY, et al. Automatic quantification of superficial foveal avascular zone in optical coherence tomography angiography implemented with deep learning. Vis Comput Ind Biomed Art 2019; 2:21.
63. Keane PA, Sadda SR. Predicting visual outcomes for macular disease using optical coherence tomography. Saudi J Ophthalmol 2011; 25:145–158.
64. Browning DJ, Glassman AR, Aiello LP, et al. Relationship between optical coherence tomography-measured central retinal thickness and visual acuity in diabetic macular edema. Ophthalmology 2007; 114:525–536.
65. Issa PC, Troeger E, Finger R, et al. Structure-function correlation of the human central retina. Plos One 2010; 5:
66. Kawczynski MG, Bengtsson T, Dai J, et al. Development of deep learning models to predict best-corrected visual acuity from optical coherence tomography. Transl Vis Sci Techn 2020; 9:51.
67. Kihara Y, Heeren TFC, Lee CS, et al. Estimating retinal sensitivity using optical coherence tomography with deep-learning algorithms in macular telangiectasia type 2. Jama Netw Open 2019; 2:e188029.
68. Ran AR, Shi J, Ngai AK, et al. Artificial intelligence deep learning algorithm for discriminating ungradable optical coherence tomography three-dimensional volumetric optic disc scans. Neurophotonics 2019; 6:041110.
69. Kauer J, Gawlik K, Zimmermann HG, et al. Automatic quality evaluation as assessment standard for optical coherence tomography. Proc. SPIE 10868, Advanced Biomedical and Clinical Diagnostic and Surgical Guidance Systems XVII, 1086814 (26 February 2019);
70. Lauermann JL, Treder M, Alnawaiseh M, et al. Automated OCT angiography image quality assessment using a deep learning algorithm. Graef Arch Clin Exp 2019; 257:1641–1648.
71. Gao M, Guo YK, Hormel TT, et al. Reconstruction of high-resolution 6 x 6-mm OCT angiograms using deep learning. Biomed Opt Express 2020; 11:3585–3600.
72. Kadomoto S, Uji A, Muraoka Y, et al. Enhanced visualization of retinal microvasculature in optical coherence tomography angiography imaging via deep learning. J Clin Med 2020; 9:1322.
73. Guo Y, Hormel TT, Xiong H, et al. Development and validation of a deep learning algorithm for distinguishing the nonperfusion area from signal reduction artifacts on OCT angiography. Biomed Opt Express 2019; 10:3257–3268.
74. Burlina P, Paul W, Mathew P, et al. Low-shot deep learning of diabetic retinopathy with potential applications to address artificial intelligence bias in retinal diagnostics and rare ophthalmic diseases. JAMA Ophthalmol 2020; 138:1070–1077.
75. Sorin V, Barash Y, Konen E, et al. Creating artificial images for radiology applications using generative adversarial networks (GANs) - A systematic review. Acad Radiol 2020; 27:1175–1185.
76. Vasuki P, Kanimozhi J, Devi MB. A survey on image preprocessing techniques for diverse fields of medical imagery. 2017 IEEE International Conference on Electrical, Instrumentation and Communication Engineering (ICEICE), 2017:1–6.
77. Khan SM, Liu XX, Nath S, et al. A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability. Lancet Digit Health 2021; 3:E51–E66.
78. Mehta N, Lee CS, Mendonca LSM, et al. Model-to-data approach for deep learning in optical coherence tomography intraretinal fluid segmentation. JAMA Ophthalmol 2020; 138:1017–1024.
79. Rieke N, Hancox J, Li W, et al. The future of digital health with federated learning. NPJ Digit Med 2020; 3:119.
80. Romo-Bucheli D, Seebock P, Orlando JI, et al. Reducing image variability across OCT devices with unsupervised unpaired learning for improved segmentation of retina. Biomed Opt Express 2020; 11:346–363.
81. Maloca PM, Muller PL, Lee AY, et al. Unraveling the deep learning gearbox in optical coherence tomography image segmentation towards explainable artificial intelligence. Commun Biol 2021; 4:170.
82. Rivera SC, Liu XX, Chan AW, et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI Extension. Nat Med 2020; 26:1351–1363.
83. Liu XX, Rivera SC, Moher D, et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Lancet Digit Health 2020; 2:E537–E548.
84. Nagendran M, Chen Y, Lovejoy CA, et al. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ 2020; 368:m689.
85. Xie YC, Nguyen QD, Hamzah H, et al. Artificial intelligence for teleophthalmology-based diabetic retinopathy screening in a national programme: an economic analysis modelling study. Lancet Digit Health 2020; 2:E240–E249.
86. Chen C, Dou Q, Chen H, et al. Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation. Thirty-Third Aaai Conference on Artificial Intelligence /Thirty-First Innovative Applications of Artificial Intelligence Conference /Ninth Aaai Symposium on Educational Advances in Artificial Intelligence 2019;33:865–872.
87. Chang WG, You T, Seo S, et al. Domain-specific batch normalization for unsupervised domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019:7354–7362.
88. El-Haddad MT, Malone JD, Hoang NT, et al. Deep-learning based automated instrument tracking and adaptive-sampling of intraoperative OCT for video-rate volumetric imaging of ophthalmic surgical maneuvers. Proc. SPIE 10867, Optical Coherence Tomography and Coherence Domain Optical Methods in Biomedicine XXIII, 108671M (22 February 2019);
89. Cheung CY, Xu DJ, Cheng CY, et al. A deep-learning system for the assessment of cardiovascular disease risk via the measurement of retinal-vessel calibre. Nature Biomed Eng 2020;
90. Sabanayagam C, Xu DJ, Ting DSW, et al. A deep learning algorithm to detect chronic kidney disease from retinal photographs in community-based populations. Lancet Digit Health 2020; 2:E295–E302.
91. O’Bryhim BE, Apte RS, Kung N, et al. Association of preclinical Alzheimer disease with optical coherence tomographic angiography findings. JAMA Ophthalmol 2018; 136:1242–1248.
92. Wagner SK, Fu DJ, Faes L, et al. Insights into systemic disease through retinal imaging-based oculomics. Transl Vis Sci Techn 2020; 9:6.

artificial intelligence; deep learning; medical image analysis; optical coherence tomography; optical coherence tomography angiography

Supplemental Digital Content

Copyright © 2021 Asia-Pacific Academy of Ophthalmology. Published by Wolters Kluwer Health, Inc. on behalf of the Asia-Pacific Academy of Ophthalmology.