Artificial intelligence (AI) using deep learning (DL) algorithms or systems has been identified as a major ‘disruptive innovation' in medicine and healthcare. In ophthalmology, AI and DL technology has been developed in several areas, with the two most prominent being in assessment of retinal photographs for detection and screening of diabetic retinopathy (DR),1-6 age-related macular degeneration (AMD),7-9 glaucoma,10,11 and retinopathy of prematurity,12,13 as well as the segmentation and assessment of optical coherence tomography (OCT) images for diagnosis and screening of major retinal diseases.14-16
For DR screening, DL has made remarkable breakthrough with some of the first few landmark studies in this area, as well as approval and registration of a fundus camera device for DR screening using DL technology by the US Food and Drug Administration (FDA).17 A study by Google Health, in particular, has drawn substantial media publicity, showing extremely high sensitivity and specificity in detecting referable DR from fundus photographs.3 These developments have been cited in mainstream media and editorials in non-ophthalmology literature.18-21
Despite the promise and hype, there remains significant challenges in the actual implementation of DL in DR screening, and translating DL research into direct clinical applications of screening in a community setting. This review will focus on the current progress and the development of using AI and DL technology for DR screening.
WHY IT IS IMPORTANT TO SCREEN FOR DR?
Screening for DR is based on a high level of evidence. By 2040, 642 million people worldwide is projected to have diabetes mellitus (DM).22 This is a global epidemic with heavy health burden to individuals and societies across the world, affecting populations not only in highly developed countries, but also increasingly in developing countries.22-24 Of note, DR is already the most specific microvascular complication of DM that leads to progressive visual impairment and even blindness. The prevalence of DR ranges from 30% to 45%, with 10% having vision-threatening DR (VTDR), defined as severe non-proliferative DR, proliferative DR, and diabetic macular edema (DME).25-28 Diabetes is estimated to account for 11.6% of the annual healthcare budgets in most countries.29 The cost of care for patients with DR is higher than those without DR, and increasing with severity.30 With a projected increase in the incidence of DM, increased life expectancy and an aging population, it is anticipated that the impact of vision impairment associated with DR will inevitably increase in the coming years. Early identification of eyes at risk of VTDR and timely treatment has been shown to be effective in preventing the onset of visual impairment.25 Recent results of randomized controlled trials have also shown that intraocular injections of drugs targeting vascular endothelial growth factor or inflammation in DME can lead to better visual outcomes.31 Early detection through regular surveillance by clinical examination or grading of retinal photographs is, therefore, the key to prevent vision loss and even blindness.
CURRENT CHALLENGES IN DR SCREENING PROGRAMS
While the rationale to screen for DR among patients with DM is clear, there are significant challenges in designing and sustaining a DR screening program in many communities and countries.
Systematic or national screening programs for DR using 2-dimensional (2D) non-stereoscopic digital fundus photography are currently implemented at primary care level in many countries, aiming to provide timely referral for more severe levels of DR to ophthalmologists for timely treatment and vision loss prevention.32-34 Diagnosis of DME requires identification of thickening of the macula; therefore, screening for DME using non-stereoscopic fundus photographs is bound to a very high false-positive rate (eg, >86% in Hong Kong35 and >79% in the UK36). Currently, spectral-domain optical coherence tomography (SD-OCT), which provides 3D volumetric data cube of the layered retinal structures, is being piloted to add in different DR screening programs in the UK, China and Singapore, aiming to reduce the false-positive rate of DME detection from fundus photography.37-39 When signs of referable DR or other referable eye diseases are identified from retinal photographs and SD-OCT, patients are referred to eye clinics at tertiary eye hospitals for clinical examination and management to prevent vision loss. However, interpretation of both retinal photographs and SD-OCT requires specialized knowledge and expertise in diabetic eye diseases and retinal imaging. Inter-individual differences in interpretation as well as potential of missing important retinal pathologies, especially with the engagement of non-ophthalmologist graders of varying skills and experience, are always concerned. Furthermore, current DR screening programs are capital- and labor-intensive, which makes it difficult to rapidly scale up and expand diabetic eye screening to meet the needs of this growing global epidemic.40 Given that the number of DM is expected to rise rapidly, manual grading of DR and DME from retinal images in DR screening programs will not be sustainable.
THE CASE FOR “AUTOMATED” MACHINE-BASED ASSESSMENT OF DR
In view of the challenges associated with DR screening, an automated program to assess DR from photographs has significant potential of increasing efficiency, reproducibility, and coverage of screening programs, and at the same time reducing cost and barriers to access, and improving patient outcomes by providing early detection and treatment.
Image interpretation using AI has been developed in the field of medicine. Over the last 2 decades, automated analysis of retinal photographs for DR has been studied using traditional machine learning methods based on feature extraction or pattern recognition specified by experts manually for DR characterization and classification.41-43 However, the detection of DR is a complex image-interpretation task. Based on feature extraction or pattern recognition using traditional machine learning algorithms, these systems still cannot reach a good level of specificity. Furthermore, traditional machine learning algorithms typically plateau in performance after it reaches a threshold of training data.
Recently, DL has been applied in the field of ophthalmology to outstrip the ceiling performance of prior technology, and has revolutionized the detection of DR/DME from retinal photographs and SD-OCT to achieve excellent diagnostic performance.
DL AS THE BREAKTHROUGH IN AI TECHNOLOGY
Deep learning is a new branch of machine learning technology under the broad term of AI. The concept is that DL permits software algorithms to be trained through a large mathematical function with millions of parameters on vast amounts of data called convolutional neural network (CNN). Such network is inspired by the ability of brains to learn complicated patterns in data by changing the strengths of synaptic connections between neurons. Deep learning uses deep networks with many intermediate layers of artificial “neurons” between the input and the output, and, like the visual cortex, these artificial neurons learn a hierarchy of progressively more complex feature detectors.44 In recent years, AI-DL with CNNs has been developed rapidly and has demonstrated strengths in intricate pattern recognition from big datasets in the medical field. For example, it is shown that the performance of automated classification for DR,1-5 retinal diseases,14-16 glaucoma10,11 as well as other medical conditions including malignant melanoma,45 tuberculosis,46 and lung cancer47 by DL systems are equal to or better than board-certified specialists. These studies underscoring the ability of automated image interpretation using DL are promising.
For image interpretation, DL is superior to traditional machine learning. In traditional machine learning, features need to be extracted manually, before they are fed in the machine learning algorithm. Using DL, the features are automatically learnt by neural networks in the feature extraction stage and then fed into a classifier for classification. No lesion-based features need to be specified in DL, and thus steps of traditional feature calculation or lesion segmentation machine learning algorithms can be skipped which required considerable domain expertise and careful engineering. Furthermore, such DL approach may identify “different features” that are associated with the desired classifi-cation and unknown to humans previously. Table 1 summarizes recent DL systems for screening DR and DME.1-6,14,15,48,49
DL IN RETINAL PHOTOGRAPH FOR DETECTING DR
There have been several DL systems developed by different groups for detecting DR and VTDR from retinal photographs.
- Abràmoff et al2 was the first to use a DL-enhanced algorithm for the automated detection of referable DR and VTDR using 1748 retinal photographs from Messidor-2 dataset. They found that the sensitivity of the DL system was not statistically different from their previously published algorithm based on traditional machine learning methods, but with a significantly better specificity.50 The DL system has recently been evaluated using data collected prospectively from 819 subjects recruited from 10 primary care sites in the US with high sensitivity (87.2%) and specificity (90.7%).48 The DL system has also been evaluated in primary care setting outside the US.51,52 In April 2018, the DL algorithm developed by Abramoff et al, called IDx-DR, has received the first approval from the FDA for detecting more-than- mild DR in adults who have DM using DL without assisted interpretation by a clinician.17 The case of IDx-DR highlights one of the earliest successes of a DL-based screening tool in completing the regulatory process.
- A major development in DL for DR detection was the study by Gulshan et al3 from Google Healthcare which developed a DL system for detecting referable DR using 128,175 retinal photographs which were graded 3 to 7 times for DR, DME, and image gradability by a panel of 54 US-licensed ophthalmologists and ophthalmology senior residents. The DL algorithm achieved a high sensitivity (≥87%), specificity (≥90%) and area under the receiver operating characteristic curve (AUC) (≥0.99) in the external validation using 2 public databases (EyePACS-1: n = 9963 and Messidor-2: n = 1748); both graded by at least 7 US board-certified ophthalmologists.3 Other groups have reported similar results. Gargeya and Leng4 developed a DL algorithm for detecting DR (any DR and mild DR) using 75,137 retinal photographs obtained from the EyePACS public dataset. The DL algorithm has further been evaluated using 2 other public databases (Messidor-2: n = 1748 and E-Ophtha: n = 405) with a sensitivity of ≥74%, specificity of ≥80%, and AUC of ≥0.83.4 Li et al5 developed another DL algorithm and internally validated it using 71,043 retinal photographs acquired from a total of 36 hospital ophthalmology departments, optometry clinics, and screening settings in China. In the external validation, the DL algorithm was evaluated using 35,201 retinal photographs from population-based cohorts of Malay, Caucasian Australians, and Indigenous Australians with a sensitivity of 92.5%, a specificity of 98.5% and an AUC of 0.955.5 Kanagasingam et al6 developed a DL algorithm based on several training dataset including DiaRetDB1, EyePACS, and the Australian Tele-eye care DR database. They deployed the DL algorithm prospectively and evaluated the performance in 193 patients at a primary care clinic in Australia with a specificity of 92% and a positive predictive value of just 12% driven by the low incidence of DR in their cohort (2 of 193).6
- A third major development addresses a major gap in previous studies in which the DL algorithms were not trained to detect other common sight-threatening eye conditions, such as glaucoma and AMD. To address this gap, Ting et al1 developed a DL system to screen for possible glaucoma and AMD, in addition to referable DR and VTDR. The DL system was composed of 8 CNNs, all using an adaptation of the VGGNet architecture: (i) an ensemble of 2 networks for the classification of DR severity; (ii) an ensemble of 2 networks for the identification of referable possible glaucoma; (iii) an ensemble of 2 networks for the identification of referable AMD; (iv) 1 network to assess image quality; and (v) 1 network to reject invalid non-retinal images. The DL system was validated using retinal photographs collected from the Singapore Integrated Diabetic Retinopathy Program (SIDRP) and from 10 additional multiethnic datasets from different countries with diverse community- and clinic-based populations with DM.1 In the primary validation dataset of 71,896 images from 17,974 patients, the AUC of the DL system was 0.936 for referable DR, 0.958 for VTDR, 0.941 for glaucoma suspect, and 0.931 for referable AMD. The DL system had AUCs between 0.889 and 0.983 for referable DR on 10 additional datasets (n = 40,752 images). In the secondary validation datasets, the DL system had AUCs between 0.889 and 0.983 for referable DR (n = 40,752 images). The DL system could also achieve high sensitivity (>90%), specificity (>73.3%) and AUC (>0.89) for identifying other referable eye conditions.
DL IN OCT FOR DETECTING DME
A major issue in DR screening is the detection of DME, which is not easily captured from 2-dimensional (2D) fundus photographs.35,38 Thus, recent studies have focused on demonstrating that DL algorithms can be trained using OCT images to detect DME and other retinal diseases. Kermany et al14 firstly applied DL and transfer learning techniques in the detection of DME as well as choroidal neovascularization, drusen, and normal from 2D OCT images. Two models (multiclass model and binary model) were firstly developed using a training dataset of 108,312 2D OCT images and a “limited model” was further trained using 1000 2D OCT images randomly selected from each class during training to compare transfer learning performance using limited data compared with results using a large dataset.14 In the validation, they evaluated the performance of the models of a dataset of 1000 2D OCT images and found that both models achieved high performance even in the limited model (sensitivity ≥96%, specificity ≥94%, and AUC ≥0.99).14 This study highlighted the application of transfer learning in conditions with small datasets.
Another major study was performed by Google's Deepmind in collaboration with Moorfields Eye Hospital, where the group applied a combined 3D segmentation and classification networks for interpreting 3D OCT images in predicting tissue volumes and identifying and referring different retinal abnormalities.15 They developed the DL algorithm using 877 segmented scans for the segmentation network and 14,884 scans from 7621 patients with multiple clinical diagnoses and referral decisions for classification network. The classification network learned to map a segmentation map to 4 referral decisions (urgent, semi-urgent, routine, and observation only) and 10 different retinal pathologies (macular retinal edema, choroidal neovascularization, full-thickness macular hole, partial-thickness macular hole, epiretinal membrane, geographic atrophy, drusen, vitreomacular traction, central serous retinopathy, and normal). Their framework (predicting tissue volumes, identifying retinal abnormalities, and providing referral suggestions) closely matched the clinical decision-making process, separating judgments about the scan itself from the subsequent referral decision, allowing a clinician to inspect and visualize an interpretable segmentation, rather than simply being presented with a diagnosis and referral suggestion.
VISUALIZING DL AND EXPLAINABILITY
A key challenge for clinical adoption of DL algorithms is a major mindset shift in how clinicians entrust clinical care to machines. The issue of ‘black box' problem has been identified as an impediment to the application of DL in healthcare.53 Previous reports have already performed visualization heat map analysis to represent intuitively the learning procedure of the CNNs for DR and DME.4,14 Recently, Keel et al54 conducted a study to analyze the CNN learning procedure for 2 validated DL systems including one for the detection of referable DR. They developed and applied a CNN-independent adaptive kernel visualization technique for visualizing the learning procedure of the networks. Using this method, discriminative image regions were highlighted when the classification possibility output of being diagnosed was greater than 70%, and a heat map was then generated highlighting highly prognostic regions in the input image. They found that typical referable DR lesions (retinal hemorrhage, exudate, intraretinal microvascular abnormality, venous beading) could be identified as the most important prognostic regions in 96% of true-positive referable DR cases whereas nontraditional fundus regions (eg, optic disc and fundus areas adjacent to vessels) were identified in 85% of false-positive referable DR cases. This study may further promote the clinical adoption of using DL algorithm and enable clinicians to understand important exposure variables in real time.
HOW DOES DL FIT IN CURRENT DR SCREENING STRATEGY?
A critical question in the conceptualization of using DL technology for DR screening is “where does the DL system fit?”. Deep learning systems could potentially be deployed in 2 different settings.1,18Figure 1 illustrates the 2 proposed DL-based screening models for DR, compared with an existing model. First, the DL system could be incorporated in existing DR screening programs such as in the UK and Singapore to assist human assessors in centralized reading centers (a semi-automated model).1,55,56 The retinal images can be firstly analyzed by the DL system and the human assessors only review images flagged as “referable” or “ungradable” by the DL system (2nd screening) for further confirmation. This model not only decreases the massive workload to read those non-referable retinal images but also avoids those false-positive cases referring to ophthalmologists (Fig. 1A). Second, the DL system could be a fully automated model in that all retinal images can be analyzed by the DL system which will be useful in communities without any existing DR screening programs (Fig. 1B). By using this screening model, the sensitivity of the DL system may need to be set higher in order to minimalize false-negative cases.
The performance of DL algorithms is extremely promising as seen from current literature. However, there are still several challenges and complexities that limit the ability to move forward quickly.
First, as mentioned above, the “black box” approach is still unacceptable. Deep learning uses millions of image features most predictive for classification rather than explicitly detecting clinical features that physicians are familiar with (eg, microaneurysms, hard exudates). It is unclear exactly what the machine “thinks” and “sees”.18 Transparency of DL algorithm is required in order to let users understand the basis to justify a particular diagnosis, treatment recommendation, or outcome prediction that the DL algorithm offered, and also allow examination for any potential bias.57
Second, AI most likely succeeds when it is used with high-quality “labeled” input data validated by multiple medical specialists for learning and classifying images in relation to outcomes. Also, DL algorithms are highly data hungry, often requiring millions of observations to reach acceptable performance levels.58 Moreover, training using single clinical dataset is always limited by potential biases (eg, same ethnicity, same device) which may affect both performance and generalizability.59,60 Using dataset from diverse settings and populations for training DL algorithm is ideal. Nevertheless, the availability of such dataset is very limited and it is very costly.
Third, “real-world” experiments are essential in the validation of DL algorithm. For example, prospective study design in appropriate clinical settings and testing DL algorithm using independent validation datasets under different settings and conditions that played no role in model development are critical to evaluate the performance of DL algorithms. Several recent studies have already reported the performance of DL algorithm using a prospective study design.6,48,61
Fourth, a continuous data supply is necessary for ongoing improvement and refinement of the DL algorithms and data may need to be shared across multiple centers for widespread implementation. However, the current healthcare environment holds little incentive for data sharing.57
Lastly, before DL algorithms or systems can be deployed in the clinical workflow of diabetic eye disease screening programs, there are still more questions need to be addressed. For example, “If a patient suffers an adverse event due to an AI-based technology, who is responsible?”.57 In addition, more rigorous scientific evidence on safety, validity, reproducibility, reliability, usability, and patient and clinician acceptability is also critical, before the clinical deployment.
In summary, AI and DL are entering the mainstream of clinical medicine. This technology can augment human intelligence to improve decision making and operational processes. Screening of DR is almost an ideal task for AI in health care. Looking forward, AI will become ubiquitous and indispensable for the screening, with expectation to improve the efficiency and accessibility of screening programs, and therefore will be able to prevent visual loss and blindness from this devastating disease.
1. Ting DSW, Cheung CY, Lim G, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA
2. Abràmoff MD, Lou Y, Erginay A, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci
3. Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA
4. Gargeya R, Leng T. Automated identification of diabetic retinopathy using deep learning. Ophthalmology
5. Li Z, Keel S, Liu C, et al. An automated grading system for detection of vision-threatening referable diabetic retinopathy on the basis of color fundus photographs. Diabetes Care
6. Kanagasingam Y, Xiao D, Vignarajan J, et al. Evaluation of artificial intelligence-based grading of diabetic retinopathy in primary care. JAMA Netw Open
7. Burlina PM, Joshi N, Pacheco KD, et al. Assessment of deep generative models for high-resolution synthetic retinal image generation of age-related macular degeneration. JAMA Ophthalmol
. 2019 Jan 10. Epub ahead of print.
8. Peng Y, Dharssi S, Chen Q, et al. DeepSeeNet: A deep learning model for automated classification of patient-based age-related macular degeneration severity from color fundus photographs. Ophthalmology
9. Burlina PM, Joshi N, Pacheco KD, et al. Use of deep learning for detailed severity characterization and estimation of 5-year risk among patients with age-related macular degeneration. JAMA Ophthalmol
10. Li Z, He Y, Keel S, et al. Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs. Ophthalmology
11. Medeiros FA, Jammal AA, Thompson AC. From machine to machine: An OCT-trained deep learning algorithm for objective quantification of glaucomatous damage in fundus photographs. Ophthalmology
12. Redd TK, Campbell JP, Brown JM, et al. Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity. Br J Ophthalmol
. 2018 Nov 23. Epub ahead of print.
13. Brown JM, Campbell JP, Beers A, et al. Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks. JAMA Ophthalmol
14. Kermany DS, Goldbaum M, Cai W, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell
15. De Fauw J, Ledsam JR, Romera-Paredes B, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med
16. Schlegl T, Waldstein SM, Bogunovic H, et al. Fully automated detection and quantification of macular fluid in OCT using deep learning. Ophthalmology
17. FDA permits marketing of artificial intelligence-based device to detect certain diabetes-related eye problems [news release]. US Food & Drug Administration; April 11, 2018. https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm604357.htm?fbclid=IwAR2h4qBpSAD6vx8EErts2nhhundX8654ZUL3xwvVahXyLxnSYSxCAOMEias.Accessed
February 24, 2019.
18. Wong TY, Bressler NM. Artificial intelligence with deep learning technology looks into diabetic retinopathy screening
19. Beam AL, Kohane IS. Translating artificial intelligence into clinical care. JAMA
20. Google AI matches diabetic retinopathy screens [medical news]. Medscap; November 30, 2016. https://www.medscape.com/viewarticle/872555
. Accessed February 24, 2019.
21. Metz C. India fights diabetic blindness with help from A.I. New York Times
. March 10, 2019: https://www.nytimes.com/2019/03/10/technology/artificial-intelligence-eye-hospital-india.html
. Accessed March 12, 2019.
22. Ogurtsova K, da Rocha Fernandes JD, Huang Y, et al. IDF Diabetes Atlas: Global estimates for the prevalence of diabetes for 2015 and 2040. Diabetes Res Clin Pract
23. Chan JC, Malik V, Jia W, et al. Diabetes in Asia: epidemiology, risk factors, and pathophysiology. JAMA
24. Chan JC, Zhang Y, Ning G. Diabetes in China: a societal solution for a personal challenge. Lancet Diabetes Endocrinol
25. Wong TY, Cheung CM, Larsen M, et al. Diabetic retinopathy. Nat Rev Dis Primers
26. Cheung N, Mitchell P, Wong TY. Diabetic retinopathy. Lancet
27. Yau JW, Rogers SL, Kawasaki R, et al. Global prevalence and major risk factors of diabetic retinopathy. Diabetes Care
28. Cheung CY, Ikram MK, Klein R, et al. The clinical implications of recent studies on the structure and function of the retinal microvasculature in diabetes. Diabetologia
29. Zhang P, Zhang X, Brown JB, et al. In: Economic Impact of Diabetes
. IDF Diabetes Atlas. 4th ed. International Diabetes Federation; 2010.
30. Zhang X, Low S, Kumari N, et al. Direct medical cost associated with diabetic retinopathy severity in type 2 diabetes in Singapore. PLoS One
31. Diabetic Retinopathy Clinical Research Network, Wells JA, Glassman AR, et al. Aflibercept, bevacizumab, or ranibizumab for diabetic macular edema. N Engl J Med
32. Wang LZ, Cheung CY, Tapp RJ, et al. Availability and variability in guidelines on diabetic retinopathy screening
in Asian countries. Br J Ophthalmol
33. Bhargava M, Cheung CY, Sabanayagam C, et al. Accuracy of diabetic retinopathy screening
by trained non-physician graders using non-mydriatic fundus camera. Singapore Med J
34. Wong TY, Sun J, Kawasaki R, et al. Guidelines on Diabetic Eye Care: The International Council of Ophthalmology Recommendations for Screening
, Follow-up, Referral, and Treatment Based on Resource Settings. Ophthalmology
35. Wong RL, Tsang CW, Wong DS, et al. Are we making good use of our public resources? The false-positive rate of screening
by fundus photography for diabetic macular oedema. Hong Kong Med J
36. Jyothi S, Elahi B, Srivastava A, et al. Compliance with the quality standards of National Diabetic Retinopathy Screening
Committee. Prim Care Diabetes
37. Olson J, Sharp P, Goatman K, et al. Improving the economic value of photographic screening
for optical coherence tomography-detectable macular oedema: a prospective, multicentre, UK study. Health Technol Assess
38. Goh JK, Cheung CY, Sim SS, et al. Retinal imaging techniques for diabetic retinopathy screening
. J Diabetes Sci Technol
39. Leal J, Luengo-Fernandez R, Stratton IM, et al. Cost-effectiveness of digital surveillance clinics with optical coherence tomography versus hospital eye service follow-up for patients with screen-positive maculopathy. Eye (Lond).
2018 Nov 30. Epub ahead of print.
40. Scanlon PH, Aldington SJ, Leal J, et al. Development of a cost-effectiveness model for optimisation of the screening
interval in diabetic retinopathy screening
. Health Technol Assess
41. Abràmoff MD, Reinhardt JM, Russell SR, et al. Automated early detection of diabetic retinopathy. Ophthalmology
42. Tufail A, Rudisill C, Egan C, et al. Automated diabetic retinopathy image assessment software: diagnostic accuracy and cost-effectiveness compared with human graders. Ophthalmology
43. Fleming AD, Goatman KA, Philip S, et al. Automated grading for diabetic retinopathy: a large-scale audit using arbitration by clinical experts. Br J Ophthalmol
44. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature
45. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature
46. Lakhani P, Sundaram B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology
47. Yu KH, Zhang C, Berry GJ, et al. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat Commun
48. Abràmoff MD, Lavin PT, Birch M, et al. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med
49. Li F, Chen H, Liu Z, et al. Fully automated detection of retinal disorders by image-based deep learning. Graefes Arch Clin Exp Ophthalmol
50. Abràmoff MD, Folk JC, Han DP, et al. Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol
51. Verbraak FD, Abramoff MD, Bausch GCF, et al. Diagnostic accuracy of a device for the automated detection of diabetic retinopathy in a primary care setting. Diabetes Care
52. van der Heijden AA, Abramoff MD, Verbraak F, et al. Validation of automated screening
for referable diabetic retinopathy with the IDx-DR device in the Hoorn Diabetes Care System. Acta Ophthalmol
53. Castelvecchi D. Can we open the black box of AI? Nature
54. Keel S, Wu J, Lee PY, et al. Visualizing deep learning models for the detection of referable diabetic retinopathy and glaucoma. JAMA Ophthalmol
. 2018 Dec 20. Epub ahead of print.
55. Nguyen HV, Tan GS, Tapp RJ, et al. Cost-effectiveness of a National Telemedicine Diabetic Retinopathy Screening
Program in Singapore. Ophthalmology
56. Scanlon PH. The English National Screening
Programme for diabetic retinopathy 2003-2016. Acta Diabetol
57. He J, Baxter SL, Xu J, et al. The practical implementation of artificial intelligence technologies in medicine. Nat Med
58. Obermeyer Z, Emanuel EJ. Predicting the future - big data, machine learning, and clinical medicine. N Engl J Med
59. Gianfrancesco MA, Tamang S, Yazdany J, et al. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med
60. Zou J, Schiebinger L. AI can be sexist and racist - it's time to make it fair. Nature
61. Keel S, Lee PY, Scheetz J, et al. Feasibility and patient acceptability of a novel artificial intelligence-based screening
model for diabetic retinopathy at endocrinology outpatient services: a pilot study. Sci Rep