The field of glaucoma diagnosis is rapidly moving towards artificial intelligence (AI) for improving and enhancing patient care.[
1 2 3 4 ] For any AI algorithm to deliver successfully, the basis starts with data annotation.[ 5 ] This is where the human interface comes into play. Data annotation is the core ingredient to the success of any AI model. Take, for example, the facial recognition AI; the only way for the image detection AI tool to detect a face in a photo is, if many photos are labeled as “face” exist within the software. If there are no annotated data, then there is no machine learning algorithm in the first place, to detect the image. Annotating data also known as labeling data is the first and most important step in creating a successful AI model.[ 6 ] One such image annotating tool, which can be utilized by all, is Microsoft Visual Object Tagging Tool (VoTT) for a comprehensive and customized data labeling [ 6 Fig. 1].[ ] Customized annotations of the optic nerve head and retinal nerve fiber layer (RNFL) images [Figs. 7 2- 5] can prove useful, in not only identifying glaucomatous discs but also in predicting segmentation of the glaucomatous cup, disc, peripapillary atrophy, and RNFL defect in the background fundus separately, which has never been reported in the literature before, according to our knowledge. This methodology of annotations, though time-consuming, can be utilized by all ophthalmologists to create their own Human-in-the-loop (HITL) AI model. Figure 1:
The annotation toolbox with tools (green arrow) utilized for labeling the dataset in the Visual Object Tagging Tool (VoTT) software comprising the rectangle tool (red arrow) and polygonal tool (yellow arrow) for various types of labeling
Sample fundus photograph of an eye with glaucomatous cupping utilized for annotating
Annotation of the dataset. (a) Customized labeling of the optic cup (red-dotted area). (b) Customized labeling of the optic disc (pink-dotted area). (c) Customized labeling of peripapillary atrophy (gray-dotted area). (d) Complete annotation of an eye with glaucomatous changes in the optic nerve head
Sample fundus photograph of an eye with glaucomatous cupping and retinal nerve fiber layer defect utilized for annotating
Annotation of the dataset. (a) Customized labeling of the optic cup (green-dotted area). (b) Customized labeling of the optic disc (pink-dotted area). (c) Customized labeling of peri-papillary atrophy (gray-dotted area). (d) Customized annotation of the retinal nerve fiber layer (RNFL) slit defect (blue-dotted area). (e) Customized annotation of the RNFL arcuate defect (gray-dotted area). (f) Complete annotation of an eye with glaucomatous changes in the optic nerve head and RNFL region
HITL is the process of leveraging the power of machines and human intelligence to create AI models, where humans annotate data. In this loop, with humans help, the machine becomes smarter to take quick and accurate decisions. Customized human-led data annotation process can pave the way for future in AI, where the pairing of humans and machines takes place to yield better results, and not to establish the supremacy of one over the other.
Declaration of patient consent
The authors certify that they have obtained all appropriate patient consent forms. In the form, the patient(s) has/have given his/her/their consent for his/her/their images and other clinical information to be reported in the journal. The patient(s) understand that his/her/their name(s) and initials will not be published and due efforts will be made to conceal their identity, but anonymity cannot be guaranteed.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
1. Li Z, He Y, Keel S, Meng W, Chang RT, He M Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs Ophthalmology 2018 125 1199 206
2. Al-Aswad LA, Kapoor R, Chu CK, Walters S, Gong D, Garg A, et al. Evaluation of a deep learning system for identifying glaucomatous optic neuropathy based on color fundus photographs J Glaucoma 2019 28 1029 34
3. Cerentini A, Welfer D, Cordeiro d'Ornellas M, Pereira Haygert CJ, Dotto GN Automatic identification of glaucoma using deep learning methods Stud Health Technol Inform 2017 245 318 21
4. Haleem MS, Han L, Hemert JV, Li B, Fleming A, Pasquale LR, et al. A novel adaptive deformable model for automated optic disc and cup segmentation to aid glaucoma diagnosis J Med Syst 2017 42 20
5. Thompson AC, Jammal AA, Medeiros FA A deep learning algorithm to quantify neuroretinal rim loss from optic disc photographs Am J Ophthalmol 2019 201 9 18
6. What is data annotation and why does it matter Available from:
Last accessed on 2021 Jul 01
7. Image Annotation Tools:Which One to Pick in 2020 Available from:
Last accessed on 2021 Jul 01