Secondary Logo

Journal Logo


Development and Validation of an Open-Source Grading Tool for Outcome Assessment in Limbal Stem Cell Treatment

Behaegel, Joséphine MD*,†; Consejo, Alejandra PhD; Wouters, Kristien MSc, PhD†,§; Koppen, Carina MD, PhD*,†; De Cock, Joost; Ní Dhubhghaill, Sorcha MD, PhD*,†

Author Information
doi: 10.1097/ICO.0000000000002282


Corneal limbal stem cells, located at the basal layer of the limbal epithelium, maintain the corneal epithelium in a clear and transparent state. These stem cells are constantly replicating to replace the epithelial cells that are regularly shed with every blink.1,2 In addition, the limbus exerts a “barrier” function, separating the cornea and conjunctiva, preventing migration of conjunctival epithelial cells and blood vessels onto the corneal surface.3–5 Injury to this stem cell population, or to their protective niches, may result in limbal dysfunction which can lead to ocular surface inflammation, persistent epithelial defects, vascularization, and conjunctivalization.6 This is clinically known as “limbal stem cell deficiency” (LSCD) and manifests with symptoms of pain, photophobia, and visual loss for the patient.

The diagnosis of LSCD is mainly made based on the patient's history and slit-lamp findings. Clinical manifestations result from an absence of the normal corneal epithelial phenotype or the presence of conjunctival epithelial cells.7,8 Clinical signs such as corneal vascularization, loss of the corneal reflex, corneal haze, or an absence of the palisades of Vogt may be indicative but are not pathognomonic of LSCD. Examination under cobalt blue light using fluorescein staining can significantly aid the diagnosis of LSCD by the staining pattern of the conjunctival epithelium.7 However, biomicroscopic evaluation is not a diagnostic test, and ideally, the diagnosis is confirmed after the phenotypic analysis of corneal surface cells using impression cytology or in vivo confocal microscopy.7,8

Management of this pathology is, in general, challenging and depends on extent, severity, and laterality of the disease. In the case of partial LSCD, a conservative approach or a sequential sector conjunctival epitheliectomy may be enough to resolve the issues,9 whereas in total LSCD with central corneal involvement, a limbal stem cell graft remains the treatment of choice.10 The past decade has seen an expansion in the options for limbal stem cell grafting from simple limbal transfer to more complex cultured grafts.11

Despite the increasing interest in the field, there is still no standardized way for evaluating the success of the diverse treatment options in LSCD. Assessment and interpretation of the outcomes after treatment is mostly subjective based on individual interpretation or on poorly defined outcome parameters because there are few better options. This introduces high biases and limits comparison between groups and techniques. Even the quality of the source images can be an issue when asking independent graders to examine the images. In this study, we aimed to develop and validate a new and improved grading tool for assessing the severity of LSCD and the outcomes after treatment. In doing so, we focused on the ease of use and accessibility of the tool, with the aim of encouraging multiple specialists grading the images, increasing the reliability of the grade.


The study was approved by the Antwerp University Hospital Ethical Committee and adhered to the tenets of the Declaration of Helsinki. All participants gave written informed consent to participate after the nature and possible consequences of the study were explained.

Development of the Grading Tool

Software Design of the Tool

The “Vascularization, Haze, and Integrity” (VaHI) tool is made up of 2 software parts that address different challenges. The VaHI backend stores all the VaHI data and provides an application programming interface to create, read, update, and delete data. The VaHI frontend provides a user interface that presents these data and allows users to rate eyes and submit those ratings, and administrators to add users, add eyes, and download data. The code is freely available and is hosted on a GitHub repository at The code is published under a Massachusetts Institute of Technology (MIT) license allowing any person to use, copy, merge, publish, distribute, or sublicense it without restriction.

Grading Parameter Selection

We selected 3 parameters, “superficial corneal vascularisation,” “corneal haze,” and “epithelial integrity,” which are commonly used in the clinical diagnosis of LSCD,8 for integration into the grading tool. For each parameter, reference images with different severity grades were selected to be presented alongside the current image to be graded (Figs. 1A–C) (normal = grade 0; mild stage = grade 1; moderate stage = grade 2; and severe stage = grade 3).

Standardized legends of corneal vascularization (A), haze (B), and integrity (C).

The severity grades range from 0 to 3 for all parameters, using the following descriptions:

  1. Superficial corneal vascularization: Grade 0: no superficial vascularization, clear cornea; Grade 1: vessels cover at most 30% of the surface zone; Grade 2: vessels cover 30% to 70% of the surface zone; and Grade 3: vessels cover 70% to 100% of the zone (Fig. 1A).
  2. Corneal haze: Grade 0: Clear cornea, no haze; Grade 1: mild haze; iris details easily visible; Grade 2: moderate haze, iris details visible with difficulty; and Grade 3: severe haze, no iris details visible (Fig. 1B).
  3. Epithelial integrity: Grade 0: fluo negative; Grade 1: stippling fluorescein staining; Grade 2: whorl-like epitheliopathy; and Grade 3: epithelial defect(s) (Fig. 1C).


A grid was designed to divide the cornea into 13 different zones, distributed in 3 concentric annuli. The outer circle of the grid represents the corneal limbus. The inner circle represents the central cornea (4-mm chord approximately), and the middle circle is located in the middle of the inner and outer circle. A central circle and 2 corneal annuli are because of this separation. The outer annulus is divided in 8 equal zones, whereas the middle annulus is divided in 4 equal zones. In a cornea with a width of 12 mm, the inner circle represents a 4 mm diameter, whereas the middle and outer annuli are 2 mm wide. The grid is manually placed onto the eye to scale the photograph as explained below. An illustration of the grid is displayed in Figure 2B.

Illustration of grading a photograph: original photograph of the eye before grading (A), same eye after manually placing of the grid (B), grading of superficial corneal vascularization (C), corneal haze (D), and epithelial integrity (E) of the same eye.

Use of the Tool

Detailed user guidelines can be found in the Supplemental Digital Content 1 (see VaHI user guidelines, Briefly, the tool can be accessed by “administrators” and “users” at Administrators can upload and scale corneal photographs, add users, and download results. After receiving an invite code issued by the administrator, users can login into the tool and grade the uploaded photographs.

An eye can be graded by clicking (or touching for a touchscreen or cellular device) on one of the zones until the desired score (0–3) appears and subsequently until all zones are graded. An overall score ranging from 0 (13 zones × 0 score/zone) to 39 (13 zones × 3 score/zone) for vascularization and haze, and ranging from 0 to 3 for integrity is indicated on the right bottom of the photograph. By clicking “next,” the user is able to grade the following parameter on a similar way (0.2). Reference legends are visible next to each photograph. Scores can be saved at all times, and the grader is able to continue his/her grading later on. The results of all graders are accessible for the administrators and can be downloaded into an excel file.

Validation of the Grid

Ocular Surface Photographs

Thirty eyes with varying degrees of LSCD were included in the study. In every clinical case, 2 ocular surface photographs were selected: one under white light and one under cobalt blue illumination after instillation of fluorescein (Fig. 2). The white light images were used to grade both corneal vascularization and haze, whereas the cobalt blue images were used to grade corneal integrity. All images were obtained from the department of ophthalmology of the Antwerp University Hospital and were masked to the identity and clinical history of the patients.

Validation of the Grading Scale

Three groups of 3 graders were recruited to examine all of the images. Group A included ophthalmologists with an expertise in corneal ophthalmology, group B included general ophthalmologists or ophthalmologists with expertise other than cornea, and group C included nonclinicians. Group C ultimately consisted of a physicist and 2 biomedical researchers. All graders were asked to grade the photographs using the open-source grading tool. The grading was then repeated after an interval of at least 3 weeks by the same group of graders.

Statistical Analysis

Statistical analysis was performed using SPSS version 25 (IBM Corp. Released 2017, Armonk, NY: IBM Corp) and R version 3.4.4 (R core team 2018, Vienna, Austria).

The intergrader repeatability (agreement among graders) was assessed in each subgroup (group A–C) and in the total group of graders (=overall intergrader agreement). Agreement between scorings per zone was assessed using Fleiss weighted kappa coefficients to account for the ranking in the ordinal scores (0–3). The intragrader repeatability (agreement or consistency of grading by a single rater) was performed by comparing the scores of each individual grader at different time points with an interval of at least 3 weeks. Agreement was computed for the subscores per zone with weighted kappa coefficients. Coefficients of graders are summarized per group by mean. For the overall scores of vascularization and haze, concurrent interrater and intrarater reliability (IRR) coefficients were computed with grader as random effect as proposed by Eliasziw et al.12 In this way, the scoring of both time intervals that are used in the computation of the intergrader reliability and a global scoring of intragrader reliability, over all graders, can be determined. Reliability coefficients were evaluated per subgroup (group A–C) and in the total group of graders (=overall repeatability).


All 9 graders completed both grading sessions, yielding a total data set of 1620 grading estimates (9 graders × 90 graded photographs × 2 sessions).

Intergrader Reliability

The overall reliability coefficients for “superficial corneal vascularisation,” “corneal haze,” and “epithelial integrity” were 0.78, 0.61, and 0.42, respectively, indicating good agreement for the first 2 parameters and moderate agreement for integrity. When evaluating the intergrader reliability in the subgroups, all groups had good agreement for the variable “superficial corneal vascularisation,” with the highest reliability coefficient in group A (IRR = 0.80) and the lowest in group C (IRR = 0.72). When assessing the variables “corneal haze” and “epithelial integrity,” good to moderate agreement was observed in groups A and B but group C scored remarkably lower. All intergrader reliability coefficients per group are listed in Table 1. The weighted kappa coefficients per zone are available in Supplemental Digital Content 2 (see Supplemental Table 1,

Intergrader and Intragrader Reliability Coefficients by Means of IRR or Weighted Kappa, When Evaluating Vascularization (V), Haze (H), and Integrity (I) Using the VaHI Grading Tool on 30 Different Images in 2 Time points

Intragrader Agreement

The overall intragrader agreement coefficient was 0.92 for superficial corneal vascularization, 0.83 for epithelial haze, and 0.66 for epithelial integrity. When analyzing the subgroups, very good agreement was found for the parameters vascularization and haze and good agreement for the parameter integrity, in all of the subgroups. The overall coefficients and the coefficients per group are listed in Table 1. The intragrader reliability coefficients per zone and per grader are available in Supplemental Digital Content 3 (see Supplemental Table 2,

Illustrative Case

A 54-year old male patient was offered a cultivated limbal stem cell transplantation for the treatment of severe LSCD after a chemical burn, 25 years earlier. For illustrative purposes, his preoperative photographs were graded using the VaHI tool by an experienced corneal specialist together with his 6- and 12-month post-op photographs (Fig. 3). As explained before, the maximum severity of vascularization and haze would be represented by 39 points, that is, if a grader would rate the 13 sectors of an eye with the maximum severity (grade 3); 13 sectors × 3 grade/sector = 39. On the other hand, maximum severity of integrity would be 3 because this parameter is not evaluated by sectors but just by a single overall image rating. In this case, the 6-month postoperative scores indicated a clear decrease of corneal vessels (score from 33/39 to 7/39) and an improvement of integrity (2/3–1/3), indicative of an early successful outcome. The scores on epithelial haze improved modestly. Although the slit lamp-changes did not demonstrate much of evolution at his 12-month follow-up period, the vascularization score increased from 7 to 15 on 39 but remained limited to the outer zone. The haze and integrity scores remained stable.

A limbal stem cell deficient eye graded by the Vascularization, Haze, and Integrity grading tool, indicating progression of the vessels over time. On the left top corner of every image, a score indicates the severity of the condition under analysis, the higher the score the higher severity.


One of the challenges in clinical limbal stem cell research is the difficult comparison between techniques and groups because of the high number of variables that are used. Recently, the Limbal Stem Cell Deficiency Working Group published a global consensus on the definition, classification, staging, and diagnosis of LSCD, which is an important step in allowing more universal communication.7 However, a standardized set of parameters to define the outcomes after treatment is not yet defined. Some groups predefine a set of outcome parameters or grading scores to reduce bias, but most of these measures are unvalidated.13–18 As long as there is no standardized grading method that is universally in use, direct comparison remains challenging. In addition, the quality of images obtained can be variable, particularly in multicentric trials, so clinical trial endpoint grading can be very difficult, even with highly trained independent graders.

Shortt et al19 were the first to propose and develop a validated grading system for quantification of the outcomes after limbal stem cell treatment, named the COASTL tool. The system consists of different grading plates with varying degrees of severity. Although their system was found to be reliable and repeatable, their plates are, up to now, not widely used in clinical practice.

In the current study, we developed and validated a new and improved method for grading stem cell deficient eyes and aimed to focus on the ease of accessibility to improve a widespread usage. We included the clinical parameters, “superficial corneal vascularisation,” “corneal haze,” and “epithelial integrity” for use in this tool. Although none of these parameters on its own are sufficient to make the diagnosis of LSCD, they are commonly observed in stem cell deficient eyes and therefore offer a way to an accessible grading method.

In contrast to the COASTL grading plates, our tool aimed to offer a more precise grading by dividing the cornea into different zones. The localization of the vessels is important during assessment of the extent of the pathology and plays a role determining anatomical success or failure. Subtle and nonactive peripheral neovascularization is a common observation in patients after LSCT13 and is not necessarily an indication of failure. On the other hand, extension of vessels toward the central cornea is likely to be sign of a failing treatment.

After agreement analysis, differences were found among the groups of graders and between the parameters. Ophthalmologists, including corneal clinicians, scored better than nonophthalmologists on all of the 3 parameters, indicating that some experience is required for interpretation of the images. Whether nonophthalmologists would grade better after formal training is yet to be seen. However, because LSCD is a rare pathology, the need to train graders for such niche is unclear.

The parameter “epithelial integrity” was scored less reliably than the parameters “superficial corneal vascularisation” and “corneal haze” by all groups. Fluorescein staining pattern has however more value in grading because it can differentiate between a healthy and an abnormal epithelium, whereas vascularization and corneal haze are not required diagnostic criteria in LSCD.7 Therefore, the lower integrity scores of this study indicate that the tool still has limitations. In addition, it demonstrates that the staining pattern is more subtle and difficult to grade, even for trained corneal specialists, and it may reflect the importance of the quality of the photographs. We believe that the use of more figure legends or a training could increase reliability of the integrity scores.

Furthermore, the tool is limited by the fact that there is still subjective assessment required and that the grading is based on clinical signs alone. The presence of corneal vascularization, haze, or epithelial defects only is not specific for LSCD, and there is also evidence that the degree of LSCD cannot be quantified based on clinical parameters exclusively.20 Adjunct diagnostic testing using in vivo confocal microscopy or impression cytology could strengthen the validity of this tool by providing a more accurate phenotypic analysis of the clinical results. However, these are more complex assessment techniques and are not available in all centers. Therefore, we decided not to include these techniques in our tool because it would impede widespread usage. By grading different zones and by eliminating the need for other assessment methods than ocular photographs, we tried to offer a novel, reliable, and validated grading option within its limitations and without the need for an expensive technique.

In this study, we did not determine a cutoff value for success or failure after treatment. The tool is, in particular, useful for evaluation of the progression over time. The main benefit of our tool is that it can be adapted to the grading parameters required by each trial protocol and present a free, very user-friendly clickable interface that reduces the tedium of repeated grading but is limited by the use of clinical signs alone. Should Limbal Stem Cell Deficiency Working Group expand their consensus agreement to define success parameters after treatment, this can be added to the tool with a view to simplify and harmonize outcome reporting in this rare and difficult condition.


1. Daniels JT, Dart JK, Tuft SJ, et al. Corneal stem cells in review. Wound Repair Regen. 2001;9:483–494.
2. Ebrahimi M, Taghi-Abadi E, Baharvand H. Limbal stem cells in review. J Ophthalmic Vis Res. 2009;4:40–58.
3. Tseng SC. Concept and application of limbal stem cells. Eye (Lond). 1989;3(pt 2):141–157.
4. Thoft RA, Friend J, Murphy HS. Ocular surface epithelium and corneal vascularization in rabbits. I. The role of wounding. Invest Ophthalmol Vis Sci. 1979;18:85–92.
5. Huang AJW, Tseng SCG. Corneal epithelial wound-healing in the absence of limbal epithelium. Invest Ophthalmol Vis Sci. 1991;32:96–105.
6. Dua HS, Azuara-Blanco A. Limbal stem cells of the corneal epithelium. Surv Ophthalmol. 2000;44:415–425.
7. Deng SX, Borderie V, Chan CC, et al. Global consensus on definition, classification, diagnosis, and staging of limbal stem cell deficiency. Cornea. 2019;38:364–375.
8. Le Q, Xu J, Deng SX. The diagnosis of limbal stem cell deficiency. Ocul Surf. 2018;16:58–69.
9. Dua HS. Sequential sector conjunctival epitheliectomy. In: Holland EJ, Mannis M, eds. Ocular Surface Disease, Medical and Surgical Management. New York, NY: Springer; 2002:168–174.
10. Dua HS, Miri A, Said DG. Contemporary limbal stem cell transplantation: a review. Clin Exp Ophthalmol. 2010;38:104–117.
11. Holland EJ. Management of limbal stem cell deficiency: a historical perspective, past, present, and future. Cornea. 2015;34(suppl 10):S9–S15.
12. Eliasziw M, Young SL, Woodbury MG, et al. Statistical methodology for the concurrent assessment of interrater and intrarater reliability: using goniometric measurements as an example. Phys Ther. 1994;74:777–788.
13. Pedrotti E, Passilongo M, Fasolo A, et al. Vivo confocal microscopy 1 year after autologous cultured limbal stem cell grafts. Ophthalmology. 2015;122:1660–1668.
14. Baradaran-Rafii A, Ebrahimi M, Kanavi MR, et al. Midterm outcomes of autologous cultivated limbal stem cell transplantation with or without penetrating keratoplasty. Cornea. 2010;29:502–509.
15. Kolli S, Ahmad S, Lako M, et al. Successful clinical implementation of corneal epithelial stem cell therapy for treatment of unilateral limbal stem cell deficiency. Stem Cells. 2010;28:597–610.
16. Bobba S, Chow S, Watson S, et al. Clinical outcomes of xeno-free expansion and transplantation of autologous ocular surface epithelial stem cells via contact lens delivery: a prospective case series. Stem Cell Res Ther. 2015;6:23.
17. Ramírez BE, Sánchez A, Herreras JM, et al. Stem cell therapy for corneal epithelium regeneration following good manufacturing and clinical procedures. Biomed Res Int. 2015;2015:4–10.
18. Rama P, Matuska S, Paganoni G, et al. Limbal stem-cell therapy and long-term corneal regeneration. N Engl J Med. 2010;363:147–155.
19. Shortt AJ, Bunce C, Levis HJ, et al. Three-year outcomes of cultured limbal epithelial allografts in aniridia and Stevens-Johnson syndrome evaluated using the clinical outcome assessment in surgical trials assessment tool. Stem Cells Transl Med. 2014;3:265–275.
20. Aravena C, Bozkurt K, Chuephanich P, et al. Classification of limbal stem cell deficiency using clinical and confocal grading. Cornea. 2019;38:1–7.

limbal stem cell deficiency; stem cells; cornea; grading tool; outcome measures

Supplemental Digital Content

Copyright © 2020 Wolters Kluwer Health, Inc. All rights reserved.