Item Banking: A Generational Change in Patient-Reported Outcome Measurement

Purpose. Patient-reported outcomes are traditionally measured with questionnaires and many have been developed to measure Vision-Related Activity Limitation (VRAL; visual disability or visual functioning), Symptoms, and Quality Of Life (QOL). These vary in quality and can be classified as First or Second Generation instruments. First generation instruments are characterized by simple summary scoring of ordinal responses, which precludes interval measurement. This problem is solved in second generation instruments where Rasch analysis is used to optimize psychometric properties. However, second generation instruments retain limitations; difficulties in comparing scores across instruments, limited applicability to populations and inability to adapt to change. A third generation approach to patient-reported outcomes measurement, item banking, can solve these problems. The aim of this project was to use Rasch analysis to calibrate all items from all instruments to form VRAL, Symptoms, and QOL Item Banks.

Methods. Six hundred twenty-four people on the waiting list for cataract surgery were recruited. Each participant completed, by self-administration, a number of the 19 instruments. A total of 353 items were calibrated using Rasch analysis (Winsteps v3.67). The psychometric properties of each item bank were optimized; items fitting the Rasch model were retained (Infit and Outfit range, 0.50 to 1.50).

Results. Items were sorted into the three traits; 226 tapped VRAL, 22 symptoms, and 60 QOL. Satisfactory measurement of each latent trait occurred with person separation of 8.11 for VRAL, 2.33 for Symptoms, and 3.20 for QOL. Rasch estimates of item difficulty were highly stable with an average standard error of 0.11 logits.

Conclusions. Item banks for the measurement of the latent traits of VRAL, symptoms, and QOL have been formed. New items can be added to enable evolution of measurement. Item banking facilitates accurate and precise measurement through computer adaptive testing. This approach provides common measurement scales, facilitating worldwide comparison of results.

Today, the practice of Optometry is accompanied by the obligation to provide treatments supported by an evidence base. This has been stressed for the treatment of glaucoma, for example, where large multi-center randomized controlled trials have been performed,¹ but it is also true across the spectrum of optometric interventions including the provision of spectacles, contact lenses, and visual training. The source of the evidence for the benefit of interventions is outcomes research. A number of end points may be used to study the effect of interventions and these are called outcome measures. In Optometry and Ophthalmology, outcome measures include survival (e.g., corneal graft), anatomic measures, physiological measures (e.g., intraocular pressure), optical quality, and visual performance. Like many others, I've long been interested in the impact of eye disease and its treatments on optical quality and visual performance.^2–4 Although this field is endlessly fascinating, we need to face the harsh reality that measures of optical quality of visual performance are only surrogate measures. Of course, what really matters is the outcome from the patient's point of view (after all it doesn't matter how much further down the visual acuity chart a patient can read with their new glasses, if they find them intolerable to wear).

Patient-Reported Outcomes

Any report coming directly from a patient concerning the outcome of an intervention is a patient-reported outcome (PRO). Usually these take the form of questionnaires and can be as simple as asking a patient whether they are satisfied with the outcome of the intervention up to large instruments with complicated scoring systems. The need for PROs should be self-evident given that outcomes research simply formally tests what occurs in clinical practice where a good practitioner would naturally ask their patients about the benefits of interventions. This is widely recognized across medicine, where the inclusion of PROs in outcomes research is often mandated by research funding bodies, ethics committees, third party payers, and regulatory agencies like the Food and Drug Administration.^5–8 Although the importance of measuring PROs has moved beyond question, the key to performing good outcomes research is to have good outcome measures, including PROs. Optometrists are experts in the measurement of visual performance and therefore understand the advantages logMAR visual acuity testing has over Snellen visual acuity testing and why the former is preferred as an outcome measure, but we may not be so expert in the measurement properties of questionnaires. Of course, questionnaires vary in their measurement properties just as other outcome measures do. The conduct of high quality outcomes research depends on an understanding of these measurement issues.

What is Being Measured?

A PRO can be designed to measure innumerable traits (concept represented by the questions in the PRO), so it is critical to determine what needs to be measured. In Optometry and Ophthalmology, the most commonly measured trait is visual disability. Visual disability is a person's reduced ability to perform tasks due to impairment of visual performance. This is also known as visual functioning (VF) or vision-related activity limitation (VRAL), which is the appropriate nomenclature to be in line with the World Health Organization International Classification of Functioning, Disability, and Health.⁹ The reason this trait is so commonly measured is because visual-related activity limitation is the traditional indication for cataract surgery and the majority of ophthalmic outcomes research has occurred for this procedure. Other traits that can be measured include satisfaction, ocular surface symptoms,¹⁰ pain,¹¹ optical/visual symptoms,¹² well being,¹³ depression¹⁴ and perhaps most importantly, quality of life (QOL).¹⁵ QOL is a holistic assessment of the effects of disease on the person and includes many dimensions such as a patient's physical, social and emotional wellbeing, spiritual, vocational, economic issues, convenience et cetera. Therefore, QOL instruments are essentially complex. Confusingly, investigators often mis-represent their instruments as a measure of QOL, when it only measures one or two dimensions; often VRAL only. Nevertheless, the trait under measurement will be evident from inspection of the questions asked. An activity limitation instrument will include questions like: how much difficulty do you have reading? A symptoms instrument may ask: how troubled are you by gritty eyes? When choosing a questionnaire, determining the trait that is sought to be measured and identifying an instrument that measures that trait are critical decisions. However, the quality of the questionnaire is also critical.

Questionnaire Quality

The development and validation of questionnaires, or instruments as they are called in the literature, is a long and complex process. The extent to which instruments have been developed and validated can be used to define their quality. We have previously published questionnaire quality assessment criteria in this journal.¹⁶ These quality assessment criteria build on previous authors' work and incorporate the latest advances in questionnaire development methodology. The field of questionnaire science is a rapidly evolving field just as Optometry is. One way to make sense of the progression in questionnaire technology is to consider instruments as belonging to different generations.

First Generation—Conventional Questionnaires

First Generation instruments dominate the questionnaires available to Optometry and Ophthalmology. These are characterized by conventional instrument development and validation (true score theory). These instruments use Likert or summary scoring of ordinal values applied to response categories. For example, “How much trouble do you have…not at all, (1) a little, (2) quite a bit, (3) a lot (4).” This scoring approach involves two assumptions regarding the response scale: it is assumed that the spacing between response categories is equidistant, and it is assumed that all questions have the same “value.” For example, questions on difficulty driving during the day and driving at night are afforded the same value yet driving at night is a more difficult task so should be weighted accordingly.¹⁷ Neither assumption is valid which makes the scoring non-linear. As a result, first generation instruments should not be considered as measures. This is important because it limits the ability of first generation instruments to detect differences or measure outcomes.^{18, 19}

First generation instruments are common and still widely used. Table 1 lists a number of VRAL instruments that are best defined as first generation instruments. These include many of the most commonly used instruments like the National Eye Institute Visual Functioning Questionnaire (NEI-VFQ)²⁰ and the VF-14.²¹ Other popular first generation instruments include the Ocular Surface Disease Index,²² the NEI Refractive QOL Questionnaire among others.²³

Table 1

Second Generation—Questionnaires Using Rasch Analysis

Second generation instruments use Rasch analysis to deal with the assumptions that corrupt the scoring method used in first generation instruments. Rasch analysis provides interval scaling with both persons and questions (called items in the PRO field) calibrated to the same scale (Fig. 1). Rasch analysis also provides greater insight into the psychometric properties of the instrument. Specifically, how well items fit to the latent trait being measured; how well the items discriminate the people (separation); and how well item difficulty targets person ability. These two benefits of Rasch analysis (scoring and insight into psychometric properties) have popularized its application in ophthalmic questionnaires. Several articles in this journal have described in detail the science behind Rasch analysis and its advantages.^{24, 25}

Figure 1

Second generation instruments fall into two categories: new instruments and legacy instruments. Rasch analysis can be applied in the development of new instruments. Examples of this include the QOL Impact of Refractive Correction,²⁶ the Contact Lens Impact on QOL,²⁷ the Ocular Comfort Index,²⁸ and the Veterans Affairs Low Vision VFQ-48.²⁹ Legacy instruments are first generation instruments, which have been re-engineered using Rasch analysis. However, the re-engineering process can be as simple as scoring using Rasch analysis³⁰ or may involve using Rasch analysis to optimize the psychometric properties of the instrument. Examples of second generation legacy instruments are the Impact of Visual Impairment,^31–33 the Catquest-9SF,³⁴ and the McMonnies Questionnaire.³⁵

Why Rasch Analysis?

Empirical evidence for the benefits afforded by Rasch analysis can be drawn from a number of recent studies. Garamendi et al.¹⁸ used the Refractive Status and Vision Profile instrument to try to discriminate between spectacle wearers seeking refractive surgery and those happily wearing spectacles on the basis of Refractive Status and Vision Profile QOL scores. They found that simply Rasch scaling all 42 items provided a 10% gain in precision, but also removing items that poorly fitted the underlying latent trait (QOL) resulted in an instrument with a 197% gain in precision for discriminating the groups. Similarly, Gothwal et al.¹⁹ used the VF-14 to measure cataract surgery outcomes and found that a Rasch scaled and revised version of the VF-14 could measure surgical outcome with a gain in precision of 148% compared with the original version. The studies demonstrate that by optimizing the psychometric properties of an instrument using Rasch analysis large gains in signal (reduction in noise) are possible facilitating more efficient measurement.

Recently, our group has embarked on a project to create second generation legacy instruments out of all existing VRAL instruments. The strategy used was to rescale the instruments using Rasch analysis and to optimize the psychometric properties of the instruments. Second generation instruments were successfully developed for the Visual Disability Assessment,³⁶ the Cataract TyPE Spec,³⁷ the Cataract Symptom Scale,³⁸ Catquest,³⁹ the VF-14,¹⁹ and the NEI-VFQ.⁴⁰ For a number of other instruments, the revision process was unsuccessful or the revised instrument was suboptimal. Examples included the Activities of Daily Vision Scale,⁴¹ the Vision Core Measure-1,⁴² Visual Function Index,⁴³ Visual Activities Questionnaire,⁴⁴ the Impact of Cataract Surgery,⁴⁴ Cataract Symptom Score,⁴⁵ Visual Function and QOL,⁴⁶ QOL and VF,⁴⁷ and the Distance Vision Scale.⁴⁸

This line of research illustrated that second generation (legacy) instruments have some limitations. Almost all have poor targeting of item difficulty to person ability; items are designed to target people less able than the population. Therefore, the addition of items to suit the more able is required. Poor targeting may not be a problem with de novo second generation instruments, at least initially. Nevertheless, a fundamental problem of questionnaires is that they are not adaptable. The origin of the targeting problem is in the shifting indication for cataract surgery; surgery is now performed at lower levels of disability that a decade ago.³⁴ Therefore, these questionnaires have become outdated. Second, although all of these instruments are calibrated onto an interval scale using logits as the unit of measurement, the scale is not identical for all instruments. The range of the latent trait over which each measures varies also, some suit the third world, others a more Western population, but no instrument suits all populations. Finally, in refining these instruments, items which did not tap the latent trait under measurement were discarded to remove noise from the measure. However, these items may be able to be used in a different instrument to tap another latent trait; but this possibility was lost in a Second Generation instrument.

Third Generation—Item Banking

All these problems can be addressed in the Third Generation approach to PRO measurement: item banking. An item bank is simply a list of items, far more than would be included in a single questionnaire, which are all calibrated for difficulty using Rasch analysis. Item banks are generated by drawing items from multiple instruments and including them in one large Rasch analysis. Once all items are calibrated onto a single scale they can be drawn on to make measurements. Items can be selected, either manually or by a computer algorithm to target the ability of the patients under test. Because all items are calibrated to a single scale, whether “easy” items are used to measure in a third world setting, or “difficult” items are used to measure an urban Western population, the scores are meaningful in the same terms. Item banks are adaptable, in that new items can be added simply by pilot testing the new items with already calibrated items and performing a Rasch analysis to calibrate the new items.⁴⁹ As with second generation instruments, item banks benefit from optimized psychometric properties, such as all items tapping the same latent trait. Therefore, item banks need to be developed for each latent trait. In the context of pooling items from multiple instruments, this may have an advantage whereby items discarded from a VRAL item bank may be able to be pooled to form another item bank (e.g., QOL).

Item banking of ability or QOL is occurring for other medical outcomes, e.g., spinal cord injury,⁵⁰ chronic pain,⁵¹ and depression⁵² all of which are current National Institutes of Health funded projects. Similarly, Massof et al.^53–55 has pioneered item banking of ability items in low vision. However, it is time item banking occurred for all PROs in Optometry and Ophthalmology, which is the aim of our current line of research.

The Vision-Related Item Banking Project—Aim

The aim was to calibrate all items from all existing VRAL instruments onto a single measurement scale to create a VRAL item bank. Our secondary aim was to identify items belonging to other latent traits, and develop item banks for these traits also.

METHODS

Instruments

Nineteen instruments were identified as including VRAL content. They are listed in Table 1. These instruments contained a total of 353 items.

Study Population

Because the majority of these instruments were developed for cataract surgery outcomes assessment, we confined our initial data collection to a cataract population. Participants were invited from the waiting list for cataract surgery at the Flinders Medical Centre, Adelaide, South Australia (average waiting period 3 to 4 months). Six hundred twenty-four participants were enrolled. Their demographic details are displayed in Table 2.

Table 2

Enrolled participants were mailed a pack containing a subset of the 19 questionnaires for self-administration. Questionnaires were presented in their native format using their native response scales. Completed questionnaires were returned via a self-addressed and prepaid envelope.

Rasch Analysis

The data were analyzed using the Andrich rating scale model⁵⁶ with Winsteps software (version 3.68).⁵⁷ The Rasch model is a probabilistic mathematical model, which provides estimates of person ability and item difficulty along a common measurement continuum, expressed in log-odd units (logits). For the VRAL, a positive item logit indicates that the item requires a lower level of ability than the average (i.e., the item is relatively easier). A positive logit for participants (person ability) suggests that the participant's visual disability is greater than the mean required level of ability for the items (i.e., the overall ability required for the task is greater than the ability that the participant possesses).

Rasch analysis was also used to optimize the psychometric properties of the item bank. First, the suitability of each item to be included in the item bank was assessed using fit statistics. A range for Infit and Outfit of 0.50 to 1.50 was considered acceptable. This is more lenient than has been proposed for questionnaires,¹⁶ but this position was deliberately taken to make the item bank inclusive of as much content as possible. Fit of all items to the construct was confirmed using principal components analysis of the residuals (observed minus expected scores).⁵⁸ The performance of the item bank was assessed in terms of measurement precision using standard error on the calibrations and overall person separation (and reliability) statistics as well as targeting of item difficulty to person ability (difference between person and item means in logits).

RESULTS

Rasch estimates of item difficulty were generated with an average standard error on the measure of 0.11 logits indicating a high level of stability across items. However, 21.2% of items mis-fit the model (measured something other than VRAL). There was contamination with other latent traits including symptoms and QOL. As a remedy, the items were organized into 3 item banks representing three latent traits using principal components analysis of the residuals, item fit, and face validity: VRAL; Symptoms (The impact of disease or interventions on sensation e.g., visual; blur, glare . . . or comfort; gritty eyes, pain . . .); and QOL (Holistic assessment of the effects of disease on the person, this includes multiple dimensions; physical, social, emotional wellbeing, concerns, convenience, independence, economic). These 3 item banks were then re-analyzed separately.

The VRAL item bank contained 231 items from 16 questionnaires. This was trimmed to 226 items to meet the fit statistic criteria of 0.50 to 1.50. The real person separation was 8.11 (reliability 0.99) and the standard error on the measure was 0.11 logits. Therefore, this was a highly precise measurement of VRAL in cataract. However, there was poor targeting of item difficulty to person ability (person mean −1.64 logits). This is illustrated in Fig. 2. Targeting could be improved with the addition of items, which target the more able.

Figure 2

The symptoms item bank contained 24 items from 8 questionnaires. This was trimmed to 22 items to meet the fit statistic criteria. The real person separation was 2.33 (reliability, 0.84). Again this represented precise measurement of symptoms. There was excellent targeting of item difficulty to person ability (person mean −0.56). This is illustrated in Fig. 3.

Figure 3

The QOL item bank contained 63 items from 7 questionnaires. This was trimmed to 60 items to meet the fit statistic criteria. The real person separation was 3.20 (reliability, 0.91). Again this represented precise measurement of QOL. There was poor targeting of items to persons (person mean −2.11). This is illustrated in Fig. 4.

Figure 4

DISCUSSION

The research presented herein represents “proof of concept” of item banking for vision-related latent traits. Pooling the items enables effective measurement of all three traits. The vision-related item banking project has identified a family of three latent traits for which items can be banked: VRAL, symptoms, and QOL. Each of these item banks contained sufficient items to measure each latent trait. However, each could benefit from the addition of content. The VRAL item bank could benefit from additional items to target the more able cataract patients. Both the symptoms and QOL item banks effectively contain residual items, which did not fit the VRAL item bank. Therefore both require the addition of content to become comprehensive measures. This is especially true of the QOL item bank for which the items were chiefly drawn from the Impact of Vision Impairment (emotional well being subscale) and NEI-VFQ (social-emotional items) instruments. However, the expansion of item banks is simple, by adding uncalibrated items to the bank and determining their calibration with Rasch analysis against the calibrated items already in the bank.⁴⁹

The calibration and banking of items enable more flexible implementation strategies like computer adaptive testing (Fig. 5). An “adaptive test” presents items that will most accurately target the individuals' visual ability. The test commences with the median item and every time the candidate answers a question, the computer re-estimates his or her ability. Based on the most recent, revised ability estimate, the computer selects the next item to be presented, such that the candidate will find the item challenging. This process provides highly accurate measurement of ability and the test can be continued until a desired level of precision is achieved. By tailoring the test to the individual, the problem of poor targeting of item difficulty to patient ability is eliminated.

Figure 5

Computer adaptive testing creates the possibility for electronic implementation for direct data capture by an internet portal with the potential for various platforms, e.g., iPhone app.⁵⁹ Our intention is to create online testing providing real-time scoring and recording of data. The platform could be designed to allow for more latent traits and other variables such as different eye diseases. Of course, item banking and computer adaptive testing need not be confined to cataract patients. Vision-related PRO measures should be developed to suit all eye diseases, all latent traits and patients from all countries of the world.

Our next step from the research presented herein was to re-format all the items on to a single response format for each latent trait; thus moving away from native questionnaire format. This removes redundancy involved in items with similar content and eliminates variance related to question format. We have done this for each of these three latent traits. We are now collecting data in different eye diseases across international settings. We are also expanding the content of the item banks by identifying new content at interviews and in focus groups. Over time, we hope this will lead us to the ultimate goal of comprehensive measurement of all vision-related PROs.

ACKNOWLEDGMENTS

My collaborators Vijaya Gothwal, Thomas Wright, and Ecosse Lamoureux contributed significantly to the work described herein. All the ophthalmology staff at Flinders Medical Centre consented to their patients being available to this project.

Konrad Pesudovs

Department of Optometry and Vision Science

NH & MRC Centre for Clinical Eye Research

Flinders Medical Centre

Bedford Park, South Australia 5042 Australia

e-mail: [e-mail]