Skip Navigation LinksHome > June 2011 - Volume 52 - Issue 6 > Agreement Between Patient- and Physician-completed Pediatric...
Journal of Pediatric Gastroenterology & Nutrition:
doi: 10.1097/MPG.0b013e3182099018
Original Articles: Gastroenterology

Agreement Between Patient- and Physician-completed Pediatric Ulcerative Colitis Activity Index Scores

Lee, Jessica J*; Colman, Ruben J*; Mitchell, Paul D; Atmadja, Melissa L*; Bousvaros, Athos*; Lightdale, Jenifer R*

Free Access
Article Outline
Collapse Box

Author Information

*Center for Inflammatory Bowel Disease, Division of Gastroenterology and Nutrition, USA

Clinical Research Program, Biostatistics Core, Children's Hospital Boston, Harvard Medical School, Boston, MA, USA.

Received 21 July, 2010

Accepted 1 December, 2010

Address correspondence and reprint requests to Jessica J. Lee, MD, MMSc, Division of Gastroenterology and Nutrition, Children's Hospital Boston, 300 Longwood Ave, Boston, MA 02115 (e-mail: jessica.lee@childrens.harvard.edu).

Supplemental digital content is available for this article. Direct URL citations appear in the printed text, and links to the digital files are provided in the HTML text of this article on the journal's Web site (www.jpgn.org).

Crohn's and Colitis Foundation of America Research Fellowship Award supported J.J.L. during the conduct of this research, and Dutch Digestive Foundation supported R.J.C..

The authors report no conflicts of interest.

Collapse Box

Abstract

Objectives: Currently validated ulcerative colitis (UC) activity measures are physician based, but incorporate patient reports of symptoms. We aimed to assess whether patient-completed Pediatric UC Activity Index (PUCAI) scores are comparable to those of physician scores.

Patients and Methods: We performed a single-center prospective study to assess agreement between patient- and physician-completed PUCAI scores. Seventy patients with UC (ages 4–29) representative of all of the disease activity categories (inactive, mild, moderate, and severe) in the currently published physician-completed scoring system were recruited. Agreement was analyzed for PUCAI scores both as continuous and categorical measures. To ascertain validity, we compared both patient- and physician-completed PUCAI scores with the physician global assessment and serum inflammatory markers.

Results: Patient- and physician-completed PUCAI summary scores were identical 49% of the time, were different but within the minimal clinically important difference (MCID) of 20 points 48% of the time, and were at or beyond the MCID only 3% of the time. In general, patients reported higher mean disease severity on their questionnaires than did their physicians, with a mean difference in PUCAI scores of 3 ± 8 (95% confidence interval 2%–5%). A categorical comparison of the 2 sets of questionnaires using the disease activity groups demonstrated perfect agreement for 60 (86%) pairs (kappa coefficient 0.78; 95% confidence interval 0.65%–0.90%). Both patient- and physician-completed PUCAI scores also correlated well with the physician global assessment and serum inflammatory markers.

Conclusions: Our data indicate strong agreement between PUCAI scores obtained directly from patients and those completed by physicians. Hence, a patient-based PUCAI could complement existing instruments in both clinical and research settings.

The Pediatric Ulcerative Colitis Activity Index (PUCAI) was developed and validated by Turner et al (1) in 2007 for noninvasive measurement of disease activity in children with UC. As a physician-administered index, the PUCAI has demonstrated high correlation with endoscopic appearance and physician's global assessment (PGA), and has been incorporated into various clinical trials. The PUCAI has also demonstrated superior performance in correlating disease activity compared with other established noninvasive disease activity indices, in both children and adults (2–4). The feasibility and validity of this index have also been confirmed in a multicenter pediatric cohort (2,5).

Although PUCAI is a physician-based index, every item must be answered using patient reports of symptoms and activity level. In turn, a key question is whether patients could complete the PUCAI themselves. The ability of patients with UC to monitor their own disease activity would increase patient autonomy, because they would be able to detect a change in disease activity and seek prompt medical attention when needed. Furthermore, sharing a common language about disease activity would improve communication between patients and health care providers. A patient-based disease activity monitor would be valuable in the current health care era of increased emphasis on personal electronic medical records and self-management.

Turner et al (5) recently addressed a similar issue by comparing the physician-completed PUCAI scores with the patient-completed questions from the bowel domain of a disease-specific health-related quality-of-life (HRQOL) questionnaire called the IMPACT (6), in addition to colonoscopic appearance scores, serum inflammatory markers, and Mayo scores (7). The present study found higher correlations between the physicians' assessment and all constructs of disease activity than with the patient's IMPACT score. The authors concluded that clinician assessments are superior to those of patients in evaluating disease activity, and suggest that physicians should administer the PUCAI. However, they also acknowledge that PUCAI and IMPACT are 2 different instruments with dissimilar scale ranges (3-point vs 5-point, respectively). To date, no studies have compared patient and physician disease activity assessments using the same scale.

We hypothesized that patient-completed PUCAI scores are comparable to those of physician scores. We asked patients to use a paper-based questionnaire featuring the PUCAI to calculate their own scores in tandem with asking physicians to calculate scores for the same patients using either a paper-based form or an electronic version of the index embedded in our hospital's electronic medical record. The aims of our prospective study were to determine the extent of agreement between patient- and physician-completed PUCAI scores, and to validate patient- and physician-completed PUCAI scores with other markers of disease severity, including PGA, erythrocyte sedimentation rate (ESR), and C-reactive protein (CRP).

Back to Top | Article Outline

PATIENTS AND METHODS

We performed a single-center prospective study to assess the agreement between a patient-completed PUCAI scoring system and the currently published physician-administered PUCAI instrument (see the online-only Appendix A at http://links.lww.com/MPG/A50).

Back to Top | Article Outline
Development of a Patient-completed PUCAI Scoring System

Because the original PUCAI was designed for physicians to complete, we developed a patient-based index that could be easily understood by patients and parents without prior medical training (see the online-only Appendix B at http://links.lww.com/MPG/A50). We were careful not to alter the content of the original PUCAI to ensure valid comparison between our patient- and the physician-completed scores. We included instructions from the original PUCAI User's Guide (1) to maintain the integrity of the questionnaire as originally intended. Our patient-based index was geared to a fourth-grade reading level using the Flesch-Kincaid Grade Level score (8,9). We also tested its readability using the Flesch Reading Ease Score (8), a 100-point scale based on the average number of words and syllables per sentence, in which higher scores indicate easier readability. Documents with a Flesch Reading Ease Score between 60 and 69 are considered standard or acceptable, and are easily understood by eighth and ninth graders. Our patient-completed PUCAI scored 79, demonstrating fairly easy readability. In addition to the items in the original PUCAI, we also collected information on the date and time of PUCAI completion, age and sex of the patient, as well as the person(s) who completed the form (ie, patient, parent, or both).

Back to Top | Article Outline
Patients

We prospectively recruited patients who were diagnosed as having UC between the ages of 1 to 18 years, based upon accepted endoscopic, histological, and radiographic criteria (10,11). We included both children and adults with UC because PUCAI also has been validated in adult patients (3). Patients with disease limited to the rectum or those who had undergone colectomy were excluded from the study because the currently validated PUCAI scoring system is not intended for use in these patients (1). For the present study, we did not recruit patients who were scheduled for colonoscopy to minimize confusion in answering questions about stool frequency and consistency due to cleanout. The Committee on Clinical Investigation at Children's Hospital Boston approved the study protocol.

Back to Top | Article Outline
Description of the Study Conduct

Upon arrival to either the outpatient clinic or on the day of admission to the inpatient ward, eligible patients were given an introductory letter describing the goals of the study, an option to opt out, and instructions for completing a patient-based PUCAI. We instructed patients who were 12 years and older to complete the form on their own, and parents to complete it for patients younger than 12 years old and/or developmentally delayed. We asked each patient to participate in the present study only once to obtain independent measurements. We were also cognizant of our hospital's practice to obtain and report PUCAI scores as part of clinical care and to evaluate disease progression on the inpatient wards. As such, we enrolled inpatients on the day of admission to ensure that patient answers were not influenced by the way clinicians administered the PUCAI questions on prior day(s).

On the same day that the patients were asked to complete a patient-based PUCAI, we asked gastroenterology physicians to complete a 4-point PGA (PGA: inactive, mild, moderate, and severe) of their patients' disease activity before completing the PUCAI, and the original version of PUCAI based on their clinical assessment (Appendix A). If patients were admitted to the inpatient ward in the evening or overnight, then we asked physicians to complete the PUCAI after seeing them the following morning. Both sets of PUCAI were completed within 24 hours of one another, and physicians were not aware of the wording of the patient-based PUCAI. If the PUCAI was completed by a trainee or a nurse practitioner, then we asked the responsible gastroenterology attending physician to review, edit if necessary, and initial the completed form. To clarify the questions as intended by Turner et al (1), we also embedded instructions from the User's Guide in the physician form. If blood was drawn on the same day as part of clinical care, then we collected ESR and CRP.

Back to Top | Article Outline
Data Analysis

The PUCAI comprises 6 multiple-choice questions, in which responses are assigned point values from 0 to 30, in multiples of 5 or 10, and summed for a total score ranging from 0 to 85. Scores are categorized into 1 of the following disease activity categories: inactive disease < 10, mild disease 10 to 34, moderate disease 35 to 64, and severe disease 65 to 85 (1). To achieve representativeness across the entire PUCAI scale, we prospectively recruited patients until we had at least 5 patients in each of 4 disease activity groups. The resulting sample size of 70 patients provided >95% power to detect a within-patient difference in patient- and physician-completed PUCAI scores of 20 points or greater, assuming a standard deviation of 8 points and a type I error rate of 5%. Because of the low incidence of moderate and severe disease in our study population and smaller sample size compared with other disease activity groups, we combined the moderate and severe disease groups into 1 category (moderate/severe).

Agreement was analyzed for PUCAI scores both as categorical and continuous measures. Categorical analysis proceeded by creating a 3 × 3 table to evaluate agreement between the patient- and physician-completed PUCAI scores. We calculated percent observed agreement for all of the groups combined and used the kappa statistic to measure the extent of agreement (12).

PUCAI as continuous score was analyzed by testing the null hypothesis of no difference between patient- and physician-completed PUCAI scores using a paired t test. Because the minimal clinically important difference (MCID) for the original PUCAI was defined as 20 points (1,2,13), we required a difference in mean paired PUCAI scores of at least 20 points as evidence that patients and physicians scored the PUCAI differently. Visual agreement between patient and physician PUCAI scores was assessed by serially plotting the difference between scores with reference lines for no difference and ±1 MCID. The strength of agreement was measured with the kappa statistic. Furthermore, we compared the 2 scores in each of the 6 items of PUCAI to determine the source of major discrepancy, if present.

Lastly, we compared the 2 PUCAI scores categorically with the PGA using a kappa statistic and continuously to serum inflammatory markers, ESR and CRP, using a Pearson correlation coefficient. All of the comparisons were made using 2-sided significance levels of P < 0.05. Analyses were performed using SAS version 9.2 (SAS Institute Inc, Cary, NC).

Back to Top | Article Outline

RESULTS

A total of 70 patients with UC, ages 4 to 29 years at the time of scoring, were included in the present study. Both sexes were equally represented. Seventy-nine percent of patients were minors (<18 years old), of whom just 23% were younger than age 12. Most patients completed the PUCAI on their own and/or with the help of parents before their doctor's visit. In all but 5 cases, the patient and physician questionnaires were completed on the same day. Additional patient characteristics are shown in Table 1.

Table 1
Table 1
Image Tools
Back to Top | Article Outline
Agreement of PUCAI Scores by Disease Activity Category

A comparison of the PUCAI disease activity categories from the 2 sets of questionnaires is shown in Table 2. There was perfect agreement for 60 (86%) pairs of patient/physician PUCAI questionnaires. In 9 of the 10 disagreements, the patient-completed PUCAI resulted in a higher disease severity category than the physician-completed PUCAI. The questionnaires for 6 of these 10 disagreements were completed with the assistance of a parent and/or completed 1 day before the physician-completed PUCAI. Among all of the patients, the kappa statistic demonstrated substantial concordance between the 2 sets of PUCAI questionnaires (kappa statistic 0.78; 95% confidence interval [CI] 0.65%–0.90%).

Table 2
Table 2
Image Tools
Back to Top | Article Outline
Agreement of PUCAI Scores as a Continuous Measure

Mean patient- and physician-completed PUCAI scores are shown in Table 3. Patients reported higher mean disease severity on their questionnaires than did their physicians, with a mean difference in PUCAI scores of 3 ± 8 (95% CI 2%–5%). Patient- and physician-completed PUCAI scores were identical 49% of the time, were different but within the MCID of 20 points 48% of the time, and were at or beyond the MCID only 3% of the time. As shown in Figure 1, the 2 PUCAI scores were within 20 points of one another in almost all of the cases. There were only 2 patients among 70 who had a score that was greater than an MCID of 20 points from his or her physician. When we carefully evaluated each of the 6 items comprising the PUCAI, the most frequent discrepancy between the patient- and physician-completed scores occurred for stool consistency (Fig. 1). However, overall the differences in the 2 scores were only 5 points, except for 1 patient. The largest score difference was observed in rectal bleeding, but the weight of each choice is 10 points for this item that is higher than others comprising the PUCAI.

Table 3
Table 3
Image Tools
Figure 1
Figure 1
Image Tools

Comparison of the PUCAI with other indicators of disease activity is shown in Table 4 and Figure 2. The patient-completed PUCAI scores showed moderate agreement with the PGA (kappa 0.57) whereas the physician-completed PUCAI showed substantial agreement (kappa 0.71). Both patient- and physician-completed PUCAI scores were moderately correlated with CRP (r = 0.5) and substantially correlated with ESR (r = 0.7).

Table 4
Table 4
Image Tools
Figure 2
Figure 2
Image Tools
Back to Top | Article Outline

DISCUSSION

The results of the present study demonstrate substantial agreement between physician-administered and patient-completed PUCAI scores. Because the information required to complete items on the PUCAI (abdominal pain, rectal bleeding, stool consistency of most stools, number of stools per 24 hours, nocturnal stools, and activity level) must be directly obtained from patients, we sought to determine whether patients could complete the index comparably to physicians, assuming the questions were written in lay language. Indeed, our results indicate that patient-completed PUCAI scores are similar to their physician scores.

In the present study, we analyzed agreement of PUCAI scores both as categorical and continuous measures. Overall, differences in patient and physician PUCAI scores were minimal. The patient-completed PUCAI was most concordant with that of the physician for the moderate/severe disease activity group and least in the inactive group; however, at 83%, agreement was still high for this latter group. We have theorized that patients with inactive disease may emphasize mild symptoms that are most troublesome to them, even though they may not have significant UC-related symptoms at the time of PUCAI completion. We, however, believe it to be reassuring that most concordance between the 2 scores is seen when patients exhibit moderate/severe disease and the need for intervention such as a change in treatment is greatest. We also demonstrated that the patient-completed PUCAI scores correlate well with other measures of disease activity, including the PGA, ESR and CRP. It is not surprising that the physician-completed PUCAI scores showed higher correlation with PGA than the patient scores because PGA is a physician-based measure that incorporates subjective assessment of overall disease severity. Reassuringly, correlations between PUCAI and other objective measures of disease activity, such as ESR and CRP, were almost identical for both patient- and physician-completed scores.

The present study results contrast with those of Turner et al that compared physician-completed PUCAI scores and patient-completed scores from the bowel domain of the IMPACT questionnaire. In that study, it appeared that physicians and patients assess disease differently and that physicians more accurately assess disease activity (7). We note that authors of the PUCAI versus the IMPACT study focused primarily on the “abdominal pain” variable and correlated it with other measurements of disease activity (colonoscopic score, Mayo score, CRP, albumin, ESR, hemoglobin, patient global assessments, and PGA of disease activity). Although the items in the bowel domain of the IMPACT cover similar clinical signs and symptoms as the PUCAI, significant subjectivity exists in the IMPACT questions as they intend to measure health-related quality of life. In the present study, we used the same index for both physicians and patients and expanded to address all 6 variables in the PUCAI.

Ours is the first study to report that patients as young as 12 years old can complete the lay version of PUCAI and provide disease activity scores comparable to physicians. However, we are not the first to evaluate the feasibility of measuring disease activity in UC using patient input (14–16). Although physicians may determine the next course of medical intervention and/or treatment based on disease activity, a validated scale should also capture patients' perception of disease activity to ensure compliance with therapy (15). Indeed, all of the items in the PUCAI belong to symptom domains that were previously shown to be frequent in UC flares and responsive to changes in disease activity (16). High correlation seen between physician and patient PUCAI scores indicate that the PUCAI is able to noninvasively capture disease activity information broadly. It would be interesting to assess whether the original PUCAI could be used in patients without translation, especially in adult patients.

A potential limitation of the present study is reduced generalizability because all of the patients were recruited from the same tertiary care institution. To minimize bias, we approached all of the patients with left-sided and extensive UC without preselection based on their prior medical history or disease activity. We have demonstrated the feasibility of completing the PUCAI by patients and their families, but it would be important to validate our findings in another, larger independent cohort. Another limitation is that we may have unintentionally altered the content of the PUCAI during the translation process. We chose to translate the PUCAI into a patient-oriented version because the majority of our patients are school-age children and adolescents, and the developers of the original PUCAI emphasized the importance of using this index by physicians (1,7). To minimize capturing inconsistent information from physicians, we asked multiple medically and nonmedically trained individuals to review our patient-based PUCAI before the start of the present study.

Our data suggest that patients can use a patient-based PUCAI at home, in the clinic, or in the inpatient ward to communicate their disease activity score with physicians. This instrument cannot replace physician's evaluation, but, when used properly, it could increase patients' awareness of their UC-related symptoms, increase efficiency in outpatient clinics and inpatient units, and serve as an invaluable tool in clinical trials to study disease activity and treatment response longitudinally. Furthermore, a patient-based PUCAI could be incorporated into an emerging personal electronic medical record system to follow disease activity remotely and provide tools for self-management. Accordingly, future research may include repeated measurements of patient-based PUCAI to track changes in disease activity.

In conclusion, we have shown that a patient-based UC disease activity measure may be used to obtain select clinical information directly from even young patients. Upon further validation, this tool will allow patients to become more actively involved in disease monitoring in both clinical and research settings.

Back to Top | Article Outline

Acknowledgments

The authors are grateful to Dr Anne Griffiths, Hospital for Sick Children, Toronto, Canada, for permission to use the PUCAI as described. We thank the patients with UC and their families at Children's Hospital Boston for making the present study possible. We also thank Richard J. Grand, MD, for kindly reviewing the article.

Back to Top | Article Outline

REFERENCES

1. Turner D, Otley AR, Mack D, et al. Development, validation, and evaluation of a pediatric ulcerative colitis activity index: a prospective multicenter study. Gastroenterology 2007; 133:423–432.

2. Turner D, Hyams J, Markowitz J, et al. Appraisal of the pediatric ulcerative colitis activity index (PUCAI). Inflamm Bowel Dis 2009; 15:1218–1223.

3. Turner D, Seow CH, Greenberg GR, et al. A systematic prospective comparison of noninvasive disease activity indices in ulcerative colitis. Clin Gastroenterol Hepatol 2009; 7:1081–1088.

4. Turner D, Walsh CM, Benchimol EI, et al. Severe paediatric ulcerative colitis: incidence, outcomes and optimal timing for second-line therapy. Gut 2008; 57:331–338.

5. Turner D, Mack D, Leleiko N, et al. Severe pediatric ulcerative colitis: a prospective multicenter study of outcomes and predictors of response. Gastroenterology 2010;138:2282–91.

6. Otley A, Smith C, Nicholas D, et al. The IMPACT questionnaire: a valid measure of health-related quality of life in pediatric inflammatory bowel disease. J Pediatr Gastroenterol Nutr 2002; 35:557–563.

7. Turner D, Griffiths AM, Mack D, et al. Assessing disease activity in ulcerative colitis: patients or their physicians? Inflamm Bowel Dis 2009; 16:651–656.

8. Flesch R. A new readability yardstick. J. Appl Psychol 1948; 32:221–233.

9. D'Alessandro DM, Kingsley P, Johnson-West J. The readability of pediatric patient education materials on the World Wide Web. Arch Pediatr Adolesc Med 2001; 155:807–812.

10. IBD Working Group of the European Society for Paediatric Gastroenterology, Hepatology and Nutrition. Inflammatory bowel disease in children and adolescents: recommendations for diagnosis—the Porto criteria. J Pediatr Gastroenterol Nutr 2005;41:1–7.

11. Silverberg MS, Satsangi J, Ahmad T, et al. Toward an integrated clinical, molecular and serological classification of inflammatory bowel disease: report of a Working Party of the 2005 Montreal World Congress of Gastroenterology. Can J Gastroenterol 2005; 19(Suppl A):5–36.

12. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33:159–174.

13. Turner D, Schunemann HJ, Griffith LE, et al. Using the entire cohort in the receiver operating characteristic analysis maximizes precision of the minimal important difference. J Clin Epidemiol 2009; 62:374–379.

14. Higgins PD, Schwartz M, Mapili J, et al. Is endoscopy necessary for the measurement of disease activity in ulcerative colitis? Am J Gastroenterol 2005; 100:355–361.

15. Higgins PD, Schwartz M, Mapili J, et al. Patient defined dichotomous end points for remission and clinical improvement in ulcerative colitis. Gut 2005; 54:782–788.

16. Joyce JC, Waljee AK, Khan T, et al. Identification of symptom domains in ulcerative colitis that occur frequently during flares and are responsive to changes in disease activity. Health Qual Life Outcomes 2008; 6:69.

Keywords:

disease activity; disease score; patient correlation; ulcerative colitis

Supplemental Digital Content

Back to Top | Article Outline

Copyright 2011 by ESPGHAN and NASPGHAN

Login

Article Tools

Images

Share

Search for Similar Articles
You may search for similar articles that contain these same keywords or you may modify the keyword list to augment your search.

Connect With Us

 

 

Twitter

twitter.com/JPGNonline

 

Visit JPGN.org on your smartphone. Scan this code (QR reader app required) with your phone and be taken directly to the site.