Institutional members access full text with Ovid®

Share this article on:

Reliability of Clinical Guideline Development Using Mail-Only versus In-Person Expert Panels

Washington, Donna L. MD, MPH*; Bernstein, Steven J. MD, MPH†‡; Kahan, James P. PhD§; Leape, Lucian L. MD∥; Kamberg, Caren J. MSPH¶; Shekelle, Paul G. MD, PhD***

Original Articles

Background. Clinical practice guidelines quickly become outdated. One reason they might not be updated as often as needed is the expense of collecting expert judgment regarding the evidence. The RAND-UCLA Appropriateness Method is one commonly used method for collecting expert opinion. We tested whether a less expensive, mail-only process could substitute for the standard in-person process normally used.

Methods. We performed a 4-way replication of the appropriateness panel process for coronary revascularization and hysterectomy, conducting 3 panels using the conventional in-person method and 1 panel entirely by mail. All indications were classified as inappropriate or not (to evaluate overuse), and coronary revascularization indications were classified as necessary or not (to evaluate underuse). Kappa statistics were calculated for the comparison in ratings from the 2 methods.

Results. Agreement beyond chance between the 2 panel methods ranged from moderate to substantial. The kappa statistic to detect overuse was 0.57 for coronary revascularization and 0.70 for hysterectomy. The kappa statistic to detect coronary revascularization underuse was 0.76. There were no cases in which coronary revascularization was considered inappropriate by 1 method, but necessary or appropriate by the other. Three of 636 (0.5%) hysterectomy cases were categorized as inappropriate by 1 method but appropriate by the other.

Conclusions. The reproducibility of the overuse and underuse assessments from the mail-only compared with the conventional in-person conduct of expert panels in this application was similar to the underlying reproducibility of the process. This suggests a potential role for updating guidelines using an expert judgment process conducted entirely through the mail.

Clinical practice guidelines are important and widely used tools for synthesizing clinical information, promoting desired practice patterns, and reducing healthcare costs. 1,2 They often accomplish this by offering explicit recommendations for reducing overuse and underuse of medical interventions, both of which can contribute to poor health outcomes. Evidence suggests, however, that clinical practice guidelines could quickly become outdated. 3 One reason they might not be updated as frequently as needed is that they are time-consuming and expensive to produce. To facilitate the development or updating of guidelines, it would be desirable to simplify the process and reduce its cost.

Most guideline development processes use mechanisms of some sort to incorporate expert opinion. 4 One commonly used mechanism for collecting and synthesizing expert opinion is the RAND-UCLA Appropriateness Method. 5,6 This method has been used to evaluate the appropriateness of performing a variety of medical and surgical interventions, 7 assign priority to patients requiring coronary revascularization, 8 and determine which low-acuity emergency department patients could have their care safely deferred. 9,10 This method incorporates an in-person group discussion about areas of clinical uncertainty and disagreement.

Convening a national or international panel of experts for face-to-face discussions is logistically complex, and increases the time and cost of producing criteria. One method for addressing these concerns is to conduct the process entirely through the mail. However, the impact of this modification is unknown. Our goal was to conduct a formal test of the reliability (reproducibility) of this method comparing feedback of round 1 ratings through mail with the standard in-person feedback of ratings and discussion.

From *VA Greater Los Angeles Healthcare System, Los Angeles, California.

From the †Veterans Affairs Ann Arbor Healthcare System, Ann Arbor, Michigan.

From the ‡University of Michigan, Ann Arbor, Michigan.

From §RAND Europe, Leiden, The Netherlands.

From ∥Harvard School of Public Health, Boston, Massachusetts.

From ¶RAND, Arlington, Virginia.

From **RAND, Santa Monica, California.

This research was performed by the Southern California Evidence-Based Practice Center at RAND under contract with the Agency for Health care Research and Quality (Contract No. HS07185–02). Dr. Washington is the recipient of a Research Associate Career Development Award from the Department of Veterans Affairs. Dr. Shekelle was a Senior Research Associate of the Department of Veterans Affairs at the time this research was conducted.

The authors of this article are responsible for its contents. No statement in this article should be construed as an official position of the Agency for Health Care Research and Quality, the U.S. Department of Health and Human Services, or of the Department of Veterans Affairs.

This research was performed by the Southern California Evidence-Based Practice Center at RAND under contract with the Agency for Healthcare Research and Quality (Contract No. HSO7185-02).

Address correspondence and reprint requests to: Donna L. Washington, MD, MPH, VA Greater Los Angeles Healthcare System, 11301 Wilshire Blvd., 111G, Los Angeles, CA 90073. E-mail

© 2003 Lippincott Williams & Wilkins, Inc.