Hess, Brian J.; Lynn, Lorna A.; Holmboe, Eric S.; Lipner, Rebecca S.
In 2002 the American Board of Medical Specialties (ABMS) developed a comprehensive assessment framework for practicing physicians called maintenance of certification (MOC) that integrates self-evaluation of lifelong learning and practice performance with a secure examination. The American Board of Internal Medicine (ABIM) developed 15 Web-based practice improvement modules (PIMs) to help internists and subspecialists enrolled in MOC to evaluate their practice performance and guide them through a quality improvement (QI) process. For most PIMs, physicians collect data about their own practice using a combination of chart reviews, patient surveys, and a practice system survey to create an interactive performance report that physicians use to select targets for improvement.1
Several PIMs were developed to address the quality of communication. Good communication between physicians and patients is vital to quality care, especially for patients with chronic illnesses.2 Physician-to-physician communication is particularly vital for effective care coordination, especially for patients who see multiple specialists. Previous work has shown that physicians can provide valid and reliable assessment of their peers.3–5 However, less is known about the value of peer assessment to help consulting physicians evaluate and improve their processes for communicating with the physicians who refer patients to them. To this end, the ABIM developed the Communication with Referring Physicians PIM (CRP-PIM), which provides consulting physicians with feedback from a survey of referring physicians intended to help them understand gaps in the quality of consultative care they currently provide. This feedback is coupled with a practice system survey to understand the interaction between systems in which consultants work and their communication with referring physicians.
The purpose of this study was to assess the value of the CRP-PIM. Six research questions were addressed:
1. What type of consulting physicians elected to complete the CRP-PIM?
2. Are communication ratings from referring physicians reliable?
3. Are communication ratings associated with specific dimensions of communication processes?
4. Are communication ratings associated with characteristics of the consulting physicians (e.g., practice characteristics, gender, subspecialty, type and history of referring relationship)?
5. Are communication ratings associated with specific practice system features?
6. What communication processes or system features did consulting physicians most frequently target for improvement?
To complete the CRP-PIM, consultants asked physicians who regularly refer patients to them to complete a survey anonymously via Internet or phone. This survey asked about specific aspects of communication from the consultant’s practice that occurred during the prior 12 months (e.g., timeliness of contact with the consultant, communication about next steps, and helpfulness of the consultant’s office staff). Thirteen items using a six-point scale ranging from Never (1) to Always (6) comprised a measure of each consultant’s communication quality. Referring physicians also reported their gender, age, type of practice, consultant’s usual role in patient care (e.g., comanagement or episodic care), and duration of the consulting relationship. Consultants distributed the survey to referring physicians by e-mail, mail, or in person. A minimum of 10 referring physicians had to complete the survey, a sample size sufficient to generate reproducible results.3 Consultants were encouraged, however, to aim for a sample of 25, and advised to distribute twice that number of surveys, anticipating a 50% completion rate. Consultants could monitor the number of survey responses received by the ABIM.
Consultants also completed a survey about their practice system, which examined elements that have been shown to improve the quality of patient care.6 Information captured included number and type of staff in the consultant’s practice, percent of time spent in the practice, type of care provided, current modes of patient communication (e.g., use of telephone coverage, Web site/e-mails), methods for managing information (e.g., electronic data systems), and the overall quality culture of the practice. This survey underwent pilot and usability testing before it was implemented as part of eight condition-specific PIMs, and it has been completed by more than 6,000 physicians enrolled in the ABIM MOC program. Results of this survey help physicians identify what currently works in the system and what changes might be made to improve their communication with referring physicians.
When a sufficient number of surveys from referring physicians were completed, the ABIM sent each consultant a confidential, aggregated performance report that included a mean rating for each survey item across referring physicians and frequencies for demographic variables and practice system features. Consultants were provided instructions for selecting a target for improvement and developing a QI plan. After a consultant implemented the QI plan, he or she reported the results to the ABIM through free-text responses to five questions pertaining to implementation and results of their plan.
Participants were 803 consulting physicians who completed the CRP-PIM from 2006 to 2008 to fulfill the self-evaluation of practice performance requirement of the MOC program. Each consultant received data from a mean of 15.21 referring physicians (SD = 5.78); overall, 12,212 referring physician surveys were received. The ABIM’s database was used to obtain each consultant’s age, gender, certification status, practice characteristics, and performance ratings received from their program directors at the end of the third year of internal medicine residency training (available for only 611 consultants because of missing data and incomparable scales). Consultants granted the ABIM approval to use their data for research at the time they enrolled in the MOC program.
Descriptive statistics, correlation coefficients, t tests, and factor analysis were used to address the research questions.
The majority of consultants (94%) had 10-year, time-limited certificates in either internal medicine or a subspecialty of internal medicine; 89% had time-limited certificates in one or more subspecialty, with 18% in cardiovascular disease, 18% in critical care medicine, 16% in pulmonary disease, 15% in medical oncology, and 10% or less in other subspecialties or in internal medicine alone. Average age was 46 years (SD = 6.0), and 80% were men. Thirteen percent were in solo practice, and 51% were in single or multispecialty group practice; 20% reported working in hospital inpatient practice, and 16% worked in academic faculty practice. Consultants spent an average of 71% of their time in direct patient care, 7% in administration, 6% in research, and 2% in teaching. Compared with the population of all physicians certified by the ABIM, this sample has a higher percentage of subspecialists and fewer solo practitioners. Because the vast majority had time-limited certificates, the sample is not representative of older physicians with time-unlimited certificates who are not enrolled in MOC.
Most referring physicians (93%) reported sharing patients with the consultant for more than one year and 48% for at least five years; 87% had referred patients at least 10 times in the prior 12 months. Consultants’ most frequent role in caring for referring physicians’ patients was comanagement of a chronic or acute condition (65%), followed by episodic care for diagnosis or treatment (18%). Referring physicians’ average age was 47 years (SD = 3.9), and 76% were men.
Table 1 displays item means and standard deviations for 13 survey items representing two categories of communication processes—contact/communication with the consultant and consultant’s office staff. Item means were generally high; the average overall referring physician rating was 5.53 (SD = 0.23) on a six-point scale, with a range between 2.46 and 5.95 (note the three negatively worded items were reverse scored when computing the average overall rating). The item “Over the past 12 months, how often did this physician show respect for you as a colleague?” yielded the highest average rating (mean = 5.91, SD = 0.16), whereas the item “Over the past 12 months, how often did this physician initiate additional diagnostic or therapeutic actions without informing you first?” yielded the lowest average rating (mean = 2.20, SD = 0.91; whereas lower ratings for this item are more positive, the average is closer to the middle of the scale). The average rating for the contacting/communication with the consultant category was 5.53 (SD = 0.24), with a range between 2.46 and 5.94; the average rating for the consultant office staff category was 5.31 (SD = 0.36), with a range between 2.33 and 6.00.
Principle axis factor analysis was applied to support the two major categories of communication processes. Two factors explained 48% of the variance in item correlations. Table 1 displays each item’s factor loading. Items generally loaded high on their respective factor, whereas loadings on the other factor (not shown) were small (<0.08), indicating that the ratings are associated with two distinct dimensions. Correlation between the two category ratings was 0.41 (P < .001).
Generalizability theory was applied to the 13 survey items. The variance component for consultants was 0.04 (SE = 0.002). For norm-referenced score interpretation, the generalizability coefficient was 0.78 and the 95% confidence interval was small (±0.04), indicating that the reliability of the ratings received from referring physicians was good.
The consultants’ gender correlated with overall ratings (r = 0.10, P < .01); women received higher ratings. Although still high, referring physicians’ overall ratings of general internists were significantly lower than for subspecialists (t = 3.22, P < .01, d = 0.24), and comparison of subspecialty groups with sufficient numbers indicated that only consultants certified in cardiovascular disease were rated higher than those certified in pulmonary disease (t = 2.37, P < .05, d = 0.30). The Cohen d effect sizes suggest modest differences in ratings. Ratings were not significantly correlated with consultants’ age, practice setting, length of referral relationship, or with referring physicians’ practice size. However, ratings from the contacting/communication with the consultant category were significantly correlated with program director ratings of consultants’ humanistic qualities (r = 0.14, P < .001), professional attitudes and behavior (r = 0.13, P < .001), and overall clinical competence (r = 0.10, P < .01).
From the practice system review, most consultants (72%) were associated with their practice for 2 to 10 years, and only 6% for less than 1 year. Most (74%) indicated that their practice system provides effective 24/7 nonphysician staff telephone coverage for patients and referring physicians; these consultants received significantly higher office staff communication ratings from referring physicians (r = 0.10, P < .01). Forty-two percent of practices provided secure e-mail or a Web site for efficient communication and self-directed services for patients and/or referring physicians. Forty-seven percent used a system outside the medical record that tracks referrals to or from other physicians; these consultants also received significantly higher office staff communication ratings from referring physicians (r = 0.12, P < .01).
Consultants were required to work on either one communication process or one specific system feature where improvement was needed. Despite receiving generally high ratings from referring physicians, 45% of consultants selected communication processes and 55% selected practice system features. Common communication processes selected for improvement were timeliness of reporting test results to referring physicians (13%), helpfulness of office staff (11%), interpersonal skills of office staff (10%), consultants’ own interpersonal skills (10%), and helpfulness of the consultation in improving the referring physician’s diagnostic or therapeutic approach (10%). When consultants focused on the practice system, common improvement goals were to use e-mail for communication between patients and/or referring physicians and the practice (38%), to use a practice Web site for communication (25%), to hold regular clinical team meetings to plan and execute QI activities (20%), to develop a system to collect and respond to patient and physician complaints about services (19%), and to develop or improve 24/7 nonphysician telephone coverage (19%).
Although consultants were generally rated highly, 30 (3.7%) consultants were rated two or more standard deviations below the overall mean (or approximately 5 or lower on the 6-point scale). Compared with other consultants, those in this lower-performing group were more likely to be general internists (50% versus 21%) and practice in an emergency department (20% versus 3%), but less likely to practice in a hospital (23% versus 48%). They also had significantly lower program director ratings of humanistic qualities (t = 2.52, P < .05, d = 0.54) and professional attitudes and behavior (t = 2.30, P < .05, d = 0.48). Fewer consultants in this group reported using effective 24/7 nonphysician staff telephone coverage compared with other consultants (60% versus 74%). For QI, they selected improving staff courtesy and respect to referring physicians more frequently than other consultants (40% versus 9%), and they selected improving the clarity of instructions about next steps for patients more frequently than other consultants (43% versus 23%).
The purpose of this study was to assess the value of the CRP-PIM for physicians in the ABIM MOC program. The CRP-PIM was completed primarily by subspecialists in multispecialty group and hospital practices. The reliability of the peer ratings is comparable with that of previous research findings.3–5 Factor analysis supported two categories or dimensions of the construct measured. Consultants on average received high ratings from referring physicians; however, ratings of consultants’ communication were modestly associated with their gender, type of internal medicine subspecialty, and program director ratings obtained at the end of residency training. Consultants’ office staff communication ratings were associated with specific practice systems (i.e., use of nonphysician telephone coverage and referral tracking systems). These correlations lend some support to the validity of the assessment. Despite receiving generally high ratings, about half of the consultants selected specific communication processes for improvement. Providing normative feedback demonstrating performance relative to other physicians in similar practices could encourage further improvement.
Ratings may have been atypically high because the CRP-PIM was elective, but there was some variability, allowing for distinction between levels of performance. Whereas the 30 lowest-rated consultants had some distinct qualities, inference made from this distinction should be done with caution because this group still received relatively high ratings. Nonetheless, the purpose of the module was not to identify “problem physicians” but to encourage physicians to improve their practices regardless of baseline quality. Further study might consider how the CRP-PIM could identify consultants who need remediation. For example, physicians who do not meet a minimal performance standard could be required to undertake focused training and undergo another round of self-assessment to ensure adequate communication practices. The use of peer ratings for quality assurance is consistent with the Canadian licensing authority, which has adopted multisource feedback that includes communication skill, to assess physicians.7
These results should be interpreted with some care. The method in which consultants distributed the surveys to referring physicians is unknown, although research shows that the ratings are not substantially biased by the method of selection.3 The referring physician survey seems to have a ceiling effect, prohibiting meaningful interpretations at the upper end of the rating scale. Finally, this study only reported on what communication and system features consultants selected for improvement. However, preliminary analysis of data from 310 recent completers of the CRP-PIM, which contains a new postsurvey that asks about the consultants’ QI plans and what improvements were observed, indicates that most consultants (98%) made a change in their practice relevant to the measure they targeted, and most (94%) said that these changes led to improvements.
The CRP-PIM is a useful tool that allows consultants to apply quality measurement in their practices to identify areas in which they could improve their communication with referring physicians, with the ultimate goal of better coordination of patient care. A follow-up study is planned to determine how much of an improvement in specific targeted measures was observed, and to identify the facilitators and barriers that consultants experienced from their QI plans. Research is also needed to examine whether the CRP-PIM could be used longitudinally to better understand the value of peer feedback, self-assessment, and self-reflection on performance.
1 Duffy FD, Lynn LA, Didura H, et al. Self-assessment of practice performance: Development of the ABIM Practice Improvement Module (PIMSM). J Contin Educ Health Prof. 2008;28:38–46.
2 Davies E, Shaller D, Edgman-Levitan S, et al. Evaluating the use of a modified CAHPS® survey to support improvements in patient-centered care: Lessons from a quality improvement collaborative. Health Expect. 2008;11:160–176.
3 Ramsey PG, Weinrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA. 1993;269:1655–1660.
4 Lipner RS, Blank LL, Leas BF, Fortna GF. The value of patient and peer ratings in recertification. Acad Med. 2002;77(10 suppl):S64–S66.
5 Lockyer JM, Violato C. An examination of the appropriateness of using a common peer assessment instrument to assess physician skills across specialties. Acad Med. 2004;79(10 suppl):S5–S8.
6 Nelson E, Batalden P, Godfrey M. Quality by Design: A Clinical Microsystem Approach. San Francisco, Calif: Jossey-Bass; 2007.
7 Lockyer J, Violato C, Fidler H. Likelihood of change: A study assessing surgeon use of multisource feedback data. Teach Learn Med. 2003;15:168–174.