With the recent emphasis on quality assurance in medicine, there has been concomitant interest in assessing and maintaining physicians' competence. The College of Physicians and Surgeons of Ontario (CPSO), Ontario's medical licensing authority, has for many years assessed a random selection of the province's 26,000 physicians. 1 In most instances, the peer—assessors detect no deficiencies, or only minor ones (for example, improper record keeping). Apprised of these minor deficiencies, physicians can make corrective adjustments to their practice patterns, and reassessment many years later has shown persisting improvement. 2
On rare occasions, however, the peer—assessors raise serious concerns about the competence of a physician. As a result, the CPSO also sponsors a much more detailed assessment program, the Physician Review Program (PREP), 3 which may result in an individualized educational prescription in the form of remedial continuing medical education (CME). It was originally hoped that, after self-directed CME, physicians would demonstrate improvement upon repeat assessment. Indeed, this is true for many physicians with mild deficiencies. However, physicians with more serious incompetencies are less likely to significantly improve. As such, the CPSO agreed to support an intensive and prolonged educational intervention for five physicians with serious incompetencies identified at PREP. The results of the intervention were assessed by reappraisal at PREP. We report here the nature of the intervention and the results at reassessment.
THE PHYSICIAN REVIEW PROGRAM (PREP)
PREP, 3 an intensive one-day assessment, has used several assessment tools: multiple-choice questions; peer-observed encounters with standardized patients; peer-directed, chart-stimulated recall; and structured office oral examinations. The results of all tests are synthesized into a global composite score. Six levels of competency have been defined, validated in part using scores from “criterion” doctors (physicians in the community who have no known deficiency).
Categories I and II consist of physicians with no or minor deficiencies and include all criterion doctors. Categories III and IV comprise physicians with moderate to major difficulties; there are serious concerns with the competence of these physicians. Category V is constituted by physicians deemed unsafe to practice without direct supervision in an approved educational setting. Category VI physicians are deemed unsafe to practice in any setting.
THE REMEDIAL CME PROGRAM
In 1992, we identified five moderately to severely incompetent physicians (PREP III or IV) who were willing and able (by proximity) to participate in an intensive and prolonged remedial CME program. The physicians, all men, had graduated in 1970, 1964, 1951, 1948, and 1948; their ages were 50, 55, 69, 72, and 72 years, respectively. Three practiced in urban settings, and two in rural settings. All were referred from PREP after unsatisfactory reassessment following usual self-directed CME. All were in solo practice, and none was a member of the College of Family Physicians of Canada.
The remedial CME program, which used a combination of individualized and small-group learning, continued for three years, after which time the physicians were reassessed at PREP. One physician, the youngest, joined the small group only in the third year. One of the authors (JP) served as the small-group facilitator.
Each physician first met with the facilitator to review in detail the results of his PREP assessment. Strengths and weaknesses were outlined, and all physicians were encouraged to undertake an ongoing program of self-study aimed at improving areas of weakness.
The physicians and facilitator then met together as a small group for two hours twice a month. Following a problem-based learning format, the physicians used cases from their own practices as stimuli to discuss basic and clinical issues. The facilitator also brought cases from his practice, especially those that he was unable to resolve easily. The physicians discussed the physiological and psychosocial components of the cases. The group also discussed evidence-based journal articles relevant to these cases and to family practice in general. The physicians also kept track of specific problems they had had in their practices. Using these problems, each physician led the discussion several times over the three-year course. These meetings included a group discussion of the major issues derived from the problems, followed by questions and comments.
The intensive CME course extensively used role playing. Periodically, the physicians interviewed simulated patients, giving them the opportunity to apply new learning. At the end of each interview, the physician, the facilitator, the observer—physicians, and the simulated patient discussed the strengths and weaknesses of the interview. Although we followed no explicit blueprint, we made a conscious attempt to cover a broad curriculum of family practice.
THE POST-CME REASSESSMENT OF THE PHYSICIANS
The results of the post-CME reassessment are shown in Table 1. One physician improved significantly, one physician received an identical grade, two physicians decreased by one grade, and one physician decreased by two grades. Location of practice (rural or urban) did not seem important, but age may have been.
We learned several lessons. First, even severely incompetent physicians can improve, as did one of the five physicians. This physician was the youngest and most engaged in the educational process. He brought articles on topics he was unsure of, contributed cases that had challenged him in his practice, took the greatest risks in discussions, offered feedback, and willingly role-played with the simulated patients.
However, most of the physicians in this highly selected group showed no improvement or declined on retest. In general, these physicians were more passive, interested in learning but less active in participation, and more likely to bring straightforward, easily resolved cases to the group. Although they enjoyed the social aspect of the small group, they often had difficulty recognizing areas of weakness in learning situations and avoided engaging in critical self-evaluation.
These results are disappointing but informative. There could be several explanations for our lack of success. First, it could be that our educational intervention, personnel, or evaluation methods were suboptimal. But while we could have designed alternative interventions, the combination of individualized feedback and focused self-study, self-audit and chart review, simulated patient encounters and role play, critical literature review and small-group learning—all extending over a considerable length of time—should have captured many of the characteristics of CME demonstrated to be effective in other settings. 4,5 Moreover, the educational facilitator had broad experience as a content expert (practicing family medicine in Ontario for over 35 years) and as an educational process expert (as a professor in the department of family medicine at McMaster University for over 19 years, including wide experience as a facilitator in the undergraduate and family practice residency programs and as an expert in group process). The combination of content and process expertise should have optimized the program's chances for success. 6,7 The evaluation methods (those used in the PREP assessment) have been widely reported and validated with normal controls; they are well accepted and form a reasonable evaluation of physicians' incompetence. 3
Moreover, the CPSO had mandated the assessment and reassessment at PREP and had strongly encouraged the physicians' participation in the educational program. The involvement of the licensing authority, which has had beneficial impacts on other educational interventions, 8 should have ensured optimal compliance and motivation. In fact, all five physicians did fully comply with the educational process. (Nonetheless, this motivation to learn might be very different from that of a self-referred physician.)
Thus, even though the number of physicians in this project was necessarily small, we are forced to conclude that most severely incompetent physicians do not improve with intensive remedial CME, at least with the techniques we used and the biases inherent in the selection process. It is possible that different techniques, such as earlier intervention, more individual attention, and more intensive activities optimized for elderly physician learners, could have produced better results.
However, other possibilities for this lack of improvement are of greater concern. These physicians were, in general, older, and it is possible that their incompetence arose from early age-related cognitive decline, early organic dementia, severe mood disturbance, or other conditions associated with neuropsychological impairment. If it could be demonstrated that at least some physician incompetence arises from irreversible cognitive difficulty, and if cognitive difficulty precludes remediation, then such physicians could be spared the embarrassment and expense of extensive and futile remediation, and in certain cases, they could receive treatment or appropriate disability coverage. It is important to emphasize that age alone is not the defining variable, as other elderly physicians perform well in practice and at PREP.
The assurance and maintenance of physicians' competence are areas of increasing societal concern, and most agree that any competency and performance screening programs should be complemented by opportunities for corrective CME. Remedial CME in particular is frequently mentioned as an area of potential involvement for medical schools and academic health science centers. However, at least in the restricted circumstances described above, even extensive remedial CME did not lead to improved competence of severely incompetent physicians. Moreover, it seems overly facile to ascribe incompetence in these instances to lack of motivation or failure to attend regular CME in the past. There is an imperative to better understand the basis of physician incompetence, to develop better methods of educating incompetent physicians, and to better predict the success or failure of remedial CME in this setting.
1. McAuley R, Henderson HW. Results of the Peer Assessment Program of the College of Physicians and Surgeons of Ontario. Can Med Assoc J. 1984;131:557–61.
2. Norton P. Results of the PEER assessment program, personal communication, 1999.
3. Norman G, Davis D, Lamb S, Hanna E, Caulford P, Kaigas T. Competency assessment as part of a peer review program. JAMA. 1993;270:1046–51.
4. Anderson RC. The notion of schemata and the education enterprise. In: Anderson RC, Spiro RJ, Montague WE (eds). Schooling and the Acquisition of Knowledge. Hillsdale, NJ: Erlbaum, 1973.
5. Davis D, O'Brien MA, Freemantle N, Wolf FM, Masmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;282:867–74.
6. Schmidt H. Problem-based learning: rationale and description. Med Educ. 1983;17:11–6.
7. Barrows HS. How to design a problem-based curriculum. New York: Springer, 1985.
8. Gagnon R. ACMC Research in CME. Presented at Association of Canadian Medical Colleges Annual Meeting, Ottawa, Ontario, Canada, 1998.