Faculty and the Observation of Trainees’ Clinical Skills: Problems and Opportunities

Abstract

The clinical skills of medical interviewing, physical examination, and counseling remain vital to the effective care of patients, yet research continues to document serious deficiencies in clinical skills among students and residents. The most important method of evaluation is the direct observation of trainees performing these clinical skills. Standardized patients and other simulation technologies are important and reliable tools for teaching clinical skills and evaluating competence and will be incorporated in the near future as part of the United States Medical Licensing Examination. Standardized patients and simulation, however, cannot and should not replace the direct observation by faculty of trainees’ clinical skills with actual patients. Faculty are in the best position to document improvement over time and to certify trainees have attained sophisticated levels of skill in medical interviewing, physical examination, and counseling. Unfortunately, current evidence suggests significant deficiencies in faculty direct observation evaluation skills. The author outlines the nature of the problems in clinical skills and their evaluation by faculty and ends with recommendations to improve the current state of faculty skills in evaluation.

Recently the National Board of Medical Examiners (NBME), responsible for the administration of the three-part United States Medical Licensing Exam (USMLE), announced that they would institute a clinical skills exam (CSE) as a component of Part 2 of the USMLE for students in the medical school class of 2005.¹ The CSE will use ten to 12 different encounters with highly trained standardized patients (SPs) to test whether a fourth-year medical student has acquired a minimum level of competence in communication and physical examination skills. Although there has been substantial resistance from some medical organizations, a Harris poll found strong support among the public for such an examination.^2,3

However, the responsibility to certify competence in these clinical skills among residents in the United States will continue to fall solely on the residency program director and associated teaching faculty. Observation and documentation of the clinical skills of residents is required by the Accreditation Council for Graduate Medical Education as part of its new outcomes-based accreditation project.^4,5 Unlike Canada, no specialty in the United States includes a direct assessment of clinical skills as part of licensure or certification. The introduction of the USMLE CSE provides an opportunity to reflect on the states of the practice and, equally important, the evaluation of clinical skills among trainees.

In an era of rapidly advancing medical information and technology, how important are these clinical skills to the successful care of patients? If important, what is the quality of these skills among trainees and practicing physicians? With the availability of SPs and other simulation technology for evaluation, what role should physician–educators play in the evaluation of clinical skills?

I wrote this article to respond to these questions and to demonstrate that proficiency in clinical skills does matter, that the state of such skills among trainees and practicing physicians remains substandard, and that the main responsibility for teaching and evaluating these skills should remain with physician–educators. Medical educators must not abdicate this responsibility to SP and simulation; rather, such approaches should complement the teaching and evaluation of clinical skills.

The Importance of Clinical Skills

Despite the explosion of technological advances, the clinical skills of medical interviewing, physical examination, and counseling remain vital to the successful care of patients. Despite documented limitations,⁶ studies over the last several decades have continually reaffirmed that a well-conducted medical interview and physical examination are still important diagnostic tools available to clinicians. Hampton and colleagues⁷ demonstrated that the medical history produced the final diagnosis in the majority of patients, with laboratory investigation providing the final diagnosis in only one of 80 consultations. More recently, Peterson et al.⁸ demonstrated that among 80 patients presenting to a primary care clinic with a previously undiagnosed condition, the history produced the correct final diagnosis in 76% of the visits. In his textbook, McGee⁶ highlights the valuable predictive value of many commonly used interview instruments, such as the mini-mental status exam, in evaluating patients. Using autopsies of 400 patients as the standard, Kirch and Schaffi⁹ compared the accuracy of the history and physical examination with diagnostic imaging in producing the correct diagnosis. They found that the combination of history and physical examination was accurate in 70% of the cases, whereas diagnostic imaging produced a correct diagnosis in only 35% of them.⁹ Chimowitz et al.¹⁰ have demonstrated the importance of the bedside examination in the accurate diagnosis of neurological disorders.

Effective physician–patient communication has also been shown to improve health outcomes. Stewart¹¹ reviewed 21 studies involving physician–patient communication interventions and found that 16 studies reported positive effects on patient outcomes, including adherence. Levinson and colleagues¹² found that physicians with malpractice claims were less likely to use patient-centered interviewing skills than were physicians without malpractice claims. Another study found that patient satisfaction is significantly lower when physicians neglect psychosocial aspects of the interview.¹³ Finally, Bordage¹⁴ recently noted that poor data-collection skills were still a significant factor in diagnostic errors. In summary, medical interviewing, physical examination, and counseling remain the most important and effective diagnostic and therapeutic tools.

The State of Trainees’ Clinical Skills

Despite the growing body of literature demonstrating the importance of clinical skills, studies continue to document serious deficiencies. Although much has been learned about effective interviewing techniques, deficiencies in interviewing skills persist, and in the views of some may actually have declined.^15–19 Furthermore, communication skills do not appear to improve even after completion of residency training. In a study using unannounced SPs, Ramsey et al.²⁰ found that a group of primary care physicians asked only 59% of essential history items. Braddock and colleagues found that among 1,057 counseling sessions involving primary care physicians and surgeons, only 9% of encounters met basic criteria for effective informed decision making.²¹ Other studies have shown that physicians fail to elicit over half of patient complaints and that many of the public complaints about physicians relate to communication problems.^22–26 Another study highlighted the importance of patient nonverbal communication and how many physicians fail to recognize these important cues.²⁷

Errors are also common in physical examination skills. In a 1976 study, Weiner and Nathanson²⁸ documented numerous types of errors among trainees. Wray and Friedland²⁹ noted in a 1983 report that residents committed at least one physical examination error in 58% of the patients examined. Deficiencies in cardiac auscultatory skills among trainees were documented as early as the 1960s.^30,31 Mangione and colleagues^32,33 recently demonstrated that poor cardiac and pulmonary physical examination skills continue to plague students and residents. Other studies have reported poor skills by residents for many aspects of the basic physical exam.^34–37

As a result of these known deficiencies in the basic clinical skills of history taking, physical examination, and counseling, there has been a significant push by medical educators and accrediting agencies to reemphasize both the training and evaluation of clinical skills.^38–42 Again, in the recent Harris poll over 95% of respondents rated communication and diagnostic skills as very or extremely important.³ Without accurate evaluation of clinical skills, which can only be accomplished by direct observation, improvement in the clinical skills of physician trainees is unlikely.

Faculty and Clinical Skills

Lack of Direct Observation of Trainees by Faculty

Perhaps the biggest problem in the evaluation of clinical skills is the lack of observation by faculty of trainees. For decades, faculty have taken at face value the veracity of the history and physical examination presented on inpatient and outpatient rounds without ever watching the trainee actually perform them. Two of the most prominent physician–scientists and physician–educators of the 20th century, the late Alvan Feinstein and George Engel, strongly advocated direct observation of the history and physical examination skills of trainees over 30 years ago.^43,44 Dr. George Engel commented in a 1976 editorial:

Evidently it is not deemed necessary to assay students’ (and residents) clinical performance once they have entered the clinical years. Nor do clinical instructors more than occasionally show how they themselves elicit and check the reliability of clinical data. To a degree that is often at variance with their own professed scientific standards, attending staff all too often accept and use as the basis for discussion, if not recommendations, findings reported by students and housestaff without ever evaluating the reporter’s mastery of the clinical methods utilized or the reliability of the data obtained.⁴⁵

Kassebaum and Eaglen⁴⁶ recently lamented that little had changed among medical schools over the last 25 years. While a growing proportion of medical schools now use SPs for some aspect of assessment, students continue to receive little direct observation from faculty. They also noted that by 1998 only 48% of medical schools used a standardized patient assessment in the fourth year. In the recent field trial of the USMLE CSE, nearly 40% of the 858 participating medical students reported they had been observed by a faculty member performing a history and physical four or fewer times during medical school. More concerning was the finding that residents and not faculty completed the majority of student observations.³

The state of affairs in residents’ education is no better. Even in a recent trial of the mini–clinical evaluation exercise (mini-CEX) by the American Board of Internal Medicine, several of the participating programs had a difficult time getting teaching faculty to complete just four direct observations of each intern over the course of a year.⁴⁷ Two other studies in the early 1990s found faculty were not able to discriminate between the various dimensions of clinical competence. In fact, both studies suggested that faculty were only evaluating two dimensions of residents’ performance: case-based medical knowledge and interpersonal skills.^48,49 One of the hypothesized reasons for these findings in both studies was that faculty rarely, if ever, directly observed the performance of the resident they were evaluating.

Similarly, another study found poor correlation between ward evaluations by attending faculty and objective structured clinical examination scores by residents, suggesting that the ward evaluations were not based on direct observation of clinical skills.⁵⁰ Finally, a recent randomized controlled trial primarily designed to improve the specificity of written comments by faculty on a general medicine rotation found that residents reported they were infrequently observed performing a history or parts of the physical examination during their ward rotation.⁵¹ Noting this difficulty getting faculty to teach and observe, Johnson and Boohan⁴⁰ recently argued in frustration that the training and evaluation of clinical skills should not be “left to teaching hospitals.”

Quality of Faculty Observation Skills

Is the problem simply insufficient quantity of observation by faculty or is quality also suspect? Unfortunately, little data exist on the quality of faculty observation of clinical skills, but the available evidence suggests significant deficiencies exist. Noel and Herbers and their colleagues, in two important studies of the traditional complete CEX, found substantial deficiencies in the accuracy of faculty ratings.^52,53 They demonstrated that faculty failed to detect 68% of errors committed by a resident when observing a videotape scripted to depict marginal performance. Use of checklists prompting faculty to look for specific skills increased accuracy of error detection from 32% to 64%, but the checklist did not improve overall accuracy of ratings of competence. More disturbing was that 69% of faculty still rated the overall performance of a resident depicting marginal performance as satisfactory or superior. Also, showing faculty a brief informational videotape about the traditional CEX and its purpose failed to improve the quality of ratings.⁵²

Elliot and Hickam⁵⁴ also noted that faculty observers did not reliably evaluate up to 32% of physical examination skills, especially involving the head, neck, and abdomen, among medical students. Kalet et al. ⁵⁵ examined the reliability and validity of faculty observation of interviewing skills using videotapes of student performance on an objective structured clinical examination. They found that faculty were inconsistent in identifying the use of open-ended questions and empathy, and that the positive predictive value of faculty ratings for “adequate” interviewing skills was only 12%. Of the eight faculty raters involved in the study, six were instructors in the student medical-interviewing course.⁵⁵

Role of Faculty in the Evaluation of Clinical Skills

Given the shortcomings with faculty evaluations, should this responsibility be taken away from physician–educators? The technology of SPs has been in use and studied for over 30 years,^56–60 and the capability of simulators continue to improve at a rapid pace.⁶¹ Much data exist on the reliability and validity properties of SPs. The Medical Council of Canada and the Educational Commission for Foreign Medical Graduates include clinical skills examinations as an integral component of the licensure process.^62,63 As noted previously, the NBME and the Federation of State Medical Boards will incorporate an SP-based clinical skills examination in the USLME starting in 2005. The development of SPs to evaluate clinical skills is unquestionably a major advance in competency assessment, and rigorous SP training and scoring methods support the reliability requirements needed for high-stakes examinations^3,56–63

There are limitations, however, in the application of SP-based methods for teaching and evaluation along the continuum of education and clinical practice. First, this technology is expensive and is not readily available in all areas and at all times. Second, SPs should supplement and not replace observation with actual patients. Standardized patients provide mostly a cross-sectional view of skills and only through the observation of actual patients can educators get a full picture of longitudinal changes and growth in clinical skills among trainees. Third, because most assessment instruments used for standardized patient exercises favor completeness over efficiency, SPs may have less validity with more advanced trainees.^64–66 Finally, although the task for a trainee is often specified before even meeting the standardized patient, such information is often not available to a trainee or physician interacting with real patients.

A distinction must also be made between the performance of a trainee in a testing situation and a trainee’s performing when caring for actual patients. Performing denotes the ongoing and continuous interaction between a patient and physician, something SPs cannot evaluate effectively. The noted educator George Miller described this difference as the physician’s ability to “show how” (e.g., demonstrate that they can execute the clinical skill) versus what the physician “does” with actual patients who are experiencing real pain, fears, and uncertainty.⁶⁷ Standardized patients are unquestionably a powerful tool to assess whether a trainee can execute a skill (performance, or “show how”) and, in my view, a valid vehicle to test for minimum levels of competence in graduating medical students. However, given the diversity and range of patients seen during training, medical educators are in the best position to evaluate the ongoing and continuous care of patients by trainees performing with actual patients who are not following a predefined script.^67,68 Furthermore, direct observation provides a valuable template for meaningful feedback to reinforce strong clinical skills and correct deficiencies.⁶⁹

Medical educators, especially those involved with residencies, must also realize that tests of knowledge by outside organizations such the NBME do not and cannot replace the observation of clinical skills by faculty. Research has repeatedly demonstrated that a multiple-choice examination cannot attest to a trainee’s proficiency in clinical skills. The addition of the CSE can only assure that a medical student has attained a basic level of clinical skills sufficient to begin the next stage of residency. The CSE was designed mostly to detect serious deficiencies among students before they matriculate into their residencies and to provide meaningful information about clinical skills as a part of state licensure. Dr. Jordan Cohen of the Association of American Medical Colleges noted that the CSE helps to fulfill an obligation to the public that the profession is ensuring that graduates have acquired these skills as a result of their medical school training.⁷⁰ However, the CSE does not ensure that the student who enters residency will necessarily acquire the higher level of clinical skills and judgment required for independent practice.

Finally, evaluation is at the heart of professionalism for the medical educator. Medical educators have a moral and professional obligation to ensure that any trainee leaving their training program has attained a minimum level of clinical skills to care for patients safely, effectively, and compassionately. Medical educators should not wait for the results of a standardized CSE or other examinations to learn whether their trainees possess sufficient clinical skills. This responsibility cannot be abdicated to SPs, licensing boards, or computer simulators. Trainees recognize the importance of these clinical skills; trainees also recognize they are not always adequately prepared to care for patients after graduating from a residency program.^71,72 Furthermore, the majority of trainees want effective evaluation and feedback from their faculty.^73,74

Challenges and Recommendations

The major challenge that lies ahead for medical educators is how to ensure that the educators themselves possess strong clinical skills but also have the necessary skill to effectively observe, evaluate, and provide feedback to trainees’ regarding clinical skills. The available research suggests that there is much work to be done to improve medical educators’ evaluations clinical skills. Research in clinical teaching has demonstrated that teaching skills can be improved with faculty development that will result in positive outcomes for learners.^75–78 Thus there is good reason to believe that properly designed training can also lead to improvements not only in faculty evaluation skills, but also in the faculty’s own clinical skills.^38,78 One thing is clear: brief faculty interventions without periodic reinforcement will not produce meaningful changes.^51–53

There are many barriers facing medical educators in this task. Time and financial pressures on clinical faculty continue to increase while government support for graduate medical education continues to decline.^79,80 Many academic medical centers and medical schools face significant financial difficulties. Work-hour reforms in medical residencies, although long overdue, add further burdens to already overextended clinical faculty. These challenges cannot, however, become excuses not to improve the evaluation of clinical skills by faculty. The public has spoken and expects us to move forward and improve. Clinical skills are important to patients, and they are clearly important to physicians who desire to be successful professionals. Therefore, I propose four broad-based recommendations as a starting point:

1. The evaluation of clinical skills by faculty must be given higher priority by medical school deans, department chairs, and residency program directors. This emphasis must translate into meaningful policies that encourage and reward the observation of clinical skills by faculty. For example, precepting in clinical setting should be counted toward productivity. A greater emphasis on faculty development in clinical skills and evaluation will also be needed, and recent research demonstrates that such faculty development can be effective.⁷⁸
2. Medical schools and residency programs should consider developing education champions and core groups of clinician-educator faculty who serve as the experts for clinical skills teaching and evaluation. Evaluation champions could potentially promote change like their counterparts in quality of care. These champions and core groups, once trained, can serve as mentors for other teaching faculty and can be role models of effective direct observation behaviors.^81–84
3. Medical schools and residencies will need to develop systems to “evaluate their evaluations.” Although more direct observation is clearly desirable and will help to improve reliability, it does not ensure better validity and accuracy of direct observations. In essence, medical schools and residencies will need to measure the outcomes of their training of clinical skills.
4. Finally, national and regional organizations involved in certification, evaluation, and accreditation should provide meaningful resources for research and development of new faculty-training programs involving the evaluation of clinical skills. Few medical schools and fewer residencies will possess enough resources to develop such programs on their own.

In summary, improving the evaluation of clinical skills by faculty should be a high priority for medical schools and particularly residency programs. Residency may be the last real opportunity to ensure that the trainee has adequate clinical skills, especially because there are no clinical skills examinations for graduating residents. Therefore, the education community, in partnership with other stakeholders, should set a research agenda and develop mechanisms to provide meaningful support for these activities. We have over 25 years of data documenting the myriad of problems; it is time to act more aggressively and proactively as an educational community to correct these problems. Although the need for better evaluation of clinical skills by faculty will have to compete with other needs and interests, encouraging such competition is long overdue.

This work was supported in part by a grant from the Robert Wood Johnson Foundation. The author is indebted to Dr. Louis Pangaro of the Uniformed Services University of the Health Sciences for his conceptual advice and review, and to Dr. Herbert Chase of Yale University for his thoughtful review of the manuscript.