Hauer, Karen E.; Teherani, Arianne; Kerr, Kathleen M.; O’Sullivan, Patricia S.; Irby, David M.
In contrast to traditional basic-science-oriented classroom coursework, the clinical experiences that dominate the last years of medical school challenge students to integrate and apply knowledge and skills in service to patients. To ensure that students have achieved minimum clinical competence, the majority of medical schools now conduct comprehensive standardized patient (SP) assessments.1 Because variability in supervision during clerkships impedes reliable identification of students with clinical skill deficits,2 the comprehensive assessment may be the first time students are systematically evaluated for their ability to integrate cognitive and communication skills during patient encounters.
Although SP examinations are now commonplace, the types of difficulties students manifest in comprehensive assessments have not been characterized. Better understanding of these problems would facilitate more effective remediation and curricular improvements. The purpose of this study is to characterize the problems students demonstrate in comprehensive SP assessments, the causes of these problems, and their amenability to remediation.
In a prior survey of curriculum deans,1 we identified persons responsible for remediation after their school’s comprehensive clinical skills assessment. A comprehensive assessment was defined as a multistation, cross-disciplinary exam outside a single clerkship involving SPs. We emailed potential subjects, inviting them to participate in an interview. Participants were asked to describe the types of problems students exhibited during the comprehensive assessment, remediation strategies, and prognosis.
We used grounded theory to analyze the data, beginning by generating open codes for three transcripts. Three investigators (K.E.H., A.T., K.M.K.) created codes, discussed and reconciled discrepancies, and then coded seven more transcripts. The subsequent 23 transcripts were each coded by two investigators, who compared results and discussed discrepancies until consensus was reached. We analyzed transcripts and collected data concurrently, using the constant comparative approach to determine when additional interviews were no longer yielding new information about themes. The University of California–San Francisco institutional review board approved the study.
We invited 53 of the 62 individuals named by curriculum deans to participate in an interview. One declined, two failed to respond, and one was excluded because the school’s exam did not meet study criteria. We interviewed remaining potential subjects, stopping at 33 interviews when we reached thematic saturation. Nine of the original 62 were not invited because study enrollment closed. Most participants were MDs (24, 73%); five were PhD faculty (15%), and four were other educational leaders (12%). Participants represented both public and private schools from all four Association of American Medical Colleges geographic regions.
We organize the results below into three types of problems: technique, cognitive, and noncognitive. Whereas the technique problems can be easily remediated, the cognitive and noncognitive problems are more challenging, and their causes and remediation are presented together after the problem descriptions.
Many participants reported that the comprehensive exam uncovered problems with history taking and physical exam techniques, such as not employing a structured approach to the history or not performing all required elements of a physical exam. Respondents were particularly dismayed by errors that contradicted foundational physical examination instruction:
Repeatedly, despite many warnings, they examine through the gown. They don’t appropriately drape the patient. They will sit up on the exam table to examine the back, and we’re horrified, saying, “I can’t believe the students did that.”
Such problems were frequently blamed on poor techniques role modeled in clerkships:
Faculty say, “it’s okay if you examine over the gown.” They get people that say the physical exam’s a complete waste of time. They’re getting a lot of sketchy education and a lot of mixed messages. And we see that play out in the exam.
Still, participants indicated that these simple technique deficits were straightforward to remediate. Respondents reported success providing students with a structure for the history, such as characterizing a chief complaint or eliciting the review of systems in a specific order. Precepted video review sessions were an effective forum for helping students recognize physical examination errors and for teaching correct techniques:
It’s easy to point out the problem when you have a student that doesn’t do the physical examination properly. We record our encounters and you can tell the students, “Well, here’s how you did it. Here’s how I would recommend doing it so that you can more likely get the answer that you’re looking for.”
When discussing discrete technique problems, respondents did not question student ability to learn the skills. In fact, many stated that students often knew what they should have done; they just failed to show their skills in the exam. Conversely, problems rooted in individual student characteristics, such as depth of knowledge, clinical reasoning ability, personality style, or professionalism, were felt to be extremely difficult to remediate. Here, deficits were so extensive or ingrained that student capacity to master or display the skills needed to pass the exam were unclear. It was not that the student had not learned the proper technique for listening to the heart; rather, the student did not know that a heart exam was called for (clinical reasoning or clinical knowledge deficit) or did not care that a heart exam needed to be done (professionalism). Respondent characterizations of these deeper problems (which we group into cognitive and noncognitive categories), their causes, and remediation issues are presented below.
Participants perceived that some students with problems in history taking or physical examination harbored underlying data-gathering, clinical reasoning, or knowledge deficits. These problems often manifested as inability to focus the encounter appropriately.
Many low-scoring students focused prematurely, failing to ask open-ended questions or adequately characterize the chief complaint. Some students failed to appreciate that the history and physical examination served to reveal pertinent negatives needed to rule out important diagnoses:
We’re really looking at the process of excluding diagnoses. We see a lot of students pick up on a few positive findings, and they’ll just go down that road, and they won’t explore other possibilities.
Respondents also observed students being too focused on the history of present illness, omitting or incompletely exploring the pertinent past medical, social, or family history, particularly as they related to the chief complaint.
In contrast, some poorly performing students were unable to direct the history or select relevant systems to examine. One participant described the resulting time-management problems:
They will go by rote through this litany list from head to toe, and they run out of time, so they can’t get to the physical exam because they have spent so much time on the history.
Noncognitive problems related to communication and professionalism contributed to the sometimes interrelated issues of detachment, poor insight, lack of empathy, or resistance to the examination process.
Poor communication was visible in both verbal and nonverbal behaviors as emotional distance. Some students failed to elicit the patient’s perspective on the illness or to determine the impact of psychosocial factors. These students treated SPs as symptoms or diagnoses rather than as people with feelings and concerns. A common professionalism problem was not showing respect for the exam process. Here, students dressed or spoke too casually, chewed gum, or arrived late.
Faculty described egregious problems in the comprehensive assessment as stemming from personality disorders or poor insight. These problems had been recognized but had not necessarily been documented previously. As one participant described:
It’s the character pathology. It’s the arrogant student. These are the students who all the ward clerks know and hate. The secretaries in the dean’s office dread their coming.
A small number of students denied their own poor performance, blaming external factors such as the testing environment rather than skill deficiencies. The artificiality of the SP examination required suspension of belief, a challenge that some students resisted. Our respondents doubted these students’ claims that they performed differently with actual patients.
Participants felt that in some cases the testing environment contributed to performance problems. Because of time constraints and perceived time pressure, some students consciously or inadvertently allocated time for history taking at the expense of physical examination, or they prioritized data gathering over communication style. One participant described: “They’ve got to get through their checklist, so they don’t get that personal attachment to the patient.”
Participants perceived that some students misinterpreted the expectation to be focused and omitted too much data gathering. Other students understood the comprehensive assessment to be a test of diagnostic acumen, rather than of the process of the encounter:
I have videotape after videotape of students who walk in and don’t really even need a patient. They see the opening scenario—shortness of breath—and they decide they know what that is. They think the diagnosis is the ultimate objective.
Faculty admitted that some students had trouble investing in a realistic physical examination with SPs “because they say, ‘Come on, I’m not going to find anything.’”
Some physical examination problems related to faulty knowledge or clinical reasoning—for instance, generating incorrect differential diagnoses or employing exam maneuvers not indicated for the patient’s problem. Even when they knew the presentation of common disorders (i.e., able to describe the signs and symptoms of a disease), these students were unable to work from a chief complaint to generate possible diagnoses. Students who knew a correct physical examination technique might be unable to apply it appropriately for a chief complaint:
They really had no idea what they should have done. They know the Cadillac version of the exam, but they don’t know what to pull out for a certain presenting complaint.
Poor clinical reasoning was in some cases attributed to inadequate knowledge. Some deficient students neglected to generate working differential diagnoses, and their encounters seemed disorganized:
Many, if not most of those students who look disorganized don’t have an organizational problem, they have a content problem. And because they don’t know what to ask they’re scattered.
Participants perceived that clinician role models contributed to cognitive and noncognitive problems. Preceptors could exacerbate the problem of premature closure by diagnosing on the basis of pattern recognition rather than modeling generation of differential diagnoses. Poor modeling of clinical skills on clerkships was cited as a cause of substandard communication. Participants felt that students’ strong patient-centered communication skills developed in foundational clinical skills courses had subsequently decayed. Respondents suspected that residents and attendings used a less patient-centered approach than clinical skills course instructors advocated.
History-taking and physical examination problems attributed to knowledge and clinical reasoning deficits were uniformly perceived to be more challenging to correct than technique issues. One participant explained, “If it’s disorganized enough that the student can’t figure out how to proceed with the physical exam,” then faculty cannot simply teach a technique or specific approach applicable to all patient presentations.
Inadequate knowledge was described as developing over years and requiring much time to correct. Participants varied in their beliefs regarding whether large knowledge gaps were correctable, but all agreed that the process was slow and involved extensive reading. Until knowledge improved, faculty found it impossible to determine whether the student could apply knowledge to cases.
The students who have major knowledge gaps are hard to remediate because they have a long way to go. You have to build on those knowledge gaps first, and then you have to see if everything else kind of clicks. Once you have that knowledge built up, are they able to access it?
Participants also found clinical reasoning deficiencies laborious to remediate. Whereas clinical experience could expose students to preceptors’ clinical shortcuts that were developmentally inappropriate for students, some participants named clinical preceptorships as helpful preparation for the clinical reasoning expectations of SP encounters:
Clinical reasoning is something that goes on, to a large extent, silently inside your head. It’s more of something that you might learn with mentorship over time.
Noncognitive problems were the most challenging to remediate. Participants perceived that the clinical years of medical school were a late time to address entrenched problems. One participant explained that, “You’re really trying to change more fundamental aspects of the individual.” Correcting communication and professionalism problems entailed major change to the environment or the individual. Problems traced to clinical supervisors’ poor interactions were difficult to isolate and improve.
Our students model what they see, and it’s not necessarily what we value and what we’re testing. What we’d like to do is complete the circle and remediate and change the hidden curriculum to be the valued curriculum.
Students with longstanding maladaptive interaction styles had often been recognized earlier in medical school, and the comprehensive assessment simply verified known but often undocumented difficulties. One participant recommended that the only solution would be to identify personality disorders during the medical school admissions process. Whereas some participants relied on preceptors for communication remediation, others admitted that they lacked dedicated communication remediation programs. Participants did report success correcting anxiety-related communication problems through practice with SPs.
Student insight was considered critical to the success of remediation efforts, and videotapes forced some students to confront their skill deficiencies in a way that scores alone did not. In particular, a precepted video review of the encounters illustrated problems that students had previously denied or not understood. Students who recognized the concerns found the sessions helpful, and subsequent performance improved. However, participants were pessimistic about students who persistently lacked insight:
If they don’t believe that they have those deficits, you’re never going to be able to remediate them.
In this study, educators most familiar with their schools’ comprehensive SP assessments described student performance problems and underlying deficits. Whereas simple technique errors were easily corrected, many history-taking and physical examination deficiencies were attributed to clinical reasoning or knowledge deficits. Communication skills deficiencies were linked to broader interpersonal and professionalism pathology.
Although the comprehensive assessments are “doing their job” by identifying students with significant problems, our findings also highlight the difficulties that arise when such deficiencies are discovered late in medical school. Modifying teaching strategies, working on the hidden curriculum, and earlier attention to interpersonal and communication problems would all help the situation.
Research supports our finding that knowledge and diagnostic reasoning are linked,3 as well as our participants’ observations that a hypothesis-driven style of questioning may be more effective for students than premature closure or exhaustive questioning.4 Our participants also attributed some physical examination errors to clinical synthesis deficiencies. Prior research showed that students who are able to demonstrate proper physical examination techniques in isolation may not be able to apply techniques appropriately in realistic encounters.5 It is possible that teaching clinical and communication skills in supervised clinical settings may improve clinical reasoning abilities.6,7
The informal or hidden curriculum needs to be addressed—particularly the way in which students’ attending and resident role models interact with patients. These interactions may be unprofessional and interfere with students’ learning.8 To counteract the decay in students’ skills during clerkships, faculty need formal and repeated education about the effects their cognitive and noncognitive behaviors have on trainees.
Personality disorders and poor insight figured prominently in descriptions of refractory student problems. Although few data exist on diagnosed personality disorders among medical students, a Thai study documented that 9% of a single class met criteria for personality disorders—a rate similar to that of the overall population.9 Attributes of personality disorders, such as lack of insight or motivation and poor interpersonal communication, were the student characteristics that our participants found most frustrating. Incorporation of reflection into clinical skills curricula may promote insight as trainees develop their clinical and communication skills. Deeper insight through intensive reflection facilitates adoption of new perspectives and critical analysis of one’s performance aimed at behavior change.10 Our participants’ observation that students who demonstrated insight and motivation to change improved during remediation is consistent with literature on the merits of reflection in challenging assumptions and prompting successful learning.11
Better orientation to exam expectations would also likely reduce the incidence of some poor performances, freeing remediators to focus on students with more extensive needs. Our findings suggest that students need reminders that, despite encounter time limits, the exam is not merely evaluating succinctness in either the cognitive or noncognitive domains. With the current exam structure, students may be left to discern for themselves that they will be assessed on and rewarded for demonstrating professionalism, engagement, and empathy.
This study is limited to the perceptions reported, although we did continue interviewing until reaching thematic saturation. Further, we do not have quantitative data about the frequency or severity of different problems.
Students scoring poorly in history taking, physical examination, or doctor–patient communication in a comprehensive assessment harbor a range of interrelated data-gathering, clinical reasoning, communication, and professionalism deficiencies. Students who perform poorly need to be assessed for underlying, significant deficits, most of which are multifaceted and do not have a quick fix. Early identification and characterization of concerns may facilitate timely remediation of underlying problems and facilitate acquisition of effective clinical skills.
The authors thank The Josiah Macy, Jr., Foundation and the participants.