Secondary Logo

Journal Logo

EDUCATING PHYSICIANS: ESSAYS

Widening the Lens on Standardized Patient Assessment: What the Encounter Can Reveal about the Development of Clinical Competence

Rose, Mike PhD; Wilkerson, LuAnn EdD

Author Information

Abstract

There's a lot of uncertainty when you're actually with a person …. Nothing in real life is ever as clear as in a book.—Fourth-year medical student reflecting on his examination of a standardized patient

For several decades now, standardized patients (SPs) have been used in medical education, primarily as a means of assessing performance.1,2 The use of the SP format has been steadily increasing, the technique has been extensively studied and refined, and there is now solid evidence that it is a valid and reliable assessment device.3–5 It has become part of the medical student's educational terrain. Some recent work6,7 and our own experience make us wonder, however, if we might be missing some essential issues in interpreting and using the results of SP examinations.

During a recent assessment of fourth-year medical students at UCLA, we added an activity to the SP routine that proved to be revelatory. Typically, the SP encounter is scored by the patient and/or faculty member using a checklist, and the scores are analyzed quantitatively and distributed to the students. For a subset of our students in this recent assessment, we added several qualitative components. Between us, we viewed approximately one fifth of the SP encounters over video monitors placed in the examining room, and we read the comments made by the patients on their evaluation forms. As soon as the series of examinations was over, we asked for a small number of students to volunteer to view with one of us one of the tapes of their examinations. Each student was asked to comment on the examination, and to recall, as well as possible, what he or she was thinking and feeling during the examination. (This is a standard technique used in cognitive psychology to study problem solving and decision making.)

As we reflected on these interviews, and on the other qualitative information we collected, we were impressed by what they revealed about the development of clinical expertise and the insight they provided into complex issues in the medical education literature concerning cognition, clinical skill, communication, and empathy. These data led us back to the SP literature, and as we reread it with our interviews in mind, we were struck by what the psychometric orientation of that literature—the focus on issues of test validity and reliability—keeps us from seeing in the SP encounter. We were struck as well by the limited conceptualizations of development, communication, and empathy underlying the psychometrically oriented studies. We surely do not want to dispute the importance of test validity and reliability; such concerns are central to the development of good assessment techniques. But there is more going on in these encounters, and it has rich pedagogic value.

Let us elaborate by presenting two passages from our interview with one of the fourth-year students. He had just completed his examinations of eight standardized patients with a wide range of presenting situations—from a middle-aged woman with chest pain, to a non-compliant diabetic, to a well-child visit—each encounter lasting 15 minutes and followed by a seven-minute period to fill out a diagnostic worksheet. The student made the following comments as he observed two points in his encounter with the woman complaining of chest pain. First, he comments on the point when he has just read her chart and is formulating his thoughts; second, he comments on the end of the examination, reflecting on the way he reasoned through the case and drew on his basic science and developing clinical knowledge.

O.K., so what I'm thinking is there's probably a checklist of information that this patient is going to have that they want me to address. So I'm trying to just get major categories that I can then springboard off of, so that I don't get in there totally speechless. I figure if I have one or two points written down that if I get nervous I can look at those and then my knowledge base will kick in and I can kinda go on auto-pilot after that. But when you get in there with a patient, it's like it can be as if you know nothing, because you become speechless.

I'm at the stage where … I'm trying to find the balance between starting out with a focus, with a hypothesis that I'm ruling in or I'm ruling out, and at the same time, having that list of, review of symptoms [in my mind] that a patient may have, and being able, almost instantaneous as possible, to scan that and pick out the three or four that I also want to ask about.

The gist of what this young man reveals is no doubt familiar to many medical students and to physicians thinking back to their own training. Granted, this examination is also being evaluated, and therefore a certain twist and pressure are added to the performance. (Thus the student is aware of “a checklist of information” the patient might have.) But, still, the mix of challenges this student articulates is, we think, typical of the challenges facing medical students at this level: Within the brief 15 minutes of an SP examination, he is trying to efficiently access his knowledge, conduct a successful interview, and avoid both the cognitive and the social blunders that will undermine his attempt to present himself as a competent professional.

As we sat in the viewing room watching the students conduct their clinical examinations, we were, of course, thinking about issues of performance and assessment: How were particular students doing with particular cases? How well prepared did they seem to be? Were there obvious gaps in their training that needed to be addressed? But we also began to think about these encounters in a somewhat different way: What were these encounters revealing about the developmental trajectories of our students as emerging physicians? These eight 15-minute encounters could be viewed as a series of bounded moments in growth: eight snapshots of performance at this time and place in students' medical careers.

Our post-encounter interviews with the students make clear how much of an integrative experience these clinical performance examinations are, how many strands of students' education—basic and clinical, formal and informal—come together (often with some discordance) during these encounters. Below we explore the various dimensions of performance that constitute one of these 15-minute examinations, which could also be considered the dimensions of the development of clinical competence.

DIMENSIONS OF THE SP EXAMINATION

Involved, first of all, in the SP examination is the significant body of basic science knowledge learned in the initial two years of medical school. This knowledge itself is varied and involves different disciplinary orientations to the human body; further, it is added to, elaborated, illustrated, qualified, even superseded by clinical experience and instruction, which another one of the students characterized as “such (a) disparate way of knowing.” Clinical experience is also widely varied in context, quality, and instructional focus. These issues, of course, have been much discussed in the medical education literature, mostly from a curriculum or assessment perspective. It's interesting, though, to shift to the perspective of the student who must execute this integration of a vast and heterogeneous body of knowledge and experience and display it in a clinical setting. As the student quoted above put it: “I'm (still) trying to find a way to organize all this information …. And it's not just knowing the information, but it's being able to access it—that's what's so important about the organization.” To be sure, a good deal of the information students learn is presented in clinically useful patterns—differentials, diagnostic algorithms, constellations of symptoms—but, still, a significant amount of cognitive work remains as students turn knowledge into clinical action.

The physical examination raises another set of issues as the accessing and deployment of knowledge are interrelated with techniques, developed sense perception (palpation, the look of things), the use of instruments, and sequences of procedures. The memory of routines is involved here (and some students commented on the degree to which their routines are not yet automatic), as is the finesse in executing them. And, of course, knowing what you're feeling and seeing and having a sense of normal variation come only with sufficient hands-on experience, which usually has not been accumulated by the beginning of the fourth year. As yet another student put it: “As a third- or fourth-year student, you haven't done enough of them to know exactly what a normal, you know, liver edge feels like …. You see them in a textbook and you see pictures, but the pictures don't necessarily translate to real life.”

Medical knowledge and technique are essential to the clinical examination, but the examination is driven by a social engine. The student has to be able to establish rapport, ask questions in a clear and effective way, listen carefully, and respond appropriately—all in the service of clinical reasoning. Watching the videotapes provides illustration after illustration of the complexity of this dimension. Most of the students—and the SPs agreed—exhibited concern and a pleasant demeanor, and were clearly trying to do well by the patient. In a way, the social dimension of the patient encounter is the least foreign aspect of the examination: even if someone is shy or nervous, there's a common-sense familiarity to meeting someone, receiving and conveying information around an issue of mutual concern, and giving advice based on something someone knows. But the familiarity is misleading. The fourth-year student is trying to access knowledge that is still being developed into clinical utility and performing routines that aren't yet automatic while at the same time establishing human connection and conducting a focused, purposive interview. It is no surprise, then, that a number of SPs, though noting that a student was decent and trying to connect, would also observe that the student was “more concerned with the diagnosis than the person” or seemed to run “through a mental checklist.” These seemingly contradictory reports reveal the developmental demands of cognitive load and communicative interaction, powerfully captured in one SP's report: “His eyes were kind and he treated me with respect. But as the encounter progressed, he became more and more lost in thought—like I was a puzzle he wanted to solve rather than a person.”

There is a certain routinization and integration of the elements of an examination that comes with experience, and, of course, most fourth-year students have not yet reached this point. In fact, the “psychosocial and communicative” and the “cognitive” dimensions of the examination, typically separated in discussion in the medical education literature, are tightly interwoven in the diagnostic work of the expert physician, and are not neatly separable. This point is nicely illustrated by the case of the diabetic SP in for a routine check-up. He was a charming, fiftyish man who, in response to the right kinds of questions, would reveal his noncompliance with diet, exercise, and the monitoring of blood sugars. Here, the ability of the physician to place her- or himself in the patient's shoes, create a trusting atmosphere, and ask appropriate questions was integral to diagnosis and the formulation of the treatment plan. There were times when we saw this occur, and there were other times when the patient felt disconnected and wasn't forthcoming or when the encounter was pleasant, but not penetrating, the student being misled by the patient's good-guy demeanor. Students' responses to this variation during routine debriefing were telling: they tended to criticize the actor's credibility rather than reflect on what the case revealed about the interconnection of the psychosocial and the clinical/diagnostic.

Connected to all the above is the issue of role. We sometimes forget how pressing an issue this can be for a medical student. Here are (mostly) 20-something people moving quickly into one of the defining periods of a most demanding profession, fraught with expectation, encountering patients usually older than they are. Students report informal ways they observe senior physicians, noting behavior and bearing. And they occasionally report, in candid moments, the kind of anxiety expressed by the first student we quoted: am I going to appear knowledgeable? The overwhelming question here is: How will I present myself? How will I bring all my training to bear in this clinical moment in a way that inspires confidence?

Finally, there is a personal or existential dimension to the patient encounter that is woven into the foregoing, but that we'll separate out for analytic purposes. Much of medical training, formal and informal, guides students to function in the realm of the diagnostic/scientific and to leave personal beliefs, biases, and concerns at the door of the examining room. There are, as well, strong norms of care, humane treatment, and empathy—though expressed within the confines of the clinical relationship. These limits and constraints on one's own beliefs and emotions are good and necessary. But it is worth considering, in the context of our discussion of development, how difficult it is to achieve a balance. First, there is something a bit paradoxical in simultaneously encouraging empathy—which can require one to access one's own emotional experience—and clinical distance. Second, is it really possible to step outside one's personal and cultural history, one's religious and ethical beliefs, one's own experiences with illness (or elaborate defenses against thinking about one's vulnerability), one's fears and apprehensions, and still connect with someone in a way that, at the least, appears authentic? And if one can do it with the UPS driver with back pain and the woman with the chronic cough, can one do it equally well with, say, the young woman who probably has cancer or a listless teenager engaging in self-defeating, possibly dangerous behavior? Ideally, yes. With a lot of experience, probably. But it seems to us to be a further challenge for at least some students who are still developing their professional selves.

DISCUSSION

Many of the foregoing issues cannot be captured in their complexity with checklist items and quantitative scores, yet they represent the kinds of issues that are pressing at this point in a medical student's career. A related concern is that the method of scoring can lead to conceptualizations of communication and empathy that are reductive.6,7 Communication is often treated in the standardized patient literature as a unitary set of general skills,8 yet, as much social science has demonstrated, communication is dynamic, interactive, and influenced by context.9,10 Furthermore, the cognitive and the communicative aspects of clinical practice are usually separated out in the literature as discrete items, yet the two interact in practice,11 as the case of the noncompliant diabetic patient illustrates. There is, as well, a cognitive dimension to empathy: it is a way of knowing things about a patient. We are in the realm here of the art of medicine, which, as the great clinicians from Osler's time to the present demonstrate, is a powerful mix of knowledge and feeling, information processing and intuition.

These dynamic phenomena emerge in the videotapes of the SP encounter, yet the tapes, if used at all, are typically archived for legal purposes or are analyzed for psychometric qualities. They are rarely used for further pedagogic gain. Given the expense of conducting SP assessments ($200-$400 per student), it seems both wasteful and a missed opportunity not to engage students in conversation and analysis about what was going on during those encounters. It is not clear from the literature how many programs include review and discussion with students of videotapes of objective structured clinical examinations (OSCEs) for purposes of exploring and improving performance.

In a study comparing scores in a family medicine OSCE given by faculty observers with those given by the SPs, the authors found only a low to moderate correlation between the two perspectives.12 The authors conclude that the “SPs' perspective should be formally included in evaluations of student performance.” While this study concerned checklist data, our experience demonstrated to us that the narrative data that can be provided by the SPs may be equally, if not more, useful. This year we included the narrative comments made by the SPs in the feedback to our senior students for the clinical performance examination.

Medical educators should, of course, continue to concern themselves with the psychometric properties of SP assessment, but there is much more to be had. Focused and thoughtful exchanges between faculty and students about performances recorded on tape or described in narrative comments by the SPs could provide a rich source of information about the touchstones of development and the many interacting dimensions of clinical practice.

References

1. Anderson MB, Stillman PL, Wang Y. Growing use of standardized patients in teaching and evaluation in medical education. Teach Learn Med. 1994;6:15–22.
2. Barrows HS. An overview of the uses of standardized patients for teaching and evaluating clinical skills. Acad Med. 1993;68:443–51.
3. Rothman AI. Understanding the objective structured clinical examination. Aust. NZ J Surg. 1995;65:302–3.
4. Colliver JA, Swartz MH. Reliability and validity issues in standardized patient assessment. In: Distlehorst LH, Dunnington GL, Folse JR (eds). Teaching and Learning in Medical and Surgical Education: Lessons Learned for the 21st Century. Mahwah, NJ: Lawrence Erlbaum Associates, 2000:229–35.
5. Reznick RK, Blackmore D, Dauphinée WD, Rothman AI, Smee S. Large-scale high-stakes testing with an OSCE: report from the Medical Council of Canada. Acad Med. 1996;71:S19–S21.
6. Colliver JA, Willis MS, Robbs RS, Cohen DS, Swartz MH. Assessment of empathy in a standardized-patient examination. Teach Learn Med. 1998;10:8–11.
7. Colliver JA, Swartz MH, Robbs RS, Cohen DS. Relationship between clinical competence and interpersonal and communication skills in standardized-patient assessment. Acad Med. 1999;74:271–4.
8. Hodges B, Turnbull J, Cohen R, Bienenstock A, Norman G. Evaluating communication skills in the objective structured clinical examination format: reliability and generalizability. Med Educ. 1996;30:38–44.
9. Riggins SH (ed). Beyond Goffman: Studies on Communication, Institution, and Social Interaction. New York: deGruyter, 1990.
10. Engestrom Y, Middleton D (eds). Cognition and Communication at Work. Cambridge, U.K.: Cambridge University Press, 1996.
11. Meredith P, Wood C. Aspects of patient satisfaction with communication in surgical care: confirming qualitative feedback through quantitative methods. Int J Qual Health Care. 1996;8:253–64.
12. Blue AV, Chessman AW, Gilbert GE, Mainous AG. Responding to patients' emotions: important for standardized patient satisfaction. Fam Med. 2000;32:326–30.
© 2001 Association of American Medical Colleges