Home Current Issue Previous Issues Published Ahead-of-Print Collections For Authors Journal Info
Skip Navigation LinksHome > October 2004 - Volume 79 - Issue 10 > Basing the Evaluation of Professionalism on Observable Behav...
Text sizing:
A
A
A
Academic Medicine:
Papers: Professional Development

Basing the Evaluation of Professionalism on Observable Behaviors: A Cautionary Tale

Ginsburg, Shiphra; Regehr, Glenn; Lingard, Lorelei

Section Editor(s): Stern, David MD

Free Access
Article Outline
Collapse Box

Author Information

Correspondence: Shiphra Ginsburg, MD, MEd, FRCPC, Mount Sinai Hospital, 433-600 University Ave, Toronto, ON, Canada. M5G 1X5; e-mail: 〈shiphra.ginsburg@utoronto.ca〉.

Collapse Box

Abstract

Problem Statement and Background. The evaluation of professionalism often relies on the observation and interpretation of students' behaviors; however, little research is available regarding faculty's interpretations of these behaviors.

Method. Interviews were conducted with 30 faculty, who were asked to respond to five videotaped scenarios in which students are placed in professionally challenging situations. Behaviors were catalogued by person and by scenario.

Results. There was little agreement between faculty about what students should and should not do in each scenario. Abstracted principles (e.g., honesty, altruism) were defined and applied inconsistently, both between and within individual faculty. There was no apparent “shared standard” that faculty held for professional behavior in students, and similar behaviors (e.g., lying) could be interpreted as either professional or unprofessional.

Conclusions. Future efforts at evaluation need to look beyond the behaviors, and should incorporate the reasoning and motivations behind students' actions in challenging professional situations.

Professional organizations have worked to define “professionalism” and to develop standards for its teaching and evaluation.1,2 Recent research has suggested that frameworks based on abstracted definitions (e.g., altruism, accountability) may be difficult to operationalize and apply in day-to-day settings, and that behavior-based approaches may be more reflective of the reality of how students and physicians practice.3,4 The current trend towards competency-based assessment has served to shift our focus towards “observable behaviors” as an attempt to improve the reliability and validity of evaluations.5,6 But the legitimacy of these approaches depends on two critical assumptions: that evaluators share a common set of standards for what constitutes professional or unprofessional behavior, and that a student's behavior is an index of his or her willingness or ability to adhere to these standards.

To test these two assumptions, this study used hypothetical scenarios to explore faculty's perceptions about students' behavior and to determine whether there is between- and within-faculty agreement about what medical students should or should not do in professionally challenging situations.

Back to Top | Article Outline

Method

This study builds on previously published work, and a more detailed description of methods and scenario development can be found elsewhere.7

Back to Top | Article Outline
Subjects

Because the scenarios developed were clinically relevant to internal medicine and surgery, we recruited 15 faculty internists and 15 faculty surgeons at the University of Toronto. Recruitment was via e-mail to teaching hospitals' active staff lists, with responses sent directly to the research assistant (RA) who scheduled the interviews. Each faculty member had to be on active staff in a teaching hospital, with significant contact with medical students (more than three months per year). Monetary remuneration was offered, and institutional review board approval was obtained. Participation was voluntary, anonymous, and confidential. The investigators did not know who had or had not participated.

Back to Top | Article Outline
Scenarios

Five videotaped scenarios were developed in a previous study, based on medical students' descriptions of professionally challenging situations; these cases were chosen to reflect a range of common, difficult, and realistic dilemmas as perceived by students. Each depicts a dramatization of a real-life event during which a student is placed in a challenging situation that requires action in response to a professional dilemma. Each scenario ends at the point at which the student must act.

Back to Top | Article Outline
Brief Summaries of Videos

Video 1: A student has just been told by the attending surgeon not to tell a patient the results of a test showing that she has a tumor; in the next scene, the patient asks the student directly what her tests show.

Video 2: A medical resident is trying to get the team out early on the last day of a rotation, but a student wants the resident to review a patient's insulin orders first. The resident says they can wait until Monday, but the student is uncomfortable.

Video 3: A student wants to go watch a bone marrow biopsy, but has just told a patient with dementia that she'd see him right now.

Video 4: A male doctor in a fertility clinic is enthusiastically teaching his students how to examine a male patient's genitals, but the patient has not been asked permission and is uncomfortable. The attending asks the female student to palpate and explain what she feels, but no one is wearing gloves.

Video 5: A student is doing her first thoracentesis while the resident supervises, when a nurse walks in and asks the student if she's ever done one before.

Back to Top | Article Outline
Interviews

Each faculty member participated in a one-hour, individual, semistructured interview, during which the five videos were played in the same sequence. After each video, the RA asked the faculty “What do you think the student should do next?” Faculty were also asked what a student should not do, as well as what they think most students would actually do in the given situation. These questions were not always asked and answered in a set order, since many faculty spontaneously spoke about the issues without being prompted; however, all faculty eventually answered all questions. Interviews were audiotaped, transcribed, and rendered anonymous before being seen by the investigators.

By using five cases per faculty, the sample size of 30 subjects yielded 150 case-based interviews. Based on theoretical sampling and our previous experiences, this data set was expected to be sufficient to saturate the range of faculty's viewpoints in relation to these five representative cases.8

Back to Top | Article Outline
Analysis

The data were analyzed in two ways. First, a content analysis was performed to catalogue all of the behaviors that faculty suggested students might do. We began with the list of behaviors generated by students for each video in a previous study, and added new behaviors to the database as they appeared in the transcripts.7 The RA catalogued all the behaviors, which were subsequently reviewed by two of the researchers; discrepancies were resolved by consensus. The database was maintained in an Excel spreadsheet. In addition, transcripts were analyzed by grounded theory, which is a qualitative methodology that is used to develop explanatory models generated by an iterative process of sampling of unstructured data, such as from interviews.9 The results of this analysis will be presented in detail elsewhere, but is included here to highlight some of the thematic categories that arose. Briefly, we began with a template developed in a previous study, and developed thematic categories that captured faculty's perceptions of the reasons that students act in challenging situations.7

Back to Top | Article Outline

Results

A criterion sample of 15 internists and 15 surgeons was initially enrolled.8 Three interviews were lost in transcription, and three initial pilot interviews were included because no changes had been made to the procedure after the pilot test. This resulted in a sample of 13 internists and 17 surgeons, which yielded 296 pages of textual material for analysis.

Faculty participants provided numerous suggestions for student action in each situation. The total number of unique alternatives suggested for each video, and the mean number of responses per subject per video, are shown in Table 1.

Table 1
Table 1
Image Tools

There were no obvious differences in responses between the medicine (M) and surgery (S) faculty for three of the videos; however, for videos 1 and 4, the medicine faculty provided on average more suggestions than the surgery faculty. Interestingly, these two scenarios involved a staff physician (one a surgeon, the other unspecified) in a directive role, whereas the other three depicted only students and residents. However, even in these two scenarios, the texture and tone of the responses were not different between the two groups, and no other differences between the groups were detected. Therefore, the responses from all faculty are reported together.

Back to Top | Article Outline
Between-Faculty Consistency

The behaviors that faculty most frequently identified for each scenario are shown in Table 1. Based on these results, there appears to be little agreement between faculty regarding what students should do, which suggests that the scenarios were sufficiently challenging—each one provoked many alternatives for action, and there was no single “right” answer prioritized by all respondents. For example, in the fertility clinic video (video 4), four out of 30 faculty thought the students should speak directly to the patient and ensure that he consents to having students examine him, and an additional 16 felt that that would be an acceptable option. However, ten of the faculty did not even mention this as a possibility, despite the fact that on average 5.1 options were mentioned by each respondent for this scenario.

Further, in several cases, what some respondents thought students should do was exactly what others felt students should not do. For example, in video 2, six faculty felt that the student should pursue the matter of the insulin orders directly with the staff physician (i.e., go over the resident's head), while two said a student should not do that; five thought the best thing to do would be to push the same resident further to get the orders written, while two thought this was a bad idea; and although 15 of the faculty said that students should not simply obey the resident and leave with the team for the weekend, three felt that this was what a student should do.

Discrepancies between faculty also appeared as inconsistencies in the application of abstract principles, such as altruism. For instance, in video 3, the student must choose between spending the necessary time with a patient or going to watch a bone marrow biopsy. Seven faculty felt that the student should go see the procedure first, because she may not get another opportunity, while two thought she should forego the bone marrow and spend whatever time is necessary with the patient. As one respondent (M10) put it, “ … it's a conflict of interest in that the student is asked to choose between something that furthers her interest … go get educated … versus something that's in the patient's interest, which is to hear the truth again…” This attending feels the best option for the student is to put the patient first. However, a different respondent (M13) thought otherwise: “… you owe it to yourself and ultimately to your patients … to take every educational opportunity … it is something they need to learn to help future patients.”

Back to Top | Article Outline
Within-Faculty Consistency

To determine whether differences in opinion were due to stable individual differences in faculty respondents, we explored two aspects of individuals' responses: the number of alternatives suggested, and the application of principles across scenarios. As Table 1 shows, for each video there was at least one respondent that generated only two alternatives for action; however, this was not the same individual in each scenario. For example, S14 gave only two suggestions in video 1 but nine options in video 2, whereas S5 gave five options in video 1 and two in video 2. Therefore, it was not the case that some faculty were more “thoughtful” than others, or that some found all of the scenarios “easy” while others struggled. Rather, the relative complexity of individuals' responses varied idiosyncratically across scenarios.

A second key finding was that individual faculty often suggested alternatives across videos that were not internally consistent in their application of abstracted principles. For example, in video 5, one faculty (S7) responded that the student should tell the patient that it's her first time doing a thoracentesis, and specifically stated that “You can't be evasive here … you should never lie.” However, this same respondent in video 4 said the students should tell the attending that they forgot that they had a seminar to attend downtown, as a means of extricating themselves from an uncomfortable situation. For S7, then, the principle of honesty was not of overriding importance across cases—it was in one scenario, but not in another. Differences in context may explain this inconsistency: in video 5 the issue was whether or not to be honest with a patient, while in video 4 the dishonesty (making up an excuse to leave) did not involve the patient at all.

However, inconsistency in the application of “honesty” was also seen within the same scenario. In video 1, M10 stated that “What [the student] absolutely should not do, is say, ‘Everything is okay,' because that's actually lying. Because everything is not okay.” This respondent further stated that lying is “morally wrong” and is never the right thing to do. However, later in the interview he went on to say that “the student could say that ‘With regards to the surgery everything is okay …' I could see her giving an answer like that and I don't think that would be absolutely wrong.” To M10, this appears to be an acceptable response for the student because this withholding of information directly relevant to the patient's question is not “actually lying, or misrepresenting the truth.”

Back to Top | Article Outline

Discussion

These results suggest that there is substantial disagreement both between and within faculty about what constitutes professional and unprofessional behavior in medical students. This concept in itself is not entirely new, in that other studies have shown inconsistencies in how certain abstract principles of professionalism are applied. For example, physicians and residents have been shown to be more willing to use deception in some scenarios than others, and disagreements have been shown amongst students about what does or does not constitute cheating.10–12 But these studies have focused on what individuals themselves might do, not what they felt others should do. One might postulate that when asked for an “ideal,” or at least appropriate, solution for medical students in common professional dilemmas, the responses from faculty would be more idealistic and standardized, based on the application of the principles espoused by the profession. However, even altruism, which is often considered the highest of ideals, is not so easily operationalized by faculty when considering students' behavior—in some cases, putting the student's patient first was not the preferred option. Such responses demonstrate a flexibility in both the definition of principles (such as honesty or altruism) and their application. The translation of abstracted ideals into defensible behaviors is therefore more complex than is usually acknowledged.

The fact that abstracted ideals and principles do not map easily onto behaviors is important, and has previously been recognized.4 But the really critical issue our results suggest is that behaviors themselves may not in fact be obvious or transparent indications of “professionalism.” Consider video 1, in which a student has been told not to disclose a specific test result (which indicates a tumor) to a patient, who then asks the student, “What do my tests show?” To one faculty member, the dilemma involves a values conflict between two options: obey the surgeon and lie to the patient, or disobey the surgeon and disclose the diagnosis in order to be honest with the patient. As M5 put it, “The student has been asked a direct question by the patient. The student knows the result… Essentially, to not tell the truth about the test result, would, in my mind be incorrect.” From the point of view of this faculty, lying to the patient is seen as unprofessional. However, for another faculty member, the values conflict might be construed differently: tell the truth about the test result and risk distressing the patient (because you're not equipped to answer all of her inevitable questions), or lie to the patient to spare her feelings, at least for the time being, as S16 stated: “They shouldn't give the patient information … that they can't adequately discuss … and giving the patient half the information is worse than not giving the patient any information at all, as far as I'm concerned.” From the point of view of this faculty, lying to the patient may be the more professional thing to do.

In both of these values conflicts, the principle of “honesty” is at stake—but if our first faculty member had witnessed the student lying to the patient, s/he might rate that student (or at least her behavior) as being unprofessional, whereas if the second had been present the student might be rated as being very professional, not despite the lie but because of it. In terms of evaluating professionalism, our focus has shifted away from “attitudes” and towards observable behaviors, as a necessary step towards improving reliability and validity. However, if we focus only on behaviors, and fail to take the context into account, we risk making what is known as the fundamental attribution error; that is, the tendency to underestimate the situation and overestimate the extent to which a behavior reflects stable traits or attitudes in an individual.13,14 Our current evaluation systems tend to ignore the fact that context inevitably drives an observer's interpretation of behaviors, and critically, the interpretation of the motivations for the behaviors.3

We are not suggesting that behaviors are irrelevant—clearly, what a student does is still important. However, sophisticated evaluation of professionalism requires an additional dimension, as behaviors alone do not give us all of the information we need to make accurate judgments. Knowing how a student construes a particular professional dilemma, and what values s/he perceives as conflicting, is critical information.3 Furthermore, analyzing the reasoning behind students' behaviors may give us significant insights into how they make decisions when faced with professional dilemmas.7 Finally, these results reinforce the need for evaluations to be based on multiple observations over time, rather than on a single instance of behavior.

Our results suggest that there is no apparent shared standard of “professionalism” that is applied uniformly and consistently by faculty. Although this study was performed in a single institution, it involved faculty from two large clinical departments; however, it is unclear whether faculty from different types of institutions (e.g., academic vs. community) or clinical disciplines (e.g., psychiatry or pediatrics) would show similar patterns of responses. Nonetheless, we would suggest that the translation of abstracted principles into action will always be complex, and depends on the context and values conflict inherent in each situation. Moreover, methods of evaluating professionalism, while retaining some focus on what students actually do, should look beyond observable behaviors to include the reasoning behind them, in order to develop a more accurate assessment of a student's developing professionalism.

The authors wish to acknowledge the Medical Council of Canada for funding this research, and the Wilson Centre for Research in Education, Faculty of Medicine, University of Toronto for its support. Dr. Regehr is supported as the Richard and Elizabeth Currie Chair in Health Professions Education Research. They also thank Maria Mylopoulous for coding the behaviors and maintaining the database.

Back to Top | Article Outline

References

1. Project Professionalism. American Board of Internal Medicine. Philadelphia: American Board of Internal Medicine, 1995.

2. ABIM Foundation. American Board of Internal Medicine. ACP-ASIM Foundation, American College of Physicians, American Society of Internal Medicine, European Federation of Internal Medicine. Medical professionalism in the new millennium: a charter for physicians. Ann Intern Med. 2002;136:243–46; Lancet. 2002;359(9305):520–3.

3. Ginsburg SR, Regehr GR, Hatala R, et al. Context, conflict, and resolution: a new conceptual framework for evaluating professionalism. Acad Med. 2000;75(suppl 10):S6–11.

4. Ginsburg SR, Regehr G, Lingard L. The anatomy of the professional lapse: bridging the gap between traditional frameworks and students' perceptions. Acad Med. 2002;77:516–22.

5. Frank J, Jabour M, Tugwell P (chair), et al., for the Societal Needs Working Group. Skills for the new millennium: report of the societal needs working group. Ottawa: Royal College of Physicians and Surgeons of Canada, 1996.

6. Accreditation Council on Graduate Medical Education. ACGME Outcomes Project 〈http://www.acgme.org/outcome/project/proHome.asp〉. Accessed 3 February 2004. Accreditation Council on Graduate Medical Education, Chicago, IL.

7. Ginsburg S, Regehr G, Lingard L. The disavowed curriculum: what motivates students in professionally challenging situations? J Gen Intern Med. 2003;18:1015–22.

8. Creswell JW. Qualitative Inquiry and Research Design. 1st ed. Thousand Oaks, CA: Sage Publications, 1998.

9. Pidgeon N. Grounded theory: theoretical background. In: Richardson JTE. Handbook of Qualitative Research Methods for Psychology and the Social Sciences. 3rd ed. Oxford: PBS Blackwell, 2003:75–85.

10. Novack DH, Detering BJ, Arnold R, Forrow L, Ladinsky M, Pezzullo JC. Physicians' attitudes toward using deception to resolve difficult ethical problems. JAMA 1989;261:2980–5.

11. Green MJ, Farber NJ, Ubel PA, et al. Lying to each other: when internal medicine residents use deception with their colleagues. Arch Intern Med. 2000;160:2317–23.

12. Rennie SC, Crosby JR. Are “tomorrow's doctors” honest? Questionnaire study exploring medical students' attitudes and reported behavior on academic misconduct. BMJ. 2001;322:274–5.

13. Sabini J, Siepmann M, Stein J. The really fundamental attribution error in social psychological research. Psych Inquiry. 2001;12:1–15.

14. Ross L, Nisbett RE. The person and the situation: perspectives of social psychology. Philadelphia: Temple University Press, 1991.

© 2004 Association of American Medical Colleges

Login

Article Tools

Images

Share