Secondary Logo

Journal Logo

Research Reports

Promoting Readiness for Residency: Embedding Simulation-Based Mastery Learning for Breaking Bad News Into the Medicine Subinternship

Vermylen, Julia H. MD, MPH; Wayne, Diane B. MD; Cohen, Elaine R. MEd; McGaghie, William C. PhD; Wood, Gordon J. MD, MSCI

Author Information
doi: 10.1097/ACM.0000000000003210
  • Free

Abstract

Physician communication with patients and families affects quality of life, satisfaction, trust in physicians, and overall health care use.1 Inadequate communication has been linked to medical errors,2 malpractice claims,3 and preventable deaths.4 Research agendas and lists of quality indicators now identify communication as a leading priority,5–10 and numerous educational interventions have begun to focus on improving communication skills training.1

Despite growing recognition of the importance of communication, gaps remain in current communication skills training practices in undergraduate, graduate, and continuing medical education. While studies show that learners gain new skills after rigorous communication skills training,11,12 the majority of published training programs that document a change in clinician behavior are not embedded within existing clerkship structures and do not ensure or document competency for all learners.1,12

Simulation-based mastery learning (SBML) offers a potential solution as it is competency based rather than norm based. SBML consists of a bundle of activities, including a pretest then didactic instruction followed by ongoing deliberate practice until learners can meet or exceed a minimum passing standard (MPS) on an assessment.13 In SBML, assessment and instruction are inseparable because assessment informs the deliberate practice and determines duration of training. SBML uses testing data as a tool, not as a weapon, but this can only be done in a climate of psychological safety14 for the learner who is free from evaluation apprehension.15 Mastery learning will work only where trust and cooperation exist among teachers and learners.

As part of an institutional effort to create a readiness-for-residency experience for graduating medical students, we developed an SBML breaking bad news (BBN) curriculum for fourth-year medical students at Northwestern University Feinberg School of Medicine (FSM), a large, urban medical school in Chicago, Illinois.16 BBN was chosen as the focus because it is a foundational serious illness conversation that encompasses skills of delivering information, providing guidance, noticing and responding to emotion, and being responsive to patients’ needs during a conversation. As such, BBN is often the first conversation taught in courses aimed at improving serious illness communication skills.11 BBN has also been identified as a key skill for graduating medical students in a national survey17 and is part of the Physician Competency Reference Set,18 the foundation of the Association of American Medical Colleges’ core entrustable professional activities for entering residency.19 SBML was chosen as the educational methodology because it has been used in a variety of health care contexts including training in procedural, team, and communication skills (e.g., code status discussions20) and has been shown to have powerful effects on both trainee- and patient-centered outcomes.21 The decision was made to embed the curriculum in the medicine subinternship because this is where graduating students are closely involved in the care of medically complex patients and are likely to participate in BBN, which improves buy-in and lends itself to immediate application of the learned skills.

We have previously described the development of the curriculum and assessment tools and how an expert panel set a defensible MPS based on the performance of a small cohort of students.16 Questions remained about whether the study could be implemented in a larger, more diverse population of students during a required clerkship and whether all students could be brought to mastery. The purpose of this study was to embed an SBML communication skills curriculum as a required component of a medicine subinternship, demonstrate feasibility, and bring graduating medical students to mastery in the foundational skill of BBN.

Method

Study design and participants

This study was a pretest–posttest design of an SBML intervention in BBN skills. All participants were fourth-year medical students enrolled in the medicine subinternship at FSM from September 2017 through August 2018. Students at FSM must complete a required 4-week subinternship rotation and may select internal medicine, surgery, obstetrics and gynecology, or pediatrics. This study was deemed exempt from review by the Northwestern University Institutional Review Board (10/27/14).

Study procedure

All eligible participants completed a pretest in which the student led a 15-minute conversation with a standardized patient (SP). The student was asked to disclose results of brain imaging showing a mass highly suspicious for malignancy. The encounter was videorecorded and viewed by faculty who assessed the student using a previously published assessment tool consisting of a 15-item checklist and 6 scaled items.16 The assessment tool has previously been shown to have moderate to high inter-rater reliability, and the same raters (J.H.V., G.J.W.) were used in this study.16 After completing the pretest, learners participated in a 4-hour workshop. At the start of the workshop, learners received focused feedback about their pretest performance. A lecture was then presented on the SPIKES22 protocol to deliver bad news and included a demonstration of a BBN encounter led by a clinical expert. We chose the SPIKES framework as it has been shown to result in greater skill acquisition compared with curricula based on another or no framework.12 After completing this didactic content, each student used their pretest feedback to develop individualized learning plans for improving their BBN skills.

As part of the 4-hour workshop, participants next completed 3 hours of deliberate practice of BBN skills. We used 4 clinical cases highlighting different scenarios. These deliberate practice sessions involved small-group training with simulated patients using methods adapted from the VitalTalk methodology.1,11 We chose the VitalTalk methodology as it has been shown to change learner communication skill behaviors and could easily be adapted to the SBML BBN curriculum.11 One of 2 VitalTalk-trained facilitators (J.H.V., G.J.W.) led all workshops, and both were involved in actor training. Small groups consisted of 4 to 6 students and 1 faculty facilitator. Each student practiced and received feedback from the faculty facilitator, emphasizing topics in their individualized learning plan. Additionally, each student used a BBN note-taking guide to help facilitate active engagement throughout the workshop and reinforce key BBN skills. Finally, the students participated in a videorecorded posttest conversation with an SP delivering bad news that was scored using the assessment tool. This initial posttest occurred between 1 and 14 days after the workshop intervention. The pretest and posttest cases had the same clinical details but were presented by different SPs. If the student achieved the MPS on their initial posttest, they were considered to have completed the curriculum.

In the SBML model, it is expected that a proportion of students will not achieve the MPS at the initial posttest.23 Students who did not achieve the MPS at initial posttest were brought back to a one-on-one session involving the faculty member and an SP and underwent continuous deliberate practice until the student was consistently able to demonstrate acquisition of key BBN skills. These students then retook the posttest with a new SP and repeated this cycle of practice and assessment until the MPS was achieved. Students also completed anonymous postworkshop surveys.

Students were pulled from their clinical duties to complete all components of this course. The curriculum was repeated 11 times during the study period, once for each 4-week block that the subinternship was offered. Primary faculty leads (J.H.V., G.J.W.) received salary support of approximately 5% full-time equivalent for this project from the FSM Office of Medical Education.

Measurement

We used a previously published BBN assessment tool that consists of a 15-item checklist as well as 6 scaled items (3 global rating scale items and 3 Likert-scaled items) to assess whether and how well key communication steps were performed during a BBN conversation.16 The MPS for this tool was set previously at 80% or 12 of the 15 checklist items correct and a score of at least 4 out of 5 on each of the 6 scaled items.16

Participants provided demographic data including gender, age, intended specialty, and previous experience with communication skills tasks and BBN.

The postworkshop survey included retrospective pre–post24 questions about participants’ confidence in conducting BBN conversations as well as questions about the relevance of the curriculum to their future career, the effectiveness of teaching, how much they enjoyed the workshop, and whether they would recommend it to other students. Last, the survey allowed students to leave comments about what they thought were the most effective portions of the workshop as well as what they thought could make the workshop better in the future.

The primary outcome measure was performance of the medical students as assessed by the BBN assessment tool before and after the BBN SBML curriculum. Secondary outcome measures were changes in self-assessed confidence and postworkshop satisfaction.

Data analysis

We evaluated overall mean differences in BBN checklist skills using paired t tests. Chi-square tests were used to evaluate differences between individual checklist items and scaled quality of communication and Likert items. We used the Spearman’s rho correlation coefficients to evaluate relationships between demographic information or clinical experience provided by the learners and pretest performance. Chi-square tests were used to compare items on the postworkshop surveys. All analyses were performed using SPSS statistical software, version 25.0 (IBM Corp., Armonk, New York).

Results

Eighty-five students completed a medicine subinternship during the study period and were therefore eligible for inclusion in the study. This included over half of the fourth-year medical student class. Seventy-nine (93%) completed the entire protocol (Table 1). Six students (7%) were unable to complete the entire protocol during the 4-week rotation due to schedule constraints or personal/family health problems and were excluded. The majority of students were male (n = 49, 62%), between the ages of 25 and 30 years old (n = 56, 71%), and planned to enter residency training in internal medicine (n = 22, 28%), emergency medicine (n = 10, 13%), anesthesiology (n = 8, 10%), or radiology, (n = 8, 10%).

Table 1
Table 1:
Demographic Characteristics of 79 Fourth-Year Medical Students Who Completed the Breaking Bad News Curriculum, From a Study of SBML in the Subinternship Curriculum, Northwestern University Feinberg School of Medicine, September 2017–August 2018

A significant proportion (n = 29, 37%) of students reported that they had not received any formal training in how to conduct a BBN conversation, yet most (n = 55, 70%) students reported that they had delivered serious news themselves to a patient alone, without supervision before this training and 11 (13%) had done it at least monthly. Many (n = 53, 65%) reported navigating challenging emotions with patients at least monthly in the past year. There were no significant associations between demographic factors or prior experience and baseline pretest scores.

Students’ overall checklist performance significantly improved from a mean of 65.0% (SD = 16.2%) checklist items correct during the pretest to 94.2% correct (SD = 5.9%; P < .001) at the mastery posttest (Table 2). The mastery posttest is defined as the posttest in which the learner achieved the MPS. This could be the initial posttest after the workshop or a subsequent posttest after further deliberate practice if the MPS was not reached on the initial posttest. Participants’ scores for nearly all of the 15 checklist items were statistically improved, including key communication behaviors such as asking permission, conveying news in clear language, avoiding medical jargon, and discussing the plan and next steps only after the patient gave permission to do so. The only 2 checklist items that did not improve significantly had a trend toward improvement and had high baseline performance at pretesting (86% and 98% success rate at pretest). These skills were creating initial rapport when first walking into the room and delivering bad news within the first few minutes of the conversation. There was also significant improvement for all 6 scaled items (Table 3).

Table 2
Table 2:
Performance of 79 Fourth-Year Medical Students on Breaking Bad News Skills Checklist in the Breaking Bad News Curriculum, From a Study of SBML in the Subinternship Curriculum, Northwestern University Feinberg School of Medicine, September 2017–August 2018a
Table 3
Table 3:
Performance of 79 Medical Students on Breaking Bad News Skills Scaled Items in the Breaking Bad News Curriculum, From a Study of SBML in the Subinternship Curriculum, Northwestern University Feinberg School of Medicine, September 2017–August 2018a

Figure 1 shows checklist scores at pretest, initial posttest, and mastery posttest. At pretest, 19 students (24%) met the MPS based on the checklist alone. Only 9 (11%) met the MPS based on both the checklist and scaled items during the pretest. At initial posttest, 73 (92%) met the MPS based on checklist and scaled items. The 6 students who did not meet the MPS on the initial posttest were below the passing standard for both the checklist and scaled items. These 6 students subsequently achieved mastery with approximately 45 minutes of additional practice time. Thus, by the end of the curriculum, all 79 students met the mastery standard for both the 15-item checklist (Figure 1) and the 6 scaled items (Table 3).

Figure 1
Figure 1:
Breaking bad news checklist performance for 79 fourth-year medical students at pretest, initial posttest, and mastery posttest, from a study of an SBML curriculum embedded in the medicine subinternship, Northwestern University Feinberg School of Medicine, 2017–2018. Circles indicate each student’s score. Asterisks indicate the students who did not meet the MPS at the initial posttest. Those students did additional deliberate practice and tested again until all reached the MPS as indicated in the mastery posttest column. Abbreviations: SBML, simulation-based mastery learning; MPS, minimum passing standard.

Postworkshop surveys showed that there was a statistically significant improvement in self-reported confidence in conducting BBN conversations. Before training, only 3 students (4%) stated that they felt confident delivering serious news, while after training, 64 (80%) students stated they felt confident delivering serious news (P < .001). Similarly, before training, only 3 students (4%) stated they felt confident navigating challenging emotions, while after training, 53 (66%) stated they felt confident navigating challenging emotions (P < .001). Additionally, all students said they either agreed or strongly agreed that delivering serious news and navigating challenging emotions were important skills for them to learn before entering residency training. Seventy-six (95%) students stated they agreed or strongly agreed with the statement: “I enjoyed participating in the BBN workshop.” All students stated they would recommend the workshop to their colleagues. In the open comment sections, representative comments from students included discussion on how they found the feedback from the pretest helpful: “The feedback from the ‘preworkshop practice’ helped me understand what to work on” and “Doing the pretest was helpful and gave us a platform to improve from.” Others commented on how helpful the deliberate practice of the role play session was important for refining their skills: “Even though I am not a traditional role playing fan, the role playing with the SPs with coaching and feedback was amazing!! Role playing practice was the most useful step,” “I thought the practice communication skills were really effective,” and “This was definitely the first role play where I felt I truly learned a lot! It was so helpful to pause as we went along so we got real-time feedback.”

Discussion

All medical schools seek to graduate competent learners who are prepared to enter graduate medical education. Prior studies, however, have documented that there are large gaps between what training program directors expect from their learners and what recent medical school graduates can actually do.25 To close this gap, medical schools must go beyond capstone experiences that aim to improve skills within the last few weeks of students’ education and instead begin to embed rigorous competency-based education into required clerkships. Our study shows that a BBN SBML curriculum can be successfully embedded into a medical school clerkship and that this approach is not only feasible but also produces powerful educational outcomes for learners. In this study, all students ultimately reached the MPS on both checklist and scaled evaluations at the end of the BBN curriculum.

Compared with previously published small-group role play communication skills training interventions, the SBML model showed more uniform skills improvement during posttest encounters. In prior studies using similar small-group role play models but without the mastery learning approach, learners performed individual skills correctly in only 38%–81% of postintervention encounters.11 In our study, however, 14 of the 15 checklist items were performed correctly in 90% of mastery posttest encounters. Thus, using proven small-group communication skills training methods within an SBML model raises the bar for expected educational outcomes by insisting on “excellence for all.”

Having such robust learning outcomes is important because our students reported frequent clinical experience with independently breaking bad news to patients and families and dealing with challenging emotions. Students also reported feeling unprepared for these difficult conversations despite traditional education that included didactics and SP encounters during medical school. Although all students in our study completed required BBN training in their first and second years, including clinical skills practice and SP encounters, many students did not recall this training when asked. Our findings suggest that providing communication skills training with more frequency and within the clinical years as embedded curricula during rotations where the skills are used is needed to best prepare students to enter residency training. Even though this intervention required time away from clinical duties during the subinternship, the time was limited (about 4.5 hours over a 4-week rotation). Additionally, the poor pretest performance reinforces what is already known about how clinical experience alone is not the best approach to ensure adequate clinical skill acquisition.20,26,27

Limitations

Our study has several important limitations. It was performed at a single academic institution, and we only enrolled students completing a medicine subinternship so replicability in other settings has not yet been determined, although other SBML curricula have been successfully replicated.28 The frequency with which students reported having BBN conversations was based on self-report and was not otherwise corroborated. The pretest and posttest clinical details were the same, so we do not yet have data to show that these skills are transferrable to other types of simulated BBN encounters. Additionally, the educational intervention focused only on BBN skills so the ability to extend this approach to other important communication tasks such as goals of care conversations is unknown. The initial posttests were completed within 2 weeks of training, and we do not yet have data on application of skills at the bedside or on retention of skills over longer periods, although both have been shown in other SBML cohorts.29,30 Finally, we acknowledge that the SBML project described here is relatively faculty and resource intensive. Further research is needed to determine if there are other less costly interventions that could be used in the SBML model that would yield similar improvements in clinician behaviors. Further study is also needed to determine the impact of training such as BBN SBML on patient care outcomes to demonstrate that the cost produces benefit, although a large body of data now exists showing that SBML produces better patient outcomes and is cost effective.31 Future studies are underway to expand the SBML educational model to other conversations and learners, as well as to evaluate both long-term retention of skill, transfer of skills to the bedside, and effect on clinical outcomes.

Conclusions

We have reported outcomes of the first SBML communication skills curriculum that we are aware of to be successfully embedded as a component of a medical school clerkship. This approach resulted in uniform skill acquisition in the fundamental task of BBN. This approach not only raises the bar for improving outcomes for communication skill training, but it also represents an important step toward creating a robust assessment system that ensures all medical school graduates possess the skills needed to safely enter residency.

Acknowledgments:

The authors would like to acknowledge Drs. Douglas Vaughan and Kevin O’Leary for their support and encouragement of this work. The authors thank the Northwestern University Feinberg School of Medicine medical students for their dedication to education and patient care.

References

1. Tulsky JA, Beach MC, Butow PN, et al. A research agenda for communication between health care professionals and patients living with serious illness. JAMA Intern Med. 2017;177:1361–1366.
2. Dietz I, Borasio GD, Molnar C, et al. Errors in palliative care: Kinds, causes, and consequences: A pilot survey of experiences and attitudes of palliative care professionals. J Palliat Med. 2013;16:74–81.
3. Levinson W, Roter DL, Mullooly JP, Dull VT, Frankel RM. Physician-patient communication. The relationship with malpractice claims among primary care physicians and surgeons. JAMA. 1997;277:553–559.
4. Zinn C. 14,000 preventable deaths in Australian hospitals. BMJ. 1995;310:1487.
5. Rubenfeld GD, Curtis JR; End-of-Life Care in the ICU Working Group. End-of-life care in the intensive care unit: A research agenda. Crit Care Med. 2001;29:2001–2006.
6. Mularski RA, Curtis JR, Billings JA, et al. Proposed quality measures for palliative care in the critically ill: A consensus from the Robert Wood Johnson Foundation Critical Care Workgroup. Crit Care Med. 2006;34(suppl 11):S404–S411.
7. Nelson JE, Mulkerin CM, Adams LL, Pronovost PJ. Improving comfort and communication in the ICU: A practical new tool for palliative care performance measurement and feedback. Qual Saf Health Care. 2006;15:264–271.
8. Clarke EB, Curtis JR, Luce JM, et al.; Robert Wood Johnson Foundation Critical Care End-Of-Life Peer Workgroup Members. Quality indicators for end-of-life care in the intensive care unit. Crit Care Med. 2003;31:2255–2262.
9. Nelson JE, Puntillo KA, Pronovost PJ, et al. In their own words: Patients and families define high-quality palliative care in the intensive care unit. Crit Care Med. 2010;38:808–818.
10. Halpern SD, Becker D, Curtis JR, et al.; Choosing Wisely Taskforce; American Thoracic Society; American Association of Critical-Care Nurses; Society of Critical Care Medicine. An official American Thoracic Society/American Association of Critical Care Nurses/American College of Chest Physicians/Society of Critical Care Medicine policy statement: The Choosing Wisely® Top 5 list in Critical Care Medicine. Am J Respir Crit Care Med. 2014;190:818–826.
11. Back AL, Arnold RM, Baile WF, et al. Efficacy of communication skills training for giving bad news and discussing transitions to palliative care. Arch Intern Med. 2007;167:453–460.
12. Johnson J, Panagioti M. Interventions to improve the breaking of bad or difficult news by physicians, medical students, and interns/residents: A systematic review and meta-analysis. Acad Med. 2018;93:1400–1412.
13. McGaghie WC. Mastery learning: It is time for medical education to join the 21st century. Acad Med. 2015;90:1438–1441.
14. Rosenbaum L. Cursed by knowledge—Building a culture of psychological safety. N Engl J Med. 2019;380:786–790.
15. McGaghie WC. Evaluation apprehension and impression management in clinical medical education. Acad Med. 2018;93:685–686.
16. Vermylen JH, Wood GJ, Cohen ER, Barsuk JH, McGaghie WC, Wayne DB. Development of a simulation-based mastery learning curriculum for breaking bad news. J Pain Symptom Manage. 2019;57:682–687.
17. Schaefer KG, Chittenden EH, Sullivan AM, et al. Raising the bar for the care of seriously ill patients: Results of a national survey to define essential palliative care competencies for medical students and residents. Acad Med. 2014;89:1024–1031.
18. Englander R, Cameron T, Ballard AJ, Dodge J, Bull J, Aschenbrener CA. Toward a common taxonomy of competency domains for the health professions and competencies for physicians. Acad Med. 2013;88:1088–1094.
19. Englander R, Flynn T, Call S, et al. Toward defining the foundation of the MD degree: Core entrustable professional activities for entering residency. Acad Med. 2016;91:1352–1358.
20. Sharma RK, Szmuilowicz E, Ogunseitan A, et al. Evaluation of a mastery learning intervention on hospitalists’ code status discussion skills. J Pain Symptom Manage. 2017;53:1066–1070.
21. McGaghie WC, Issenberg SB, Barsuk JH, Wayne DB. A critical review of simulation-based mastery learning with translational outcomes. Med Educ. 2014;48:375–385.
22. Baile WF, Buckman R, Lenzi R, Glober G, Beale EA, Kudelka AP. SPIKES—A six-step protocol for delivering bad news: Application to the patient with cancer. Oncologist. 2000;5:302–311.
23. McGaghie WC, Barsuk JH, Wayne DB. AM last page: Mastery learning with deliberate practice in medical education. Acad Med. 2015;90:1575.
24. Skeff KM, Stratos GA, Berman J, Bergen MR. Improving clinical teaching. Evaluation of a national dissemination program. Arch Intern Med. 1992;152:1156–1161.
25. Lypson ML, Frohna JG, Gruppen LD, Woolliscroft JO. Assessing residents’ competencies at baseline: Identifying the gaps. Acad Med. 2004;79:564–570.
26. McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011;86:706–711.
27. Sharma RK, Jain N, Peswani N, Szmuilowicz E, Wayne DB, Cameron KA. Unpacking resident-led code status discussions: Results from a mixed methods study. J Gen Intern Med. 2014;29:750–757.
28. Barsuk JH, Cohen ER, Potts S, et al. Dissemination of a simulation-based mastery learning intervention reduces central line-associated bloodstream infections. BMJ Qual Saf. 2014;23:749–756.
29. Wayne DB, Moazed F, Cohen ER, Sharma RK, McGaghie WC, Szmuilowicz E. Code status discussion skill retention in internal medicine residents: One-year follow-up. J Palliat Med. 2012;15:1325–1328.
30. Barsuk JH, McGaghie WC, Cohen ER, Balachandran JS, Wayne DB. Use of simulation-based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit. J Hosp Med. 2009;4:397–403.
31. Cohen ER, Feinglass J, Barsuk JH, et al. Cost savings from reduced catheter-related bloodstream infection after simulation-based education for residents in a medical intensive care unit. Simul Healthc. 2010;5:98–102.
Copyright © 2020 by the Association of American Medical Colleges