Secondary Logo

Journal Logo

Innovation Reports

Using a Modified A3 Lean Framework to Identify Ways to Increase Students’ Reporting of Mistreatment Behaviors

Ross, Paula T. PhD; Abdoler, Emily MD; Flygt, LeeAnne MD; Mangrulkar, Rajesh S. MD; Santen, Sally A. MD, PhD

Author Information
doi: 10.1097/ACM.0000000000002033
  • Free

Abstract

Problem

Mitigating learner mistreatment has remained elusive within medical education.1,2 Further, the proportion of students who experience mistreatment behaviors is significantly higher than the proportion of students who report mistreatment behaviors.3 In 2011, the Association of American Medical Colleges (AAMC) Graduation Questionnaire (GQ) defined mistreatment as any behavior that the beholder believes “shows disrespect for the dignity of others and unreasonably interferes with the learning process.” Since then, the AAMC GQ has continued to ask students about mistreatment behaviors, such as if they were ever threated with physical harm, subjected to racially or ethnically offensive remarks or names, or publicly humiliated (see the AAMC GQ for the full list).3 Unfortunately, the impact that mistreatment can have on learners, including posttraumatic stress, depression, lack of confidence, burnout, and specialty choice decisions, has been well documented.4–6 Identifying ways to improve students’ reporting of these incidents is one strategy for increasing opportunities to achieve resolution and prevent future occurrences. In this Innovation Report, we describe our efforts to decrease the difference between the proportion of fourth-year medical students at our medical school who experience mistreatment and the proportion of those who report mistreatment using a modified A3 Lean framework.

Approach

At the University of Michigan Medical School, we applied a modified A3 Lean framework to examine medical student reporting of mistreatment behaviors at our institution in 2013–2016. Initially introduced by Toyota Motor Corporation to improve manufacturing processes,7 Lean strategies have also been adopted in the health care setting to improve patient safety, patient satisfaction, and quality of care.8 Most Lean strategies use the A3, a systematic approach to problem solving and continuous improvement.7 The label “A3” is derived from an international metric paper size (11 × 17); in an A3 Lean framework, one constructs boxes that describe the practice problem and tell the story of the improvement process.7 This stepwise approach involves outlining the background to establish the context of the problem, describing the beginning condition and what is known about the program, identifying the goal or desired outcome, analyzing causes of the problem, providing proposed countermeasures or action plan items for improvement, and creating follow-up plans.7 The steps we used to address the underreporting of mistreatment behaviors at our institution from 2013 to 2016 are described below (see also Chart 1).

Chart 1
Chart 1:
Steps Used to Address the Underreporting of Mistreatment at the University of Michigan Medical School, 2013–2016,a Using a Modified A3 Lean Framework

Background

Data from the AAMC GQ indicate that nationally, fourth-year medical students report less than half of the mistreatment behaviors they experience.3 In 2013, the reporting of mistreatment behaviors by fourth-year medical students from all schools (18.7%) considerably lagged behind their experiences of mistreatment (42.1%).3

Beginning situation

An examination of the University of Michigan Medical School fourth-year medical student responses to two questions on the 2013 AAMC GQ—(1) “Did you personally experience any of the listed behaviors, excluding publicly embarrassed?” (44.1%) and (2) “Did you report any of the behaviors listed above to a designated faculty member or a member of the medical school administration empowered to handle such complaints?” (12.7%)—revealed that the inconsistency was not only a national phenomenon but also a problem at our institution.9 Of the 143 University of Michigan Medical School students who experienced a mistreatment behavior, only 63 reported the incident.9 Thus, the proportion of our students who experienced mistreatment in 2013 (44.1%) was significantly higher than the proportion of students who reported mistreatment (12.7%).9 Using a test of independent proportions, a difference of 31.4% ([95% confidence interval = 17.78, 42.47]; P < .001) was found. This difference served as the beginning state of our problem. We found this difference particularly unsettling because as an educational institution we aim to provide all learners with a supportive environment that facilitates their learning and professional development.

Goal

Our goal was to increase students’ reporting of mistreatment behaviors to more accurately reflect their experiences. Specifically, the goal was to reduce by half the difference between students’ experiences and their reporting of mistreatment by the release of the 2018 AAMC GQ.

Analysis

We sought to identify current reporting pathways (how and where students report) and potential barriers to reporting (why they do not report). We reviewed available, objective data from the AAMC GQ and institutionally collected data. Our students’ responses from the 2013 AAMC GQ question “If there were incidents of these behaviors that you did not report, why didn’t you report them?” indicated that students did not report mistreatment behaviors because the incident did not seem important enough to report (54.0%), they did not think anything would be done (54.0%), they feared reprisal (41.3%), they did not know what to do (14.3%), or they resolved the issue independently (4.8%).9

The institutionally collected data used in our analysis included student responses to curriculum evaluation questions about mistreatment and the learning environment from courses, clerkships, annual class evaluations, and preclinical and clinical faculty and resident teaching evaluations from 2013 to 2015. Additionally, in 2013, the Medical Student Learning Environment Task Force (MSLETF) launched the first midyear survey of third-year students about mistreatment experiences and led a student focus group to explore student reporting of mistreatment. (See Table 1 for more information on the institutionally collected data.) We also used quarterly learning environment reports generated from our institution’s mistreatment website from 2013 to 2015. This internal website serves as a centralized hub for students to report episodes of misconduct and incivility. The website is monitored by a psychiatrist faculty member (with no responsibility for student teaching), and any concerning comments regarding a student’s safety are dealt with immediately.

Table 1
Table 1:
Data Sources, Data Collected, and Data Dissemination for Mistreatment and Learning Environment Data, University of Michigan Medical School, 2013–2016

Over the course of two years (2013–2014), the MSLETF and school leadership (associate dean and assistant dean) met regularly to review all data to determine areas of improvement as well as areas where there were continued differences. The data revealed that students were unaware of where to report incidents and there was an inconsistency in terms of to whom incidents were reported. Although students identified the mistreatment website as the one mechanism for official reporting, many reports were made less formally either on a faculty or resident evaluation, curriculum evaluation, or directly to the clerkship director. A review of informal reports revealed that no process existed to formally monitor information reported in these ways or ensure that incidents were appropriately addressed. This made tracking and follow-through challenging.

Using a modified A3 Lean framework, we iteratively identified three overarching causes for the difference between students’ experiences and reporting of mistreatment: (1) students not knowing where to report, (2) students’ perceptions of not having a safe place for reporting, and (3) students lacking confidence in the action that would be taken on their report.

Countermeasures/action plan

Below are the countermeasures/action plan items we developed through numerous meetings with the MSLETF and school leadership to address the difference between students’ experiences and their reports of mistreatment behaviors. This team met almost quarterly and often included other stakeholders such as clerkship directors, student council members, and department chairs to identify possible approaches to improve underreporting. Our improvement campaign began by targeting strategies that addressed the reasons given for not reporting incidents on the AAMC GQ.

1. Engage stakeholders (students).

School leadership recognized the need for student involvement in our reporting improvement initiatives. With the guidance and advice of the assistant dean for student services, we created the Medical Student Mistreatment Task Force, led by members of the student council. This task force served as a liaison between the medical students and the administration, creating a space for dialogue and identification of issues and solutions. The group was later renamed the MSLETF to better convey its broader mission of improving the learning environment.

2. Reduce students’ fear of reprisal.

We sought to address the fear of reprisal by updating our mistreatment website. As an additional layer of protection, the website allows students the option to embargo reporting the incident until after completion of the clerkship, the end of the year, or after graduation. The most recent iteration on the website encourages students to report professional as well as unprofessional behavior.

3. Inform students of resolution process.

In 2013, over half (54.0%) of our students reported that they did not think anything would be done about a reported incident of mistreatment.9 To address this, we created a standard work process to improve consistency of mistreatment reporting as well as to give students the opportunity to receive information on how their report was resolved. Under this new process, all submitted reports are processed by the senior associate dean of medical education to facilitate timely and appropriate evaluation and follow-up. Additionally, the MSLETF and the assistant dean of student affairs share the reported cases and the actions taken on them with third-year students twice a year to further demonstrate that the administration is acting on reported concerns. When possible, the senior associate dean of medical education provides the reporting student with information about the actions taken on his/her report.

4. Reinforce the importance of reporting.

Because in 2013 approximately half of our students indicted that they did not feel many incidents were important enough to report,9 the senior associate dean for education and the associate dean for medical student education developed scenarios depicting various types of mistreatment to trigger a rich and open discussion around mistreatment behaviors and reporting. The scenarios were presented to third-year students to help them work through many of the nuances of mistreatment, particularly in the clinical setting (on the wards, rounds, clinical seminars), and to remind them of official reporting procedures. Additionally, third- and fourth-year students present a mistreatment session at clerkship orientation to encourage reporting. Further, to improve students’ confidence in reporting, twice each year we disseminate reported cases of mistreatment and the actions taken to address the incidents to third-year students. We also host periodic sessions, open to all students, in which conversations about inclusivity and equity in the learning environment are directly addressed with the associate dean for medical student education.

5. Inform students about reporting system and mistreatment policies.

We hold annual information sessions during the academic year to inform students of what to do should they encounter a mistreatment incident. Additionally, at the beginning and midpoint of each school year, we inform students in person and electronically about the reporting system and resolution process. Our school’s results on the 2013 AAMC GQ indicated that only 5.0% of graduating medical students were unaware of the mistreatment policy9; therefore, we continued to widely disseminate the policy through multiple modalities, including the medical student intranet and e-mail. Individual clerkship directors also communicate the policy to faculty, residents, and staff involved in medical student education each year.

Outcomes

With our use of a modified A3 Lean framework, the proportion of students reporting mistreatment at our institution has increased 21.4% between 2013 and 2016 (see Figure 1). On the basis of these data, it appears that we were able to achieve our primary goal of reducing the difference between students’ experiences and reports of mistreatment.

Figure 1
Figure 1:
Comparison of University of Michigan Medical School fourth-year medical student responses to two mistreatment behavior questions on the Association of American Medical Colleges Graduation Questionnaire, 2013–2016.

In 2016, students indicated that they did not report mistreatment behaviors because the incident did not seem important enough to report (67.4%), did not think anything would be done (41.9%), were afraid of reprisal (37.2%), did not know what to do (2.3%), or resolved the issue themselves (11.6%).10 Compared with the proportions for these in 2013, more students in 2016 indicated not reporting an incident because the incident did not seem important enough (54.0% in 2013) or because they resolved the issue on their own (4.8% in 2013)9; therefore, we continue to encourage students to report all incidents through the mistreatment website. These results have been shared with the school leadership (Table 1).

In contrast to the findings noted above, we also found that student responses to the AAMC GQ question “How satisfied are you with the outcome of having reported the behavior(s)?” indicated decreased satisfaction from 50.0% in 2013 to 21.4% in 2016.9,10 We are concerned about this finding and believe that this suggests that efforts to increase reporting have not been matched by demonstrable resolutions and improvement in the incidence of mistreatment. We believe that this finding highlights that the complexity of this problem requires a multifaceted approach, and that although reporting has increased, issues with the overall learning environment remain, especially in the clinical setting. In addition to continuing efforts to increase reporting, we will continue the dialogue between departments, the dean’s office, and students, particularly in regard to informing students about resolutions. Although satisfaction is not a direct reflection on actions taken, it is still important to consider students’ preferences in addressing the incidents they encounter.

Next Steps

We feel that using a modified A3 Lean framework helped us identify mistreatment reporting pathways and barriers to reporting. Our 2016 AAMC GQ results indicated an increase in reporting mistreatment; however, it is critical that we continue to review all available data and address incidents as they happen. We suggest using an A3 Lean framework as a way to approach improving student mistreatment reporting or other critical problems in medical education.

Additionally, because many incidents of mistreatment occur in the clinical years, we recognized the need to broaden the scope of the learning environment to include the larger health system. We have enlisted the support of our health system’s human resources department to present the inaugural grand rounds on improving the learning environment in 2016. The aim of this grand rounds was to clearly state formal and informal mechanisms for reporting episodes of uncivil behavior, including mistreatment, and to get information into the hands of those who can help improve the learning environment. As the audience for this session included residents, faculty, and other health professionals, we aimed to highlight that they too are members of the learning environment and share in the responsibility to avoid student mistreatment and report any incidents they witness.

We are also partnering with our health system’s human resources department to add questions about student mistreatment and civility to the annual employee engagement survey distributed to all 20,000 employees, use the executive coaching program as a source of remediation for faculty who have consistent unprofessionalism issues with learners, and develop critical conversations workshops for faculty and staff to promote an inclusive environment in the clinical setting.

Finally, as a result of the heightened visibility this initiative has brought to mistreatment, including an increase in reporting frequency, the academic medical center launched a Civility and Wellness Task Force, comprising a broad range of stakeholders, including physicians, nurses, staff, and medical students, in 2016. The purpose of this task force is to enhance civility, respect, and inclusivity in the clinical learning environment. This charge was also part of the expectations set for all senior leaders at the health system, including the dean, vice presidents, and department chairs in 2016.

Acknowledgments:

The authors would like to thank the University of Michigan Medical School Medical Student Learning Environment Task Force for their ongoing improvement efforts, Dr. John (Jack) E. Billi for insightful conversations, and Ms. Sara Weir for her efforts in the collection and dissemination of institutional mistreatment data.

References

1. Mavis B, Sousa A, Lipscomb W, Rappley MD. Learning about medical student mistreatment from responses to the medical school graduation questionnaire. Acad Med. 2014;89:705711.
2. Fried JM, Vermillion M, Parker NH, Uijtdehaage S. Eradicating medical student mistreatment: A longitudinal study of one institution’s efforts. Acad Med. 2012;87:11911198.
3. Association of American Medical Colleges. Medical School Graduation Questionnaire: 2013 All Schools Summary Report. https://www.aamc.org/data/gq/allschoolsreports [no longer available]. Accessed February 8, 2016.
4. Oser TK, Haidet P, Lewis PR, Mauger DT, Gingrich DL, Leong SL. Frequency and negative impact of medical student mistreatment. Acad Med. 2014;90:755761.
5. Richman JA, Flaherty JA, Rospenda KM, Christensen ML. Mental health consequences and correlates of reported medical student abuse. JAMA. 1992;267:692694.
6. Wilkinson TJ, Gill DJ, Fitzjohn J, Palmer CL, Mulder RT. The impact on students of adverse experiences during medical school. Med Teach. 2006;28:129135.
7. Shook J. Managing to Learn: Using the A3 Management Process to Solve Problems, Gain Agreement, Mentor and Lead. 2008.Cambridge, MA: Lean Enterprise Institute
8. Sobek DK, Smalley A. Understanding A3 Thinking: A Critical Component of Toyota’s PDCA Management System. 2008.New York, NY: Productivity Press
9. Association of American Medical Colleges. Medical School Graduation Questionnaire: 2013 University of Michigan Medical School Report. 2013.Washington, DC: Association of American Medical Colleges
10. Association of American Medical Colleges. Medical School Graduation Questionnaire: 2016 University of Michigan Medical School Report. 2016.Washington, DC: Association of American Medical Colleges
Copyright © 2017 by the Association of American Medical Colleges