Problem
The Accreditation Council for Graduate Medical Education (ACGME) milestones for internal medicine (IM) were written by physicians and thus may not reflect all the behaviors necessary for physicians to optimize their performance as a key member of an interprofessional team. Milestones are observable behaviors that describe the developmental progression of competence from the beginning of training to the proficiency necessary for independent practice.1 They are also used to guide curriculum development2 and to provide a framework for resident assessment.3 When assessed via cultural consensus analysis, patients, nurses, and physicians share a mental model of competence.4 However, these groups have not yet built milestone-based assessment tools together.
Because, as mentioned above, the current IM curricular and ACGME IM reporting milestones were developed by physicians, the interprofessional competencies (akin to the milestones rubric in IM) they include are not as robust as those developed by interprofessional teams. For example, the current curricular and reporting milestones include one milestone pertaining to the roles and responsibilities of providers on an interprofessional team, whereas the Interprofessional Education Collaborative Expert Panel report,5 which was developed by an interprofessional team, delineates nine individual interprofessional competencies. Furthermore, consultation with patients and families regarding their perception of key behaviors may also be integral in developing a comprehensive list of behaviors.6
As an ideal hospital discharge is inherently an interprofessional activity, we used teams of patients or family members, nurses, physician trainees, physician educators, and other staff (optional) (hereafter IP teams) to generate a list of the necessary behaviors (or milestones) perceived to be critical for trainee competence in the entrustable professional activity (EPA) of a safe and effective discharge of a patient from the hospital. We then used this list of behaviors to identify components of a safe and effective discharge and developed a discharge EPA assessment tool for direct observation.
Approach
In 2009, the Educational Research Outcomes Collaborative (E-ROC), a group of IM program leaders affiliated with the Alliance for Academic Internal Medicine,7 created and implemented training curricula for EPAs. EPAs are a means to translate competencies into clinical practice.8 Our first collaborative project focused on training for the essentials of ambulatory care.7 We chose our second project, the discharge EPA, for the complexity of the activity and also to engage nonphysician team members. The E-ROC faculty for the discharge EPA project consisted of E-ROC leaders and the site principal investigators (PIs) from each of the 11 participating IM programs (see below). The E-ROC faculty collaborated on a monthly conference call and in person biannually to develop the research protocol, analyze the data, and ultimately develop and pilot the discharge EPA tool.
As the ACGME milestones were written by physicians only, we wanted to look outside the ACGME framework for behaviors related to the discharge EPA. We used a qualitative, theory-generating approach to identify behaviors that are expected of a physician competent in the discharge EPA. From April to May 2013, we assembled IP team discussion groups, including patients and family members, at 11 diverse E-ROC IM programs (3 university, 8 community) to generate and prioritize behaviors for the discharge EPA.
During 90-minute group discussions, each IP team answered the question, “What are the behaviors of the expert physician when discharging a patient from the hospital?” Our goal was to develop a list that had more value for interprofessional teams than the current curricular and reporting milestones.
The site PIs invited a maximum of 12 participants to join the IP team discussion group. Each discussion group consisted of 2 to 3 participants from each category, including IM patients or family members, nurses from the IM floor, IM physician trainees, IM physician educators, and an optional category of other staff from the IM floor (e.g., social workers, pharmacists, physical therapists). The site PIs were asked to choose participants who were actively involved in the hospital discharge process on the IM unit, with an actively involved patient (or family member) defined as a patient (or family member of a patient) who had been discharged from the hospital within the last 6 months. Ninety-seven individuals participated in the group discussions: 16 patients, 11 family members, 20 nurses, 19 physician trainees, 23 physician educators, and 8 other staff members (including 1 pharmacist, 6 case managers, and 1 physical therapist).
As E-ROC leaders, we developed a facilitator script (see Supplemental Digital Appendix 1 at https://links.lww.com/ACADMED/A344) and, in a one-hour orientation, used it to teach the site PIs how to be facilitators for the IP team discussion groups. Although site PIs were not expert facilitators, they were coached on group brainstorming and creating an environment of inquiry. Site PIs were instructed to read directly from the script for the introduction of the discussion group, which explicitly encouraged equal participation from all participants to counter the potential for power imbalances due to medical hierarchy. For example, to reinforce the culture of inquiry and encourage patient and family member participation, the script reads:
Before getting started, I want to say something to the patients or family members or others here who may feel that their input is not as important as that of the medical doctors or nurses here today. Your opinions are so important for us to hear! We don’t think we can have a clear picture of good care without your voice, so I will be looking for opportunities during the session to be sure your opinions are clearly heard by the group.
The discussions were divided into the following activities: (1) define goals and create a safe environment, (2) develop a list of behaviors, and (3) prioritize those behaviors. Participants sat in a circle and were prompted to list physician behaviors needed during the days before discharge, the day of discharge, and after discharge. A scribe wrote all suggested behaviors on a white board. Participants were encouraged to think about behaviors by listening to ideas and allowing for an open process of augmenting and revising ideas. The group continued to list behaviors until no new ideas were generated. After this was complete, the group was asked to review the list and to combine behaviors that were duplicative and to separate behaviors that were distinct.
Participants were then guided to prior itize the behaviors. Every participant was asked to rank the behaviors from 10 to 0 (with 10 being the most important behavior for discharge and 0 being the least) by placing a sticky note with a number next to the behaviors on the white board. Thus, though participants ranked behaviors while gathered together at the white board, every participant individually ranked the group’s list of behaviors by importance with his/her own numbers. At the end of the prioritizing activity, the scribe documented the list of behaviors and participants’ rankings with deidentified participant information (i.e., patient #1, patient #2, nurse #1, and so on).
We received exempt status from the institutional review board of Baystate Medical Center for this research.
Outcomes
We collected and organized the behavior lists from all discussion groups. A total of 182 behaviors were listed, with lists consisting of between 10 and 29 behaviors. Many, but not all, of the behaviors overlapped across the 11 IM programs. Each behavior was also linked with a cumulative weighted prioritization value between 10 and 0.
In addition to providing us with the groups’ prioritized lists of behaviors, the site PIs reported their impressions and summarized comments provided by participants. For example, patients expressed gratitude for the chance to be part of the physician training process and have their perspective heard, and the nurses played an important role in bridging the gap between the patient experience and the physician perspective while also contributing the nursing perspective on physician training. Overall, the PIs described all participants as emerging from the activity with a new understanding of the complexity of training physicians for the discharge EPA.
Two of us (L.B.M., T.A.) separately batched the behaviors by theme (or components of a safe and effective discharge) and used the prioritization value to rank the components. Over the course of eight weekly phone calls, the two of us elicited opinions from the E-ROC faculty to ascertain disagreement or confirmation. Differences of opinion were vetted by the E-ROC faculty. This resulted in the identification of six components: medication reconciliation, discharge summary, patient/caregiver communication, team communication, active collaboration, and anticipation of posthospital needs (see Table 1).
;)
Table 1: Prioritization Value and Example Behaviors for the Six Components for a Safe and Effective Dischargea
After organizing the behaviors into com ponents, we returned to the prioritized behaviors from the discussion groups to identify specific, high-priority behavior examples for each component. For example, under the patient/caregiver communication component, listed behaviors included “uses teach back” and “gives education about the final diagnosis.” For each component, we assigned descriptor categories based on the behaviors listed by the IP teams. This set of descriptors was used to make an assessment tool for direct observation (see Supplemental Digital Appendix 2 at https://links.lww.com/ACADMED/A345). Similar to a prior E-ROC tool,7 this tool includes space for formative, sequential feedback and an overall competence statement. The tool also includes descriptions of the six components, their associated IM curriculum milestones (for competency reporting), related individual behaviors (from the lists generated by the discussion groups), and an entrustment scale. We distributed the tool to the 11 participating IM programs and made adjustments to it based on feedback from educators at these programs.
The goal of using this tool is to determine a trainee’s competence in the discharge EPA. The tool is intended for serial assessments of the trainee by either the attending physician or a senior trainee who functions as a junior educator. Instructions for the tool’s use are described on the back of the tool (see Supplemental Digital Appendix 3 at https://links.lww.com/ACADMED/A346). The tool specifically guides the educator to catch those moments of patient care throughout the hospital stay that relate to discharge behaviors, such as anticipating the discharge or communicating the final diagnosis to patients. During the moments of discharge care, the tool prompts the educator about the behaviors that trainees need to display to become entrusted with the activity of discharging a patient. Over time, trainee entrustment for each component of the safe discharge can be assessed and, by adding instructions on how to improve, advanced. The feedback tool is unique in that it is intended to record multiple observations of trainee progress over time. This exemplifies for trainees and educators that ongoing assessments in multiple areas are required to assess a trainee’s readiness to go from direct to indirect supervision during discharge.
Next Steps
The change to milestone-based assess ment for assessing competencies in training allows for the development of new models of evaluating trainees with new curricula and assessment tools. In parallel is the recognition that physicians can no longer provide safe patient care alone; they must work on patient-centered health care teams.9 We provide an exam ple of using an IP team, inclusive of patients and family members, to generate a list of behaviors and of using that list to inform an assessment tool for observing trainees as they work through patient discharges from the hospital in real time.
We learned that a group consisting of both patients and family members and health care professionals can work together to generate a list of physician behaviors related to an EPA—in this case the discharge EPA. With a uniform facilitator script and a one-hour orientation, the site PIs were able to guide groups to have an open exchange of ideas, express them to each other, and channel the group’s energy into a final product (i.e., a list of behaviors for safe and effective discharge). This work helps to extend patient and family engagement beyond patient care to curriculum devel opment. Patients and family members were able to recall their personal expe riences and add a valuable perspective to physician training. It was the patients in the group discussions that emphasized the importance of preparing patients days before the discharge and giving them anticipatory advice. Nurses were a critical bridge between the patient and family and the physician perspectives, and they augmented the physician perspective by thinking about care planning for the discharge. Finally, trainees articulated which areas of competence are lacking for their level of training.
We are currently evaluating trainee and educator perceptions of the assessment tool after implementation in 15 IM programs. Additional next steps include developing tools for other EPAs, as well as a broader evaluation of patient outcomes in the era of milestone-based assessment. Other disciplines and health professions could apply this method of establishing a list of relevant behaviors for any EPA with an IP team and using that list to develop an assessment tool.
The ACGME milestones were developed by expert physician educators, without patient and family or nurse input. These physician-centric milestones may be flawed in our inherently interprofessional-team-reliant health care workplace. The addition of the patient and family and nurse perspectives to the physician perspective may prove invaluable for training physicians to be patient-centered and team-based providers.
Acknowledgments: The authors would like to acknowledge the participation of the Educational Research Outcomes Collaborative site principal investigators including Aijaz Noor, Aurora Health Care, Milwaukee, Wisconsin; Emily Mallin, Banner Good Samaritan Medical Center, Phoenix, Arizona; Mitchell Black, Duke Medical Center, Durham, North Carolina; Dave Paje, Henry Ford Hospital, Detroit, Michigan; David Wininger, Ohio State University, Columbus, Ohio; Eric Dahms, Scripps Mercy Hospital, San Diego, California; Alvin Calderon, Virginia Mason Hospital & Medical Center, Seattle, Washington; and Sandhya Samavedam, Wright Center for Graduate Medical Education, Scranton, Pennsylvania.
References
1. Caverzagie KJ, Iobst WF, Aagaard EM, et al. The internal medicine reporting milestones and the next accreditation system. Ann Intern Med. 2013;158:557559.
2. Green ML, Aagaard EM, Caverzagie KJ, et al. Charting the road to competence: Developmental milestones for internal medicine residency training. J Grad Med Educ. 2009;1:520.
3. Iobst W, Aagaard E, Bazari H, et al. Internal medicine milestones. J Grad Med Educ. 2013;5(1 suppl 1):1423.
4. Smith CS, Hill W, Francovich C, et al. Developing a cultural consensus analysis based on the internal medicine milestones (M-CCA). J Grad Med Educ. 2011;3:246248.
5. Interprofessional Education Collaborative Expert Panel. Core Competencies for Interprofessional Collaborative Practice: Report of an Expert Panel. May 2011. Washington, DC: Interprofessional Education Collaborative;
http://www.aacn.nche.edu/education-resources/ipecreport.pdf. Accessed February 11, 2016.
6. Weinberger SE, Johnson BH, Ness DL. Patient- and family-centered medical education: The next revolution in medical education? Ann Intern Med. 2014;161:7375.
7. Meade LB, Caverzagie KJ, Swing SR, et al. Playing with curricular milestones in the educational sandbox: Q-sort results from an internal medicine educational collaborative. Acad Med. 2013;88:11421148.
8. ten Cate O, Snell L, Carraccio C. Medical competence: The interplay between individual ability and the health care environment. Med Teach. 2010;32:669675.
9. Weller J, Boyd M, Cumin D. Teams, tribes and patient safety: Overcoming barriers to effective teamwork in healthcare. Postgrad Med J. 2014;90:149154.
