The convergence of the recently released Accreditation Council for Graduate Medical Education (ACGME) milestones with the concept of entrustable professional activities (EPAs) presents an opportunity to change the way medical educators evaluate and entrust resident physicians.1,2 EPAs are activities essential to the practice of medicine that educators progressively entrust learners to perform. As learners develop from being able to observe an activity, to doing it under supervision, to performing it independently, their supervisor must make a decision about whether he or she trusts them to perform the activity safely and well. An EPA takes this development of trust and explicitly codifies it using competency and milestone language.
Residency programs are required to ensure their residents’ readiness for independent practice.2 However, this determination is too often based on insufficient direct observation of behaviors and inadequate assessment methods that focus on specific dimensions of residents’ abilities rather than their integrated performance.3,4 EPA assessment allows for the evaluation of whole clinical tasks, providing an opportunity to observe, and intentionally entrust, residents to practice important clinical tasks independently.1
The ACGME, beginning in June 2014, requires reporting on resident performance in selected milestones and incorporation of EPAs into training programs.5 Although there is a growing literature describing the concept of EPAs and their potential use in medical education, to our knowledge there are few descriptions of the development of EPA assessments and only one describing their implementation in residency programs.1,6,7 In this article, we share our model for developing an assessment of a handoff EPA within our internal medicine and combined internal medicine–pediatrics programs at the University of Minnesota Medical School. We believe the process we describe can serve as a model to other residency programs interested in developing EPA assessments.
Development of the Handoff EPA Assessment
The educational leadership of the internal medicine and internal medicine–pediatrics residency programs developed the handoff EPA assessment in 2012 following the nine steps described below (Table 1). Of note, the order of the steps might change depending on the EPA and the local circumstances, but addressing the core concepts remains important.
Selecting the EPA
The first step in our process was to define which EPA to pilot. Through multiple discussions, the educational leadership group selected handoffs. Although our ultimate goal is to develop EPAs representing the full spectrum of behaviors essential to becoming a practicing physician, the selection of a single initial EPA, with which we had experience, provided us the opportunity to learn from our experience and refine our process before developing additional EPAs.
A brief review of the literature supported that handoffs are an important professional activity, and that safe handoffs are essential for patient safety.8–11 With the implementation of residency duty hours restrictions and the associated increased transfers of care, the ACGME has emphasized the handoff as a critical skill for residents.2 Additionally, the specialties of internal medicine and pediatrics have both identified the handoff as one of the core EPAs using the framework of developmental milestones.12,13
At teaching hospitals, interns often have primary responsibility for handing off the team for cross-coverage.14 Despite this, most programs do not have a system in place to holistically assess that residents possess the required knowledge base, communication, and system skills to perform a handoff in the clinical environment.14 Given the absence of any system, interns are often de facto entrusted with this critical activity.
The educational group thus felt that the handoff would be an ideal EPA to assess because of its importance to patient safety, the need for residents to be competent in handoffs early in their training, and the belief that our current assessment was inadequate.
Evaluate the current environment for learning, practicing, and assessing skills that relate to the EPA
University of Minnesota residents receive instruction in handoffs as part of their intern orientation. The internal medicine and medicine–pediatrics interns all participated in a 90-minute handoff workshop.15 After orientation, interns participate in handoffs during all of their inpatient rotations under the supervision of senior residents. No formal assessments of handoffs were in place prior to the implementation of this EPA assessment.
Identify and address challenges in the learning environment to assessing the EPA
We considered both the context and culture of the learning environment because addressing both areas is critical to successful implementation of an EPA.
Two principal challenges in the learning environment that needed to be addressed before implementation of the EPA assessment were system factors and faculty time.
System factors may either impede or facilitate the evaluation of handoffs. Necessary systems include an electronic medical record (EMR) or other system that facilitates transmission of essential data. Also important is a scheduling system that facilitates the timing for the handoffs to occur as well as a place for the handoff that is distraction-free with adequate time allowed.
We resolved this challenge by the time of implementation by fully operationalizing our EMR and arranging a standardized time and quiet place for the handoff to occur.
Faculty are seldom present during the handoff.14 Potential reasons include that handoffs often occur without a schedule, making it difficult for faculty to know when and where they should be present. Additionally, faculty may perceive that handoffs are a “resident-level” activity and that they are superfluous to the process. Finally, ever-present patient care and administrative pressures make every minute count for faculty.
We resolved this challenge by selecting the general medicine wards at our main teaching hospital as the site to assess the interns in the handoff EPA. All interns rotate through this site, and our hospitalists provide in-house 24-hour coverage and are available to observe and assess handoffs. Chief residents are also available to assess the afternoon handoff. We established a standard time and place for handoffs, which facilitated faculty being able to observe handoffs more consistently.
Additionally, hospitalist leadership was interested in improved handoffs from a patient safety standpoint. This created an alignment of the residency goal of increased observation and the hospitalists’ goal of improved patient care. Coupled with our new EPA assessment form, this made for a valuable use of hospitalist time.
Determine the components of the EPA to be assessed
We performed a literature review using the search terms “hand-off” OR handoff OR “sign-out” OR signout, which returned 662 articles. Forty-eight were related to physician handoffs in an educational setting on inpatient services. A subset of these were identified that addressed key components to successful handoffs.16–20 These articles informed subsequent discussion with educational leadership and hospitalist educators, resulting in identification of eight domains (Appendix 1).
Determine the key characteristics of the tools needed to assess the EPA and evaluate existing resources against those criteria
We determined that we needed a milestone-based tool allowing for guided direct observation and assessment of learner developmental progression. We searched PubMed and MedEdPortal and were unable to find a suitable tool fulfilling these criteria.
If assessment tools do not exist, develop the needed tool(s)
We developed a milestone-based handoff observation assessment form that includes behavioral descriptors (Appendix 1) designed to be completed by observing faculty members or chief residents. We used the American Board of Internal Medicine (ABIM) and American Board of Pediatrics (ABP) milestone documents as well as the Dreyfus model of progression toward competency as a primary framework.12,13,21 Quality improvement, aerospace, and teamwork literature was used to refine the framework.13–24 This framework and the eight previously identified domains were used to create a rubric of domains and a progression of observable behaviors.
Using ten Cate’s five levels of entrustment, where level 1 indicates unable to perform and level 5 indicates able to teach and supervise others, we described the expected learner behaviors at each level of entrustment for each domain (Appendix 1).1 This iterative process began with the medicine–pediatrics program director drafting descriptions of each level within each domain based on previous experience as an educator and observing resident development of handoff skills. We refined the behavioral descriptors over several meetings with our education and hospitalist groups. We could not identify five distinct levels of observable behaviors for several domains, so we collapsed two or more levels together for some domains. The resulting draft was reviewed, refined, and approved by education leadership.
The goal of this step is to develop activity-specific anchors that are developmental and allow the observer to provide formative feedback to the learner. This formative feedback should serve as a roadmap to the learner with direction as to what is needed to move to the next level. This “guided gestalt” approach should provide sufficient guidance to the observer and the learner without being overly prescriptive.
Determine the criteria for advancement
During the development phase of the EPA assessment, the resident handoff assessment has been used for formative feedback. Once fully implemented, it will be used to make decisions about advancing residents through the levels of entrustment. Assessment forms from the development phase, the data from rater training, as well as discussions with the faculty observers and the educational leadership will inform decisions about criteria for advancement at each level of entrustment. The final entrustment decisions will be based on the observer’s assessment of the resident’s performance using an assessment form, as well as the resident’s self-assessment. Although individual domains are assessed, the ultimate decision is whether the resident can hand off a patient panel at a given level of entrustment.
Map milestones to the EPA
The next step was to map the behavioral descriptors to the ABP and ABIM reporting milestones. The descriptors, whose development was informed by the ABP and ABIM curricular milestones, serve as handoff-specific milestones that relate to the more general language of the ABP and ABIM milestones. The mapping allows for a common language and reporting to the ACGME while preserving the specificity of the behavioral descriptors to the handoff process. For example, the descriptor “Succinctly prioritized all relevant patients issues, including those related to patient’s trajectory, and potential system issues. Altered presentation based on receiver” was mapped to the reporting milestones ICS-2, “Consistently and actively engages in collaborative communication with all members of the team” and “Verbal, non-verbal and written communication consistently acts to facilitate collaboration with the team to enhance patient care.”
This process was done iteratively and was reviewed by the educational leadership to ensure that the relationship between the descriptors and the mapped terms was appropriate. We completed our mapping to the ABP and ABIM reporting milestones after we developed our assessment tool, as the release of the reporting milestones happened after we developed our tool. Going forward, this mapping step could also occur at the time of developing the EPA-specific behavioral descriptors.
Before implementation, educational leadership met with the hospitalist division and the internal medicine chief residents to discuss and finalize the observation and assessment process. Two months into the pilot project, we conducted a pilot rater training session with the chief residents. The chief residents watched videotaped handoffs, assessed the handoff using the assessment form, and then discussed ratings until consensus was reached. Subsequently, we conducted the same rater training session for the hospitalists division.
Implementation of the Handoff EPA Assessment
Beginning with the first rotation of 2012–2013, interns were observed and assessed giving handoffs both from the long call to night float and night float to the primary team on the general medicine wards at our main teaching hospital. Handoffs were observed by a faculty member or chief resident at a set time and place. The observer completing the assessment form used it to provide the immediate feedback. Forms were collected at the end of each handoff session.
During the pilot phase from June 2012 to January 2013, the majority of University of Minnesota Medical School internal medicine and combined medicine–pediatrics interns (25/32; 78%) who rotated through the general medicine services were assessed. In total, 105 independent assessments of resident handoffs were made. The number of assessments completed for each intern ranged from 1 to 12, with an average of 4 for all interns. There were 18 observers including 16 faculty members and 2 chief residents (4 assessments did not report whether the observer was a faculty member or chief resident). The chief residents observed the majority of the handoffs (66/101; 65%).
No interns were assessed at level 1 for the handoff EPA. Sixty percent of observations of intern handoffs (56/93) were assessed at levels 2 and 3. Thirty-nine percent (36/93) were assessed at level 4, and 1 handoff was assessed at level 5. Of the 19 interns who were observed more than once, 13 of the interns demonstrated improvement over multiple observations. This study was reviewed and approved by the University of Minnesota institutional review board.
Continuous quality improvement cycle
After the first rotation, the observers provided feedback on the observation process and assessment form to the working group. Their suggested changes to the behavioral descriptors were incorporated into a second iteration of the assessment form. This process was repeated after each subsequent rotation. No further requests for changes to the behavioral descriptors were made after the third rotation, suggesting that the observers found the behavioral descriptors sufficiently accurate to be able to locate residents’ performance in each of the domains.
We have learned several important lessons from developing and piloting a handoff EPA assessment. The first is the value of selecting a clinical skill that is both important to our residents’ development as physicians and integral to the safety of our hospitalized patients. Because this EPA was important both to our residency program and to the hospitalist group who would be per forming the assessments, we were able to create a system that could overcome the barriers to evaluating handoffs.
The second lesson learned is the importance of an iterative process during development and implementation. Our process of review and refinement at each step produced an evaluation and entrustment process that is more meaningful to residents and observers. The third and related lesson is that by involving the observers in the implementation process of determining the implementation process and refining the assessment form, they have a greater commitment to observing and assessing resident handoffs. We are encouraged by the active participation of the observers in the continuous improvement of the process and their continued willingness to observe handoffs.
Creating a fixed time and place for hand offs allowed observations to occur more consistently. Chief residents and attending physicians now see assessing handoffs as part of their job responsibility as an educator. We also noted that the milestone-based assessment form em powered observers to provide real-time feedback to residents as they were able to use the behavioral descriptors.
Finally, when considering the criteria for advancement through the levels of entrustment, we determined that two entrustment levels were high-stakes decisions with consequences for the resident’s participation in handoffs. First, we determined that if a resident is assessed in any domain at level 1 (unable to perform), then immediate remediation is required, and the resident cannot participate in handoffs until the remediation is completed. Similarly, we determined that advancement to level 4 (perform independently) would be the final level of entrustment evaluated and monitored by the residency program. Once residents achieved level 4, they would be fully entrusted to practice independently. We deemed level 5 as beyond the expectations of residency training.
Entrustment decisions at levels 2 and 3 were determined to be lower-stakes decisions with no consequences for residents’ ability to participate in handoffs. In practice, there is little difference between a resident assessed at level 2 (direct supervision) as opposed to level 3 (indirect supervision) in terms of supervision. What is most important at these two levels is providing feedback about behaviors to improve. This pattern of assessing some levels of entrustment as high stakes, whereas others are lower stakes, may be present for all complex activities. As EPA assessments become validated in the literature, this potential pattern should be tested. Additionally, our collapsing of two or more levels together within a domain may suggest that although five levels of entrustment are important for overall EPA entrustment decisions, there may not be five distinguishable levels for each domain or component within an EPA.
Results from our pilot phase suggest that, despite detailed behavioral descriptors and one rater training session, considerable variability between raters persists. It may be worthwhile to target rater training at the high-stakes assessments at levels 1 and 4. Ongoing faculty development will be necessary as a refresher and as new faculty members arrive.
Process of entrustment
There are critical questions to be considered in the process of entrustment including the number of assessments required at a particular level to advance. Will the assessment forms be reviewed by program directors at some fixed point to advance residents? Or will this process happen in real time once a resident meets the advancement criteria? Will there be any further assessments once entrustment occurs? What are the practical consequences of entrustment? Are these tools valid for assessment of EPAs? These questions must be answered before the process and promise of intentional entrustment can be fully realized.
The sustainability of educational innovations is often fragile because of clinical demands on both residents and faculty members. A commitment from multiple stakeholders including faculty members, chief residents, and residents will be essential to sustaining our handoff EPA. Handoff observation is now part of the job description for the both hospitalists and chief resident, yet the majority of handoffs were observed by the chief residents. This suggests that direct observation by faculty members remains a challenge.
Our handoff EPA and milestones-based assessment creates a framework that allows for both formative feedback to the learner and programmatic decisions about entrustment. The result is more meaningful assessment and tracking of residents’ evolving competence in handoffs and intentional entrustment of this skill. The process of EPA development we have described may serve as a model for EPA development for other skills and specialty areas.
Acknowledgments: The authors wish to thank Alisa Duran-Nelson, MD, Daisy Nieto, MD, and Andrew Olson, MD, for their support and data collection.
1. Ten Cate O, Scheele F. Competency-based postgraduate training: Can we bridge the gap between theory and clinical practice? Acad Med. 2007;82:542–547
2. . Accreditation Council for Graduate Medical Education 2012 Milestones. http://www.acgme-nas.org/milestones.html
. Accessed March 23, 2014
3. Epstein RM. Assessment in medical education. N Engl J Med. 2007;356:387–396
4. Van der Vleuten CPM, Schuwirth LWT. Assessing professional competence: From methods to programmes. Med Educ. 2005;39:309–317
5. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system: Rationale and benefits. N Engl J Med. 2012;366:1051–1056
6. Mulder H, Ten Cate O, Daalder R, Berkvens J. Building a competency-based workplace curriculum around entrustable professional activities: The case of physician assistant training. Med Teach. 2010;32:e453–e459
7. Hauer KE, Soni K, Cornett P, et al. Developing entrustable professional activities as the basis for assessment of competence in an internal medicine residency: A feasibility study. J Gen Intern Med. 2013;;28::1110–1114.
8. Solet DJ, Norvell JM, Rutan GH, Frankel RM. Lost in translation: Challenges and opportunities in physician-to-physician communication during patient handoffs. Acad Med. 2005;80:1094–1099
9. Brennan TA, Localio AR, Leape LL, et al. Identification of adverse events occurring during hospitalization: A cross-sectional study of litigation, quality assurance, and medical records at two teaching hospitals. Ann Intern Med. 1990;112:221–226
10. Petersen LA, Brennan TA, O’Neil AC, Cook EF, Lee TH. Does housestaff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med. 1994;121:866–872
11. Arora V, Johnson J, Lovinger D, Humphrey HJ, Meltzer DO. Communication failures in patient sign-out and suggestions for improvement: A critical incident analysis. Qual Saf Health Care. 2005;14:401–407
12. American Board of Pediatrics Milestones Project. https://www.abp.org/abpwebsite/publicat/milestones.pdf
. Accessed March 23, 2014
13. American Board of Internal Medicine Milestones Framework. http://www.abim.org/milestones/public/
. Accessed March 23, 2014
14. Horwitz LI, Krumholz HM, Green ML, Huot SJ. Transfers of patient care between house staff on internal medicine wards: A national survey. Arch Intern Med. 2006;166:1173–1177
15. Aylward M, Vawter L, Roth C. An interactive handoff workshop to improve intern readiness in patient care transitions. J Grad Med Educ. 2012;4:68–71
16. Farnan JM, Paro JAM, Rodriguez RM, et al. Hand-off education and evaluation: Piloting the observed simulated hand-off experience (OSHE). J Grad Med Educ. 2010;25:129–134
17. Arora VM, Johnson JK, Meltzer DO, Humphrey HJ. A theoretical framework and competency-based approach to improving handoffs. Qual Saf Health Care. 2008;17:11–14
18. Derienzo CM, Frush K, Barfield ME, et al. Handoffs in the era of duty hours reform: A focused review and strategy to address changes in the Accreditation Council for Graduate Medical Education common program requirements. Acad Med. 2012;87:403–410
19. Riesenberg LA, Leitzsch J, Little BW. Systematic review of handoff mnemonics literature. Am J Med Qual. 2009;24:196–204
20. Arora V, Johnson J. A model for building a standardized hand-off protocol. Jt Comm J Qual Patient Saf. 2006;32:646–655
21. Carraccio CL, Benson BJ, Nixon LJ, Derstine PL. From the educational bench to the clinical bedside: Translating the Dreyfus developmental model to the learning of clinical skills. Acad Med. 2008;83:761–767
22. Patterson ES, Wears RL. Patient handoffs: Standardized and reliable measurement tools remain elusive. Jt Comm J Qual Patient Saf. 2010;36:52–61
23. Reader TW, Flin R, Mearns K, Cuthbertson BH. Team situation awareness and the anticipation of patient progress during ICU rounds. BMJ Qual Saf. 2011;20:1035–1042
24. Patterson ES, Woods DD. Shift changes, updates, and the on-call architecture in space shuttle mission control. Comput Support Coop Work. 2001;10:317–346
Evaluation Tool Form for Assessing Eight Domains of Patient Hand-Offs, Developed at the University of Minnesota Medical School for an Entrustable Professional Activity Assessment, 2012
Note: If “Cannot perform” is circled for ANY items, notify the chief resident.