Secondary Logo

Journal Logo

Research Reports

Medical Students’ Perspectives on Entrustment Decision Making in an Entrustable Professional Activity Assessment Framework: A Secondary Data Analysis

Caro Monroig, Angeliz M. MD, MPH1; Chen, H. Carrie MD, PhD2; Carraccio, Carol MD, MA3; Richards, Boyd F. PhD4; ten Cate, Olle PhD5; Balmer, Dorene F. PhD6; and the EPAC Study Group

Author Information
doi: 10.1097/ACM.0000000000003858

Abstract

Throughout medical training, supervisors routinely make decisions to entrust trainees with patient care activities. However, only recently have there been concerted efforts to make entrustment decisions explicit for trainee assessment using entrustable professional activities (EPAs). 1,2 In an EPA assessment framework, supervisors explicitly assess trainees based on their readiness to be entrusted with a specific activity. The outcome of that entrustment decision is the transfer of some or all of the responsibility to carry out that activity—or not—to the trainee. 3–5

Five factors that influence entrustment decisions have been identified in the literature: supervisor, trainee, supervisor–trainee relationship, clinical task, and clinical context. 6–19 As displayed in Table 1, each factor consists of several component parts. For example, the component parts of the trainee factor described in the literature may include clinical competence, truthfulness, willingness to take on new challenges, and openness toward patients.

Table 1
Table 1:
Summary of Components of the 5 Factors That Influenced Entrustment Decisions of Medical Students, From a Secondary Analysis of the Education in Pediatrics Across the Continuum Project, 2014–2018

These factors were derived almost exclusively from studies that report supervisors’ perspectives. 6–18 Despite trainees being key stakeholders in entrustment decisions, to our knowledge, there are few studies of resident trainee perspectives, and even fewer studies of student trainee perspectives. An exception is a study by Karp and colleagues that explored clerkship students’ perceptions of supervisor trust within an assessment framework where neither trust nor entrustment was explicitly tied to assessment. 19 Students reported the same 5 factors described above as contributing to feelings of trust, but they perceived trust as primarily something that happened to them. Students felt unable to address their experience of over- or undertrust and supervision, and often had to infer reasons for attendings’ inappropriate trust. While these findings help to close the gap in the literature about student trainee perspectives, they highlight another problem. There is a temptation to conflate trust (reflecting belief or confidence in the trainee) in the context of assessment frameworks that rate trainees’ performance with entrustment (reflecting the choice of the supervisor to transfer responsibility for a patient care activity to the trainee) in an EPA assessment framework. Decisions in the former may or may not relate to a trainee’s assessment outcome and typically do not result in a change in trainee responsibility. In contrast, decisions in the latter are the assessment outcome that signifies and results in the extent to which trainees are permitted to engage in practice at a specific level of supervision. 20,21

What traditional and EPA assessment frameworks have in common is their application to learning in clinical workplaces. Billett argues that the richness of learning depends not only on the kinds of workplace activities that trainees are afforded but also on trainees’ agentic engagement in workplace activities. While supervisors, clinical tasks, and clinical contexts can directly or indirectly afford workplace activities that serve as learning opportunities, trainees must be proactive and intentional about their own learning. 22–24 From this perspective, Karp and colleagues’ findings of students taking a passive stance toward their experience of over- and undertrust and supervision is concerning since actively negotiating entrustment decisions in an EPA assessment framework can influence students’ ability to engage in learning.

To expand understanding of student trainee perspectives and to target entrustment decision making within an EPA assessment framework, we (the research team) capitalized on existing qualitative data collected as part of an evaluation of a pilot project that successfully implemented an EPA assessment framework. In reviewing program evaluation data accumulated over a 4-year period, we posed 2 questions to the data in a secondary data analysis: Do medical students assessed using an EPA framework identify one or more components of the same 5 factors (supervisor, trainee, supervisor–trainee relationship, context, task) that influence entrustment? And given the importance of learner agency in learning from workplace activities, how might agency influence entrustment decision making, from the learner’s perspective?

Method

Research design

We conducted a secondary analysis of qualitative data that were systematically collected via focus groups as part of program evaluation. In this analysis, we did an in-depth investigation of students’ perceptions of entrustment, which emerged as a post hoc matter of interest. 25

The program evaluation project was reviewed by the institutional review board at The Children’s Hospital of Philadelphia and granted exemption from full review. Verbal consent for audiotaping was obtained at the start of each focus group.

Context

At the time of our secondary analysis, Education in Pediatrics across the Continuum (EPAC) was a pilot program sponsored by the Association of American Medical Colleges designed to advance trainees through medical school and pediatric residency training based on competence, as opposed to the traditional model based on time-in-training. 26,27 Specifically, EPAC used the Core Entrustable Professional Activities for Entering Residency as their EPA assessment framework for medical students in the program. 28 An existing entrustment supervision scale was used to track the developmental progress of EPAC students for each EPA and to determine students’ readiness for advancement in training, including the transition into internship. 29

A maximum of 4 students at each of 4 participating schools were annually admitted into EPAC before their major clinical year, and some as early as the beginning of their second year of medical school. Thus, the start point for participating in EPAC varied to some degree, but the end point was completion of residency, unless trainees opted out (a complete description has been previously published). 26,27 There was no standard EPAC curriculum; however, all students in EPAC had longitudinal experiences in pediatrics. Two sites had longitudinal integrated clerkships; the others offered longitudinal involvement in a half-day per week continuity clinic throughout medical school or at minimum during the clinical years. It was within these longitudinal contexts that students developed relationships with their preceptors (to be consistent with the literature, we use the term “supervisor”). Supervisors were expected to assess students and guide their learning using an EPA assessment framework.

Selection into the program included an early acceptance to the pediatric residency program affiliated with the respective site. Although all schools used the same EPA assessment framework, the advancement-to-residency process varied between schools. For 2 schools, advancement occurred in a time variable fashion where students could begin their internship early, once supervisors could consistently entrust them with an activity at the supervision level of 3a (allowed to practice EPA under reactive/on-demand supervision, with supervisor immediately available; all findings “double checked”) on all 13 EPAs. 29 In comparison, the other 2 schools increased responsibility and tailored learning experiences once students demonstrated readiness to advance, but there was no formal advancement to residency.

Sample and data collection

Between 2015 and 2018, we collected program evaluation data, which included focus groups twice a year (spring and fall) for a total of 8 focus groups. One student at each of 4 sites was asked by EPAC faculty to attend these meetings, based on student’s availability and interest in participating in the meeting. In total, 27 EPAC students participated in at least 1 of the focus groups; 3 students participated in 2 focus groups (see Supplemental Digital Appendix 1, at http://links.lww.com/ACADMED/B45). Focus groups lasted about 60 minutes and consisted of 3–4 students, with heterogeneity of school and year in training. Twenty-three students were in their third or fourth year of medical school, and 4 had progressed into internship.

Because the primary purpose of these biannual meetings was to share lessons learned across sites, we asked the same primary questions in each focus group (see Supplemental Digital Appendix 2, at http://links.lww.com/ACADMED/B45). We designed these questions to initiate rich discussions about students’ experiences that could inform program improvement, including what attracted students to EPAC, their concerns about EPAC, advice they would give to incoming EPAC students, their perspective on how EPAC supervisors made supervisory decisions (implying entrustment), and their experience of receiving feedback in EPAC. All focus groups were audiotaped, transcribed by a professional transcription company, and deidentified at the time of transcription. In reviewing the transcripts, we determined that the data were sufficiently rich to support an exploration of the research questions.

Reflexivity

Three of us on the research team (C.C., B.F.R., D.F.B.) were involved in the EPAC program evaluation, and to some extent in studying the assessment framework, which H.C.C. and O.tC. helped to create. None of us worked with or supervised EPAC students or had a role in the EPAC program at the site level. Three of us (A.M.C.M., O.tC., H.C.C.) were not involved in EPAC program evaluation. A.M.C.M. brought a medical student perspective, while O.tC. and H.C.C. brought expertise in EPA-based assessment. 27

Data analysis

We took an interpretivist approach to scrutinizing secondary qualitative data, seeking to understand students’ experiences of learning and supervision across an existing dataset. 30 In line with this approach, we used thematic analysis that allowed us to work with 2 types of themes: explicit/manifest and implicit/latent. 31 The former was relevant to our first research question about existing factors that influenced entrustment and ways of understanding those factors. The latter was relevant to our second research question and potentially expanded ways of thinking about trainee agency.

D.F.B. and A.M.C.M. led the analysis. They reviewed clean, uncoded focus group transcripts, curious about factors that students identified as influencing entrustment. They met weekly to create both descriptive codes (i.e., labels for concepts directly linked to raw data) and interpretative codes (i.e., labels for concepts requiring one level of abstraction above the raw data). 32 Interpretative codes were sensitized by the entrustment literature. Over the course of 6 weeks, D.F.B. and A.M.C.M. iteratively revised codes and applied them to the data using ATLAS.ti software, version 8.3 (Scientific Software Development GmbH, Berlin, Germany). They reviewed each other’s coding, came to agreement on any discrepancies, and then shared segments of coded data with B.F.R., C.C., and H.C.C. to verify the code application.

As analysis progressed, D.F.B. and A.M.C.M. derived 3 themes from coded data: students acted to mediate entrustment decisions, students valued longitudinal supervisor–trainee relationships that affect entrustment, and students recognized factors affecting entrustment over which they had little control. D.F.B. and A.M.C.M. checked themes against the dataset for internal consistency and found data supporting themes in data from each focus group. They engaged with the entire team in dialogue around illustrative quotes for each theme. In this way, D.F.B. and A.M.C.M.’s familiarity with the data (as lead investigator and medical student, respectively), H.C.C. and O.tC.’s expertise and views on EPAs, and C.C. and B.F.R.’s extensive experience in medical education contributed to joint construction of knowledge put forth as study findings.

Trustworthiness

In secondary analyses, questions may arise about the trustworthiness of data originally collected for another purpose. Three aspects of our approach to our secondary analysis served as a check on credibility of the findings 25,33: there was a high degree of convergence between what students consistently shared in focus group discussions and our research questions, a single investigator (D.F.B.) led both primary and secondary analyses and knew the study context, and there was close proximity of these analyses in regard to time. We did not, however, explicitly ask students about each of the 5 factors (supervisor, trainee, supervisor–trainee relationship, context, task). We assessed the dependability of our existing data by looking for consistency across the 4 years of data collection. As a check on confirmability, we maintained an audit trail of our coding decisions and engaged in peer debriefing with team members throughout the analysis.

Results

Participating students whom we assessed in an EPA assessment framework described one or more components of the 5 factors that influence entrustment. However, they spoke more frequently and in more depth about components of factors in which they played an active role (trainee and supervisor–trainee) than components of factors over which they had little control (supervisor, clinical task, and clinical context), as summarized in Table 1.

Student actions within the trainee factor mediated entrustment decisions

Students were deliberate in their actions and how they exhibited components of the trainee factor. This was evident in their active engagement in learning and discernment.

Self-directed learning.

Students routinely stressed the importance of active engagement in learning from encounters in clinical practice to become the best pediatrician they could be. At its core, self-directed learning entailed soliciting information that could improve their clinical performance. Students sometimes leveraged the EPA assessment framework as a tool to guide their questions. One student indicated:

I thrive on feedback and wanting to improve and get better. I think EPAC has given me tools to do that in a structured and methodical manner. If I didn’t have EPAs, I would’ve gone into rotations saying, “I don’t really know what they’re looking for.” (Focus group C)

Beyond requesting explicit information about what they needed to know or to do in the clinical workplace, students orchestrated their own learning. For example, one student expressed how she specifically asked to take a more active role in patient care with supervision this way:

I asked to do a lot on my rotation because when I showed up, they’re like “We don’t really expect much out of you. You just started third year. You can just shadow.” I said, “I can do more than that.” I would ask, “Will you come into the room with me and let me manage the visit, but you can watch and jump in if I’m failing.” They were like, “Great, go for it.” (Focus group C)

Action-oriented discernment.

Students talked about their internal discernment process, not just as an awareness of personal limits of knowledge and skills, but as something they shared with their supervisors as part of their learning process. In the words of one student: “That willingness to be honest with yourself and recognize your shortcomings is the only way to learn” (Focus group D).

Some students took the next step and actually helped supervisors recalibrate entrustment decisions so that EPA assessments more closely reflected student’s readiness to progress. In recalibrating supervisor assessments, students focused on optimal learning from a clinical encounter, not optimal evaluations. As one student acknowledged: “Toward the end of clerkship, I feel like preceptors just want to give me a 3b* without really thinking about it, so I’ll dial them back down” (Focus group C).

Others provided explicit cues about EPA assessment to their preceptors. One reminded their supervisors that advancement in this type of framework was not based on grades:

I find myself pushing my preceptors. They will be like, “It was really good today. I don’t want to be nit-picky.” It’s like, “No, I need you to be nit-picky because I want to be up tomorrow. I want to be better.” It has been nice to be able to push them and say, “This isn’t about a grade; it’s not about anything. I just need your advice. What would you have done?” (Focus group D)

In sum, students directed their own learning by asking for information to improve their clinical performance and seeking opportunities to actively engage in learning from patient care. They shared the limitations of their knowledge and skills with their preceptors and pushed preceptors to recognize the same when making entrustment decisions.

Students valued longitudinal supervisor–trainee relationships that affected entrustment decisions

Both self-directed learning and action-oriented discernment routinely occurred in the context of longitudinal supervisor–trainee relationships. From the students’ perspective, having an enduring supervisor–trainee relationship mattered because supervisors’ assumption of a “trustor role” took time. Students routinely distinguished between credible entrustment decisions within longitudinal relationships versus decisions within shorter-term relationships. One student explained the difference this way:

They [longitudinal preceptors] can recognize when you’re having an off day. When someone is new to you, they’re building an opinion of you based on those early encounters. Whereas someone who knows you would recognize typical versus atypical [performance] quicker than others would. (Focus group G)

The length of that relationship had implications for how actively students engaged in learning. One shared:

It is just easier to ask when you have a relationship. If I want help reading an X-ray that I am not sure about, I’ll ask, can we just go over them and see if I am thinking about them, right? (Focus group F)

It also had implications for how much students activated discernment. One admitted:

I can tell my supervisor that I totally forgot to ask about the last menstrual period in this pregnant patient because I know that tomorrow I’m going to see her and we’ll start over. (Focus group D)

In response to self-directed learning and action-oriented discernment, supervisors typically made entrustment decisions that permitted more engagement in clinical care. The student who asked for help reading an X-ray went on to say:

Sometimes we’ll see a patient with similar complaints, and they have seen me manage it before, so then they invite me to take more responsibility. Instead of just reporting to the preceptor with the questions parents have, the preceptor says, “You know the answer to those questions. Let’s go back in and you take the lead answering these questions for the family.” (Focus group F)

In sum, students valued longitudinal relationships with supervisors who knew students’ overall trajectory. These relationships facilitated students’ agentic actions by allowing them to feel comfortable asking for help and to build confidence in accepting more responsibility for patient care, as permitted by their supervisor.

Student recognized components of factors affecting entrustment over which they had little control

In addition to components of the trainee and supervisor–trainee relationship factors described above, students mentioned components of the supervisor, clinical task, and clinical context factors, but less frequently and without the same sense of agentic action. They described supervisor characteristics that influenced entrustment decisions and how much supervisors allowed them to work to the edge of their competence. One characteristic, for example, was level of experience. As one student described, supervisors with more experience provided less direct supervision than their less experienced colleagues:

I have noticed that the shorter time that an attending has been an attending, the more likely they are going to want to supervise everything I do. One preceptor who has only been out of fellowship for a year still double checks all of my findings even though I’ve worked with her for 2 years. (Focus group D)

Students recognized that the nature of the clinical task affected entrustment decision making. They performed tasks that were not complex and/or routine with less direct supervision than tasks that were complex and out of the ordinary. One said, “It varies by the seriousness of the patient’s complaint, or how many times I have seen that” (Focus group F).

Students described aspects of the clinical context that affected supervisors’ decisions to entrust. They were likely to have less direct supervision in the context of nonacute care. Conversely, students were likely to have more direct supervision in the context of intensive care. One student remarked:

In our PICU [pediatric intensive care unit], we see very sick kids from surrounding states. It’s less autonomy from our side of things, like if someone’s coming in and they need to be intubated, there’s a higher threshold for us to be able to do things just because of the nature of care. (Focus group G)

In sum, students mentioned components of the supervisor, clinical task, and clinical context factors that influenced entrustment decisions. But they spoke of these less frequently than components of the trainee and supervisor–trainee relationship factors and from a nonagentic vantage point.

Discussion

In focus groups, 27 medical students in a pilot project that used an EPA assessment framework talked about components of the same 5 factors affecting entrustment decisions as those previously reported. 6–19 However, they talked about components of 2 of these factors (trainee and supervisor–trainee relationship) from a perspective of active engagement in learning (see Table 1). While the nature of the focus group questions (i.e., designed to elicit students’ perceptions of their experience of learning and supervision in EPAC) may account for the predominance of the trainee factor, the consistency of how students made deliberate choices in directing their learning and acting on discernment was salient. Similarly, the impact of longitudinal supervisor–trainee relationships on facilitating students’ agentic actions was notable.

Our study expands understanding of components of the trainee factor. Self-directed learning and action-oriented discernment can be viewed as agentic actions. 20 In our study, agency was evident when students orchestrated their learning and used the EPA assessment framework to push for feedback. It was also evident when students made their limitations known to supervisors who could in turn calibrate assessments and entrustment decisions.

Concurrent with our study, ten Cate and Chen have expanded the model of trainee factor influences on entrustment decisions to include the following components: Agency (proactive toward work, team, safety, personal development), reliability (conscientious, predictable, accountable, responsible), integrity (truthful, benevolent, patient centered), capability (specific knowledge, skills, experience, situational awareness), and humility (recognizes limits, asks for help, receptive to feedback). 34,35 Our findings, derived from medical students in an EPA assessment framework, lend support to ten Cate and Chen’s model and aligns with recent research proposing that residents can promote entrustment though self-management, relationships, advocacy, and patient centeredness. 36

The agency our students described suggests that they could navigate, with their EPAC supervisors, situations in which they perceived too little or too much trust (within the context of entrustment decisions), rather than passively attribute such a misalignment of trust to factors over which they had little or no influence. 19 Our data showcase the capacity of trainees to become “self-directed assessment seekers,” 37 at least in the context of an EPA assessment framework with longitudinal supervisor relationships. Developing this habit of seeking evidence from assessments to inform and corroborate self-reflections has important implications for continuous professional development.

Our findings also underscore the value of longitudinal supervisor–trainee relationships for entrustment decision making. Students in our study linked their ability to structure their own learning and renegotiate entrustment decisions to longitudinal supervisor–trainee relationships. From Billet’s point of view, this interaction between trainee and supervisor–trainee relationships is not unexpected. 23 The richness of learning from workplace activities depends on both workplace affordances and trainee agency, and workplace affordances can influence trainee agency. Thus, our findings support educational models that marry longitudinal relationships and EPA assessment frameworks. 38,39 Although programs cannot always provide longitudinal clinical experiences, they can promote other structures to promote longitudinal coaching or continuity across clinical supervisors.

Finally, our findings suggest that trust and entrustment function differently in an EPA versus traditional assessment framework. In the former, entrustment justifies engagement in patient care, and advancement is explicitly at stake for the learner, the supervisor, and the patient. In the latter, grades are at stake, and trust can be inhibited when students experience discrepancies between verbal/informal and written/formal feedback. 40 We caution against making assumptions about entrustment in an EPA assessment framework based solely on studies of trust in other assessment frameworks. Conversely, we acknowledge that our findings may apply to our student population only or to other frameworks in addition to EPA assessment frameworks. Further research is needed to explore how assessment, more broadly positioned, promotes agency in ways similar to the ones we report.

Limitations

Because our study was a secondary analysis, data collection and analysis were not iterative, as is often the case in thematic analysis. The research questions guiding our secondary analysis were not identical to the program evaluation questions used in the focus groups. Furthermore, students who participated in the focus groups represent a select group of medical students who all had longitudinal relationships with supervisors and had a guaranteed position in the pediatric residency affiliated with the medical school. Although EPAC provided a unique context in which to explore students’ perceptions of entrustment, our findings may not fully transfer to entrustment decisions in other settings. Nonetheless, the principle of agentic action as it relates to self-directed assessment seeking may be useful beyond the limitations of this study. We did not investigate site- or student-specific differences because data were deidentified. In spite of these limitations, our findings provide important new information about entrustment in an EPA assessment framework from a student perspective.

Conclusions

Students’ perspectives on entrustment in an EPA assessment framework included components from previously reported factors but emphasized components of the trainee and supervisor–trainee factors that shaped decisions to entrust trainees to perform professional activities. Students employed agentic actions to influence entrustment decisions, and their actions were facilitated by their longitudinal relationships with their supervisors. Their perspective complements the literature by highlighting students’ agentic actions and the promotion of agentic action through practices that incorporate longitudinal supervisor–trainee relationships.

Acknowledgments:

The authors acknowledge their collaborators in the Education in Pediatrics Across the Continuum (EPAC) Study Group: Patricia M. Hobday, MD (University of Minnesota Medical School), Michele Long, MD (University of California, San Francisco, School of Medicine), Lisa Howley, PhD (Association of American Medical Colleges), James F. Bale Jr, MD (University of Utah School of Medicine), John S. Andrews, MD (American Medical Association), J. Lindsey Lane, BMBCh (University of Colorado School of Medicine), and Daniel C. West, MD (The Children’s Hospital of Philadelphia and Perelman School of Medicine, University of Pennsylvania). Additionally, the authors acknowledge the EPAC students who participated in the focus groups.

*3b indicates “Allowed to practice EPA under reactive/on-demand supervision with supervisor immediately available and key findings double-checked.” 29

References

1. ten Cate O, Hart D, Ankel F, et al.; International Competency-Based Medical Education Collaborators. Entrustment decision making in clinical training. Acad Med. 2016; 91:191–198
2. ten Cate O. Entrustment as assessment: Recognizing the ability, the right, and the duty to act. J Grad Med Educ. 2016; 8:261–262
3. Cianciolo AT, Kegg JA. Behavioral specification of the entrustment process. J Grad Med Educ. 2013; 5:10–12
4. Holzhausen Y, Maaz A, Cianciolo AT, ten Cate O, Peters H. Applying occupational and organizational psychology theory to entrustment decision-making about trainees in health care: A conceptual model. Perspect Med Educ. 2017; 6:119–126
5. Mayer RC, Davis JH, Schoorman FD. An integrative model of organizational trust. Acad Manage Rev. 1995; 20:709–734
6. Dijksterhuis MG, Voorhuis M, Teunissen PW, et al. Assessment of competence and progressive independence in postgraduate clinical training. Med Educ. 2009; 43:1156–1165
7. Hauer KE, ten Cate O, Boscardin C, Irby DM, Iobst W, O’Sullivan PS. Understanding trust as an essential element of trainee supervision and learning in the workplace. Adv Health Sci Educ Theory Pract. 2014; 19:435–456
8. Hauer KE, Oza SK, Kogan JR, et al. How clinical supervisors develop trust in their trainees: A qualitative study. Med Educ. 2015; 49:783–795
9. Sterkenburg A, Barach P, Kalkman C, Gielen M, ten Cate O. When do supervising physicians decide to entrust residents with unsupervised tasks? Acad Med. 2010; 85:1408–1417
10. Choo KJ, Arora VM, Barach P, Johnson JK, Farnan JM. How do supervising physicians decide to entrust residents with unsupervised tasks? A qualitative analysis. J Hosp Med. 2014; 9:169–175
11. Duijn CCMA, Welink LS, Bok HGJ, ten Cate OTJ. When to trust our learners? Clinical teachers’ perceptions of decision variables in the entrustment process. Perspect Med Educ. 2018; 7:192–199
12. Landreville JM, Cheung WJ, Hamelin A, Frank JR. Entrustment checkpoint: Clinical supervisors’ perceptions of the emergency department oral case presentation. Teach Learn Med. 2019; 31:250–257
13. Sheu L, O’Sullivan PS, Aagaard EM, et al. How residents develop trust in interns: A multi-institutional mixed-methods study. Acad Med. 2016; 91:1406–1415
14. Sheu L, Kogan JR, Hauer KE. How supervisor experience influences trust, supervision, and trainee learning: A qualitative study. Acad Med. 2017; 92:1320–1327
15. Tiyyagura G, Balmer D, Chaudoin L, et al. The greater good: How supervising physicians make entrustment decisions in the pediatric emergency department. Acad Pediatr. 2014; 14:597–602
16. Wijnen-Meijer M, van der Schaaf M, Nillesen K, Harendza S, ten Cate O. Essential facets of competence that enable trust in medical graduates: A ranking study among physician educators in two countries. Perspect Med Educ. 2013; 2:290–297
17. Wijnen-Meijer M, van der Schaaf M, Nillesen K, Harendza S, ten Cate O. Essential facets of competence that enable trust in graduates: A delphi study among physician educators in the Netherlands. J Grad Med Educ. 2013; 5:46–53
18. Bonnie LHA, Visser MRM, Kramer AWM, van Dijk N. Insight in the development of the mutual trust relationship between trainers and trainees in a workplace-based postgraduate medical training programme: A focus group study among trainers and trainees of the Dutch general practice training programme. BMJ Open. 2020; 10:e036593
19. Karp NC, Hauer KE, Sheu L. Trusted to learn: A qualitative study of clerkship students’ perspectives on trust in the clinical learning environment. J Gen Intern Med. 2019; 34:662–668
20. ten Cate O. When I say … entrustability. Med Educ. 2020; 54:103–104
21. ten Cate O, Schwartz A, Chen HC. Assessing trainees and making entrustment decisions: On the nature and use of entrustment-supervision scales. Acad Med. 2020; 95:1662–1669
22. Billett S. Learning through health care work: Premises, contributions and practices. Med Educ. 2016; 50:124–131
23. Billett S. Work-based learning as a pathway to competence-based education. Bahl A, Dietzen A, eds. In: Securing Occupational Capacities Through Workplace Experiences: Premises, Conceptions and Practices. Bonn, Germany: UNEVOC Network Contribution, 2019; 25–43
24. Sheehan D, Wilkinson TJ, Billett S. Interns’ participation and learning in clinical environments in a New Zealand hospital. Acad Med. 2005; 80:302–308
25. Heaton J. Reworking Qualitative Data. London, UK: Sage Publication, 2004
26. Andrews JS, Bale JF Jr, Soep JB, et al.; EPAC Study Group. Education in Pediatrics Across the Continuum (EPAC): First steps toward realizing the dream of competency-based education. Acad Med. 2018; 93:414–420
27. Murray KE, Lane JL, Carraccio C, et al.; Education in Pediatrics Across the Continuum (EPAC) Study Group. Crossing the gap: Using competency-based assessment to determine whether learners are ready for the undergraduate-to-graduate transition. Acad Med. 2019; 94:338–345
28. Association of American Medical Colleges. Toolkits for the 13 core entrustable professional activities for entering residency. https://www.aamc.org/what-we-do/mission-areas/medical-education/cbme/core-epas/publications. Accessed November 5, 2020
29. Chen HC, van den Broek WE, ten Cate O. The case for use of entrustable professional activities in undergraduate medical education. Acad Med. 2015; 90:431–436
30. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006; 3:77–101
31. Kiger ME, Varpio L. Thematic analysis of qualitative data: AMEE guide no. 131. Med Teach. 2020; 42:846–854
32. Richards L, Morse JM. README FIRST for a User’s Guide to Qualitative Methods. 3rd ed. Los Angeles, CA: Sage Publishing, 2012
33. Ruggiano N, Perry TE. Conducting secondary analysis of qualitative data: Should we, can we, and how? Qual Soc Work. 2019; 18:81–97
34. ten Cate O. Competency-based education and entrustable professional activities (EPAs) for the health professions. Presented at: Symposium 25 years Unit R&D of Medical Education; June 27, 2019, University of Geneva, Geneva, Switzerland
35. ten Cate O, Chen HC. The ingredients of A-RICH entrustment decision. [Under review, available from the authors on request]
36. Pingree EW, Huth K, Harper BD, et al. Encouraging entrustment: A qualitative study of resident behaviors that promote entrustment. Acad Med. 2020; 95:1718–1725
37. Eva KW, Regehr G. “I’ll never play professional football” and other fallacies of self-assessment. J Contin Educ Health Prof. 2008; 28:14–19
38. Hirsh DA, Holmboe ES, ten Cate O. Time to trust: Longitudinal integrated clerkships and entrustable professional activities. Acad Med. 2014; 89:201–204
39. Englander R, ten Cate O. LICs and entrustable professional activities: A perfect match. Poncelet AN, Hirsh DA, eds. In: Longitudinal Integrated Clerkships Principles, Outcomes, Practical Tools, and Future Directions. North Syracuse, NY: Alliance for Clinical Educators, 2016; 261–269
40. Dolan BM, Arnold J, Green MM. Establishing trust when assessing learners: Barriers and opportunities. Acad Med. 2019; 94:1851–1853

References cited only in Table 1

41. Santen SA, Wolff MS, Saxon K, Juneja N, Bassin B. Factors affecting entrustment and autonomy in emergency medicine: “How much rope do I give them?” West J Emerg Med. 2019; 20:58–63
42. Duijn CCMA, Welink LS, Mandoki M, ten Cate OTJ, Kremer WDJ, Bok HGJ. Am I ready for it? Students’ perceptions of meaningful feedback on entrustable professional activities. Perspect Med Educ. 2017; 6:256–264
    43. Schumacher DJ, Martini A, Bartlett KW, et al.; Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network Clinical Competency Committee Study Group. Key factors in clinical competency committee members’ decisions regarding residents’ readiness to serve as supervisors: A national study. Acad Med. 2019; 94:251–258

    Supplemental Digital Content

    Copyright © 2020 by the Association of American Medical Colleges