The terms trust and entrustment are increasingly being used in the medical education literature.1–7 (We provide definitions of these terms later in this article.) Trust between patients and clinicians is foundational to good care, and trust between trainees and supervising medical professionals is key to effective clinical education. Entrustment decision making—that is, deciding how far to trust trainees to carry out patient care on their own—reflects an attempt to align assessment in the workplace with everyday clinical practice.8–10 It has not been analyzed extensively, perhaps because trust is so natural and tacit during everyday clinical work that its significance may be overlooked. Trusting trainees with clinical work while they progress in their training is a natural course of action. Discrete moments of entrustment of new tasks to trainees mark the increasing development, privileging, and certification of emerging medical professionals. The introduction of competency-based medical education11,12 and, more recently, of entrustable professional activities (EPAs),13,14 have catalyzed a desire to understand how supervisors come to entrustment decisions and what features make trainees trustworthy.
In this article we analyze mechanisms that appear to affect the process of entrustment decision making for trainees in undergraduate and postgraduate medical education. Our analysis is based on our extensive discussions and content analysis during a two-day summit of the International Competency-Based Medical Education Collaborators (September 29 and 30, 2013), which we supplemented with reference citations gained from our review of the literature. As authors, we combine substantial experience in clinical education (emergency medicine, pediatrics, internal medicine, and family medicine) with experience in educational research and development, assessment in the workplace, and knowledge from a regulatory perspective.
Trust, Control, and Autonomy
The trust that patients and society bestow on the medical profession, in return for high-quality service and a professional obligation to control its quality, has been named a social contract.15 In the past decades, a decrease in public confidence in health care systems has been observed, challenging this contract.16,17 Critical incidents in health care have triggered this process. A 1984 college woman’s preventable death after hospitalization in the United States,18 the Institute of Medicine’s report To Err Is Human,19 and the Mid-Staffordshire Hospital catastrophe in the United Kingdom20 served as wake-up calls. Significant numbers of patients have suffered iatrogenic harm as a result of deficiencies in health care systems and patient care practices. While the ability to measure quality, safety, and harm in health care has increased substantially since the 1970s, it has also made both the profession and the public aware of quality deficiencies and of the need for systematic quality improvement.21,22 The decrease of public trust in health care has thus sparked interest in greater public control over the health care industry23 as well as efforts to improve internal control within the medical profession.24
Control strives to exclude risk, while trust implies that risk must be accepted to a certain extent. Internal control is intended to create a safer health care world, but measures to exert control often convey a message of distrust7,17 that can ultimately jeopardize patient safety. In an attempt to mitigate possible risks, supervision of residents has increased in the past decades to a point where learners hardly experience the full responsibility of health care before they commence unsupervised practice.24 Fear of litigation for having trainees take responsibility has contributed to this trend. Mattar et al25 recently found that surgeons at the end of training were, in many respects, not ready for fellowships, let alone unsupervised practice, and an unprecedented number sought to extend their training through fellowships. New training models are needed to reinstate a system of graded responsibility.24 This can be achieved only with adequate supervision, to avoid the dangerous circumstances created by the unsupervised practice of learners that was previously common.
Based on a system of graded responsibility,25 these new training models must stimulate trainees to push for higher levels of mastery. An imminent need to acquire new knowledge or skills necessary to do a requested job often serves as that stimulus. An adequate distance between what a learner has mastered and an incremental new level of proficiency has been called a zone of proximal development.26 The constructive friction that this gap causes is unavoidable and necessary for learning.27 Effective education requires supervisors to create this friction while taking the risk that a learner will act imperfectly when performing new tasks the first time. Creating a manageable level of risk is inherent in decisions about entrustment. Balancing effective supervision with the risk inherent in creating constructive friction is critical to preserving patient safety.28
The goal throughout the medical continuum is to educate learners to be ready to provide safe, unsupervised, professional care. Reaching that goal establishes a threshold of trust, meaning that the physician is now capable of self-directed, continued learning. This involves continuous reflection through self-assessment and adaptation and the incorporation of feedback sought and received from colleagues, patients, and others.
Registration and certification formally mark moments of trust, responsibility, and autonomy. The word autonomy is emotionally charged. Psychologists tend to stress the freedom to make one’s own choices as a key condition for intrinsic motivation,29 sociologists view autonomy as a core feature that distinguishes professions from other occupations,30 and physicians have traditionally claimed autonomy as a necessary condition to maximize benefits for patients.31 Current views on quality, safety, accountability, and transparency restrict autonomous practice to the freedom to make professional choices within the boundaries of shared professional standards with the interdependence of collaboration. As health care ceases to be practiced by soloists, the term relational autonomy was introduced to stress the new characteristic of necessary interdependence.32 Deciding to entrust a trainee with critical responsibilities without supervision aims at this type of autonomy. Unsupervised practice by medical trainees does not imply autonomy without oversight—since there will be continued departmental and institutional oversight—but the fading of required educational supervision, such as by teachers and mentors.
Defining Trust and Entrustment in Health Care and Training
Trust, according to the Oxford English Dictionary, is “confidence in or reliance on some quality or attribute of a person or thing.” To entrust is “to confide the care or disposal of [a thing or person] or the execution of [a task] to or with a person.” Entrustment is “the action of entrusting or the fact of being entrusted.”33 Translated to clinical training, the object of care may be the patient in a general sense, and the task is a professional activity that usually involves this patient. Pollock23 explains that
trust involves the confident expectation that a person can be relied on to honour implied or established commitments to an individual and to protect [the individual’s] interest. It renders the individual vulnerable to the extent he cannot oversee or control the actions of the other, on whose expertise or integrity he may depend.
A definition of trust provided by Mayer et al34 is
the willingness of a party to be vulnerable to the actions of another party based on the expectation that the other will perform a particular action important to the trustor, irrespective of the ability to monitor or control that other party.
According to Mayer et al, trust assumes ability, benevolence, and integrity.
In health care, trust is often discussed in relation to the patient’s trust in doctors, in the medical profession, in a medical institution, or in the health care system.15,16,23 In a medical training setting, trust can be understood by interpreting the dictionary definition to be “the reliance of a supervisor or medical team on a trainee to execute a given professional task correctly and on his or her willingness to ask for help when needed.” Trust requires interdependence between truster and trustee, and creates supervisor vulnerability, as mistakes made by a trainee may affect the supervisor personally.1 Trust thus entails an acceptance of being vulnerable to the actions of a trustee—an acceptance based on the expectation that the trustee will probably perform in a predictable way.34–36 The first time a clinical supervisor asks a trainee to care for a patient or to perform a procedure without his or her direct supervision implies a willingness to take some risk of adverse events.37
Presumptive, initial, and grounded trust
Cianciolo and Kegg3 have proposed a model that moves from first observations, sometimes triggered by errors, to a “readiness” judgment based on multiple informal assessments, followed by a more formal assessment that includes a risk-mitigating strategy incorporating the evaluation of situational conditions. Combining these distinctions leads us to propose three modes of trust in clinical supervisor–trainee relationships: presumptive trust, initial trust, and grounded trust.
Presumptive trust15 is based solely on credentials, without prior interaction with the trainee. It is trust in diplomas, institutions, or referents recommending the person and is often present as a default unless it is breached. Restoring a breach of presumptive trust usually requires time.38
Initial trust is based on first impressions and is sometimes called swift trust or thin trust.39 The accuracy of first impressions is likely affected by trainee variables (desire to make a favorable impression) and supervising clinician variables (mood, gender, intelligence, experience with similar trainees).40 Trust propensity, defined as a dispositional willingness to rely on someone, can vary.35 Initial trust is vulnerable to halo effects (first impressions color judgment of qualities observed later) and self-fulfilling prophecy effects (e.g., early bad impressions may create lasting tension or a lack of confidence in a relationship). Initial trust bears some resemblance to pattern recognition in medical decision making.40
Grounded trust is based on essential and prolonged experience with the trainee. For example, society places trust in individuals to drive a car unsupervised, provided this trust is grounded with specified conditions (minimum age, having passed a theoretical exam, having passed a practical exam, and implicitly having executed a number of hours of supervised driving). Grounded trust in medical trainees should be preceded by sufficient observation and pertinent data to qualify the trainee to act without supervision.
Ad hoc and summative entrustment decisions
The decision to trust a trainee with a critical task can be prompted by a necessity stemming from the immediate context (“Task X must be done; someone must do it”) or may be a chosen option for educational purposes; trust may pertain to small actions as well as to a large responsibility, such as licensing for a specialty. It is therefore useful to categorize entrustment decisions.
We propose two categories. Entrustment decisions may be ad hoc,1 as happens continuously in health care situations, or summative,2 when the entrustment decision is based on grounded trust and has the nature of a generalized certification and privilege to act without supervision for a specified unit of professional practice.41Ad hoc entrustment decisions by clinical supervisors about trainees are usually based on a mix of estimated trustworthiness of the trainee, estimated risk of the situation, urgency of the job to be done, and suitability of this task at this moment for this learner. They do not necessarily constitute a precedent for similar decisions in the future. In contrast, summative entrustment decisions, grounded in sufficient evaluation and made by educational program directors or clinical competency committees, should lead to certification and privileging of the trainee to act in the future with a specified level of supervision.
An analogy with medical decision making is useful. Clinicians use a rapid and intuitive thinking process (called System 1 thinking) to act efficiently in routine cases, as they quickly recognize a pattern and decide on a diagnosis and a course of action. In complex cases they need to rely on slow analytic reasoning strategies (System 2 thinking) that take more time.42 System 1 thinking would apply in ad hoc entrustment decisions, when there is no time to collect much information about a learner, while System 2 thinking would be more common in grounded summative decisions. When one “feels” that trusting a trainee is not justified when expected, analytic reasoning may be necessary to detect why this is the case. The learner’s level of achievement of milestones,43 or preentrustable and entrustable descriptions of the learner’s behavior,44 may serve as references to support the analytic thinking in case of uncertainty.
The progression toward being trusted
The outcome of an entrustment decision is the determination of a level of required supervision, ranging theoretically from the permission to touch or talk to a patient in the presence of a supervisor to being given full autonomy to carry out a complex surgical procedure unsupervised. For practical purposes, a framework of five levels of supervision has been proposed. Table 1 summarizes the specifications of these levels from various previous publications.14,41,45,46 Trainees may progress through these levels as early as it is deemed safe. Waiting to trust trainees to act unsupervised until after they are legally qualified deprives them of the valuable opportunity to practice unsupervised while still in a supportive training environment. Allowing trainees to perform activities with indirect supervision prepares them for true unsupervised practice after graduation. Balanced, distant supervision that ensures safe practice while at the same time stimulating the authentic experience of responsibility in learners is key.47 Direct supervision for the full duration of residency, as sometimes happens, may not prepare trainees well for independent practice, while a careless lack of supervision, as sometimes happens during night shifts, endangers patients’ and residents’ safety.
Entrustment Decision Making as Workplace Assessment
A dominant topic in the medical education literature of the past decade is the need to rethink the assessment of medical trainees in the workplace.48–52 Workplace-based assessment is fraught with difficulty, as it relies on observations by medical professionals. It cannot be standardized, as it depends on varying contexts, patients, and supervisors. Known difficulties include rater leniency bias (generosity error), halo effects, restriction of range, poor discrimination between trainees, lack of documentation of deficits, low intra- and interrater and cross-occasion consistency,48,53 and idiosyncratic limitations of raters when categorizing trainees.54
In the past decade, the quest for more valid assessment in the workplace has focused on increasing the reliability and generalizability of scores. For example, breaking down the traditionally long clinical evaluation exam for internal medicine residents into multiple mini-clinical evaluation exercises led to better sampling and generalizability,55 but the number of observations to establish a reasonable reliability has been estimated to be as large as 50.56 Designing assessment instruments to validly assess competencies has had limited success.57 Clinicians who assess competencies require a mind-set somewhat remote from their daily mode of thinking. Improving the alignment of rating tasks with regular clinical tasks is likely to increase their quality,8,58,59 such as relabeling abstract rating scales (e.g., poor to excellent) with the question “How much supervision is justified?”9
The consequential validity of an assessment stems from its purpose.60 Entrustment decisions have a clear purpose, which is to confirm not only the ability, but also the right and the duty, for a trainee to act.61–63 The focus is not on rating current observable performance but, instead, on the transfer of the trainee’s capacity to new, unfamiliar situations, without direct supervision. A recent Dutch–German project, the Utrecht–Hamburg Trainee Responsibility for Unfamiliar Situations Test, is an early attempt to assess just that capacity.64
Factors Involved in Entrustment Decision Making
Several authors4,52,65–67 have investigated how entrustment decisions for medical trainees come about. Sterkenburg et al4 found vast differences among anesthetists in their estimation of when residents would be ready to carry out six critical procedures unsupervised. Cianciolo and Kegg3 identified different entrustment styles among attendings. The literature suggests that five categories, each of multiple factors, together determine whether an ad hoc decision is taken to entrust a trainee with a new and critical task in the workplace. These factors are specific characteristics of (1) the trainee, (2) the supervisor making that decision, (3) the context or circumstances, and (4) the task or activity. In addition, (5) the relationship between trainee and supervisor has been suggested as a category, with its own factors,2 as this appears conditional for the development of trust. Hauer and colleagues’2 recent review was supplemented with the works of Sterkenburg, Choo, Dijksterhuis, Wijnen-Meijer, Kennedy, O’Neill, and their colleagues4,5,65,67–70 to establish the most salient factors in entrustment decision making within these categories (see Table 2).
Grounding Summative Entrustment Decisions
In making a summative entrustment decision, the aim is to focus on trainee factors only. In contrast to ad hoc decisions, summative decisions should depend little on supervisor features and context characteristics. EPAs, which are units of professional practice that trainees are permitted to execute unsupervised once they have demonstrated sufficient competence,13,41 can be the focus of summative entrustment decisions. These summative entrustment decisions should be generalizable and may pertain to an array of small tasks in various contexts, combined into one reasonably broad unit of practice, such as “caring for the well newborn” in pediatrics. For undergraduate medical education, the Association of American Medical Colleges has proposed 13 core EPAs for entering residency,44 which together should reflect the level of clinical competence required for full licensing at the MD level.
Factors affecting these summative entrustment decisions will vary, depending on stage of training, specialty, and the EPA, but there will always be multiple information sources involved to support the validity of such decisions. Table 3 considers the requisite types and sources of information that are stated in the literature. This list is not comprehensive and may not always be applicable to all EPAs, but it helps one understand the reasoning process in summative entrustment decision making.
The most foundational factors—competence, conscientiousness, truthfulness, and discernment of one’s own limitations, as proposed by Kennedy et al69—are fundamental to any entrustment decision. They align, for instance, with factors expressed in philosophy, dating as far back as Aristotle’s Rhetoric.71 Aristotle suggested that a speaker’s ethos should be based on the listener’s perception of three things: intelligence, character (reliability, honesty), and goodwill (favorable intentions). These are parallel to the ethics of ability, integrity, and benevolence proposed by Mayer et al34 to support trust and with the current philosopher O’Neill’s17,70 proposed trust conditions of competence, reliability, and honesty.
The evaluation of these qualities in a trainee requires longitudinal observation, preferably across different contexts. In some jurisdictions, transferring student information across rotations is precluded, to avoid prejudice against learners starting new rotations. Although this argument has some validity, it is also in the interest of learners, teachers, patients, and the program to share information across settings to carefully build a solid foundation for summative entrustment decisions, and to correct learners in an early phase when required.72
In this article we have discussed and elaborated definitions of trust and entrustment in medical education. We define the goal of medical education as readiness for unsupervised practice and recommend that the outcome of assessment in the workplace be measured in terms of the level of supervision to be provided.
The movement toward competency-based education ultimately serves to increase the trust of society in the competence of medical professionals.73,74 A model that aligns trust in the trainee with the assessment of the trainee’s competence may help to proceed in this direction.
The entrustment of clinical tasks to medical trainees is a seemingly easy process that occurs multiple times every day in almost every clinical setting where medical students, residents, or fellows are trained. Yet, when analyzed, many factors appear to determine how, when, and whether learners are granted responsibilities under indirect or distant supervision. Deliberate entrustment decisions take these factors into account and aim to reconcile the educational need to push learners to stretch their scope of performance with the need for safe, high-quality patient care. Making entrustment decisions explicit can help to change the status quo. First, this requires more effective assessments to make the optimum entrustment. Second, making safe, effective, patient-centered care the frame of reference for the entrustment brings the focus back more firmly to the patient.59 Third, explicit entrustment requires the hard conversations around curriculum design.
Our goal in this article has been to lay a theoretical foundation for a new approach to workplace training and assessment, rather than to provide practical guidance of how to implement the major changes in education that follow from this approach. For that, readers can refer to a different publication.75 We believe that once these concepts are well understood—and faculty development will be necessary to create that understanding—entrustment decision making about medical trainees in the workplace will be a more natural, logical, and valid mind-set in assessment than many of the more traditional approaches currently in place. Entrustment decision making forces clinicians to think more deliberately about opportunities bestowed and risks incurred in the near future by granting responsibility to trainees, rather than simply reporting observed performance.
Acknowledgments: The authors have gratefully used notes from a workshop on entrustment decision making held during the annual meeting of the Association for Medical Education in Europe in 2014 in Milan, Italy, particularly for completing Table 3.
1. ten Cate O. Trust, competence, and the supervisor’s role in postgraduate training. BMJ. 2006;333:748–751
2. Hauer KE, Ten Cate O, Boscardin C, Irby DM, Iobst W, O’Sullivan PS. Understanding trust as an essential element of trainee supervision and learning in the workplace. Adv Heal Sci Educ. 2014;19:435–456
3. Cianciolo AT, Kegg JA. Behavioral specification of the entrustment process. J Grad Med Educ. 2013;5:10–12
4. Sterkenburg A, Barach P, Kalkman C, Gielen M, ten Cate O. When do supervising physicians decide to entrust residents with unsupervised tasks? Acad Med. 2010;85:1408–1417
5. Choo KJ, Arora VM, Barach P, Johnson JK, Farnan JM. How do supervising physicians decide to entrust residents with unsupervised tasks? A qualitative analysis. J Hosp Med. 2014;9:169–175
6. Wallenburg I, van Exel J, Stolk E, Scheele F, de Bont A, Meurs P. Between trust and accountability: Different perspectives on the modernization of postgraduate medical training in the Netherlands. Acad Med. 2010;85:1082–1090
7. Hirsh DA, Holmboe ES, ten Cate O. Time to trust: Longitudinal integrated clerkships and entrustable professional activities. Acad Med. 2014;89:201–204
8. Crossley J, Johnson G, Booth J, Wade W. Good questions, good answers: Construct alignment improves the performance of workplace-based assessment scales. Med Educ. 2011;45:560–569
9. Weller JM, Misur M, Nicolson S, et al. Can I leave the theatre? A key to more reliable workplace-based assessment. Br J Anaesth. 2014;112:1083–1091
10. George BC, Teitelbaum EN, Meyerson SL, et al. Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance. J Surg Educ. 2014;71:e90–e96
11. Swing SR. The ACGME outcome project: Retrospective and prospective. Med Teach. 2007;29:648–654
12. Frank JR The CanMEDS 2005 Physician Competency Framework. 2005 Ottawa, Ontario, Canada Royal College of Physicians and Surgeons of Canada
13. ten Cate O. Entrustability of professional activities and competency-based training. Med Educ. 2005;39:1176–1177
14. ten Cate O, Scheele F. Competency-based postgraduate training: Can we bridge the gap between theory and clinical practice? Acad Med. 2007;82:542–547
15. Cruess RL, Cruess SRCockerham W, Dingwall R, Quah S. Professional trust. The Wiley Blackwell Encyclopedia of Health, Illness, Behavior, and Society. 2014 Hoboken, NJ John Wiley & Sons, Ltd
16. Mechanic D. The functions and limitations of trust in the provision of medical care. J Health Polit Policy Law. 1998;23:661–686
17. O’Neill O A Question of Trust. 2002 Cambridge, UK Cambridge University Press
18. Halpern SD, Detsky AS. Graded autonomy in medical education—managing things that go bump in the night. N Engl J Med. 2014;370:1086–1089
19. Kohn LT, Corrigan JM, Donaldson MSCommittee on Quality of Health Care in America. To Err Is Human: Building a Safer Health System. 2000 Washington, DC National Academy Press
21. Baker GR, Norton PG, Flintoft V, et al. The Canadian adverse events study: The incidence of adverse events among hospital patients in Canada. CMAJ. 2004;170:1678–1686
22. Marciniak TA, Ellerbeck EF, Radford MJ, et al. Improving the quality of care for Medicare patients with acute myocardial infarction: Results from the Cooperative cardiovascular project. JAMA. 1998;279:1351–1357
23. Pollock KCockerham W, Dingwall R, Quah S. Patient trust. The Wiley Blackwell Encyclopedia of Health, Illness, Behavior, and Society. 2014 Hoboken, NJ John Wiley & Sons, Ltd.
24. Dauphinee WD. Self regulation must be made to work. BMJ. 2005;330:1385–1387
25. Mattar SG, Alseidi AA, Jones DB, et al. General surgery residency inadequately prepares trainees for fellowship: Results of a survey of fellowship program directors. Ann Surg. 2013;258:440–449
26. Vygotsky LS Mind in Society. The Development of Higher Psychological Processes. 1978 Cambridge, Mass Harvard University Press
27. Vermunt JD, Verloop N. Congruence and friction between learning and teaching. Learn Instr. 1999;9:257–280
28. Brunett P. Autonomy versus control: Finding the sweet spot. Acad Emerg Med. 2013;20:952–953
29. Ryan RM, Deci EL. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol. 2000;55:68–78
30. Stevens FCockerham W, Dingwall R, Quah S. Health professions, sociology of. The Wiley Blackwell Encyclopedia of Health, Illness, Behavior, and Society. 2014 Hoboken, NJ John Wiley & Sons, Ltd.
31. Dingwall RCockerham W, Dingwall R, Quah S. Professional ethics and accountability. The Wiley Blackwell Encyclopedia of Health, Illness, Behavior, and Society. 2014 Hoboken, NJ John Wiley & Sons, Ltd.
32. Holmboe E, Ginsburg S, Bernabeo E. The rotational approach to medical education: Time to confront our assumptions? Med Educ. 2011;45:69–80
33. Oxford English Dictionary. www.oed.com
. Accessed October 19, 2015
34. Mayer RC, Davis JH, Schoorman FD. An integrative model of organizational trust. Acad Manag Rev. 1995;20:709–734
35. Colquitt JA, Scott BA, LePine JA. Trust, trustworthiness, and trust propensity: A meta-analytic test of their unique relationships with risk taking and job performance. J Appl Psychol. 2007;92:909–927
36. Schoorman FD, Mayer RC, Davis JH, Davis H. A integrative model of organizational trust: Past, present, and future. Acad Manag Rev. 2007;32:344–354
37. Ten Cate O. Trusting graduates to enter residency: What does it take? J Grad Med Educ. 2014;6:7–10
38. Lount RB, Zhong C-B, Sivanathan N, Murnighan JK. Getting off on the wrong foot: The timing of a breach and the restoration of trust. Pers Soc Psychol Bull. 2008;34:1601–1612
39. Hauer KE Evaluating Clinical Trainees in the Workplace: On Supervision, Trust and the Role of Competency Committees. 2015 Utrecht, the Netherlands Utrecht University
40. Wood TJ. Exploring the role of first impressions in rater-based assessments. Adv Health Sci Educ Theory Pract. 2014;19:409–427
41. Ten Cate O. Nuts and bolts of entrustable professional activities. J Grad Med Educ. 2013;5:157–158
42. Croskerry P, Petrie DA, Reilly JB, Tait G. Deciding about fast and slow decisions. Acad Med. 2014;89:197–200
43. Swing SR, Beeson MS, Carraccio C. Educational milestone development in the first 7 specialties to enter the next accreditation system. J Grad Med Educ. 2013;5:98–106
45. Mulder H, Ten Cate O, Daalder R, Berkvens J. Building a competency-based workplace curriculum around entrustable professional activities: The case of physician assistant training. Med Teach. 2010;32:e453–e459
46. Chen HC, van den Broek WE, ten Cate O. The case for use of entrustable professional activities in undergraduate medical education. Acad Med. 2015;90:431–436
47. Babbott S. Commentary: Watching closely at a distance: Key tensions in supervising resident physicians. Acad Med. 2010;85:1399–1400
48. Govaerts MJ, van der Vleuten CP, Schuwirth LW, Muijtjens AM. Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Adv Health Sci Educ Theory Pract. 2007;12:239–260
49. van der Vleuten CP, Schuwirth LW, Scheele F, Driessen EW, Hodges B. The assessment of professional competence: Building blocks for theory development. Best Pract Res Clin Obstet Gynaecol. 2010;24:703–719
50. Schuwirth LW, Van der Vleuten CP. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33:478–485
51. van der Vleuten CP, Schuwirth LW. Assessing professional competence: From methods to programmes. Med Educ. 2005;39:309–317
52. Schuwirth L, Ash J. Assessing tomorrow’s learners: In competency-based education only a radically different holistic method of assessment will work. Six things we could forget. Med Teach. 2013;35:555–559
53. Albanese MA. Challenges in using rater judgements in medical education. J Eval Clin Pract. 2000;6:305–319
54. Gingerich A, Regehr G, Eva KW. Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Acad Med. 2011;86(10 suppl):S1–S7
55. Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: A method for assessing clinical skills. Ann Intern Med. 2003;138:476–481
56. Alves de Lima A, Barrero C, Baratta S, et al. Validity, reliability, feasibility and satisfaction of the mini-clinical evaluation exercise (mini-CEX) for cardiology residency training. Med Teach. 2007;29:785–790
57. Lurie SJ, Mooney CJ, Lyness JM. Commentary: Pitfalls in assessment of competency-based educational objectives. Acad Med. 2011;86:412–414
58. Biggs J. Enhancing teaching through constructive alignment. High Educ. 1996;32:347–364
59. Kogan JR, Conforti LN, Iobst WF, Holmboe ES. Reconceptualizing variable rater assessments as both an educational and clinical care problem. Acad Med. 2014;89:721–727
60. Plake B, Wise L Standards for Educational and Psychological Testing. 2014 Washington, DC American Educational Research Association
61. Kane MT. Current concerns in validity theory. J Educ Meas. 2001;38:319–342
62. Kane MT. An argument-based approach to validity. Psychol Bull. 1992;112:527–535
63. Downing SM. Validity: On meaningful interpretation of assessment data. Med Educ. 2003;37:830–837
64. Wijnen-Meijer M, Van der Schaaf M, Booij M, et al. An argument-based approach to the validation of UHTRUST: Can we measure how recent graduates can be trusted with unfamiliar tasks? Adv Health Sci Educ Theory Pract. 2013;18:1009–1027
65. Dijksterhuis MGK, Voorhuis M, Teunissen PW, et al. Assessment of competence and progressive independence in postgraduate clinical training. Med Educ. 2009;43:1156–1165
66. Wijnen-Meijer M, van der Schaaf M, Nillesen K, Harendza S, Ten Cate O. Essential facets of competence that enable trust in graduates: A delphi study among physician educators in the Netherlands. J Grad Med Educ. 2013;5:46–53
67. Tiyyagura G, Balmer D, Chaudoin L, et al. The greater good: How supervising physicians make entrustment decisions in the pediatric emergency department. Acad Pediatr. 2014;14:597–602
68. Wijnen-Meijer M, van der Schaaf M, Nillesen K, Harendza S, Ten Cate O. Essential facets of competence that enable trust in medical graduates: A ranking study among physician educators in two countries. Perspect Med Educ. 2013;2:290–297
69. Kennedy TJ, Regehr G, Baker GR, Lingard L. Point-of-care assessment of medical trainee competence for independent clinical work. Acad Med. 2008;83(10 suppl):S89–S92
71. Roberts WR Aristotle. Rhetoric. 2004 Mineola, NY Dover
72. Pangaro L. “Forward feeding” about students’ progress: More information will enable better policy. Acad Med. 2008;83:802–803
73. Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: From Flexner to competencies. Acad Med. 2002;77:361–367
74. Frank JR, Snell LS, ten Cate O, et al. Competency-based medical education: Theory to practice. Med Teach. 2010;32:638–645
75. ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf MF. Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE Guide No. 99. Medical Teacher. 2015;37:983–1002