Entrustment Decision Making: Extending Miller’s Pyramid

Abstract

The iconic Miller’s pyramid, proposed in 1989, characterizes 4 levels of assessment in medical education (“knows,” “knows how,” “shows how,” “does”). The frame work has created a worldwide awareness of the need to have different assessment approaches for different expected outcomes of education and training. At the time, Miller stressed the innovative use of simulation techniques, geared at the third level (“shows how”); however, the “does” level, assessment in the workplace, remained a largely uncharted area.

In the 30 years since Miller’s conference address and seminal paper, much attention has been devoted to procedures and instrument development for workplace-based assessment. With the rise of competency-based medical education (CBME), the need for approaches to determine the competence of learners in the clinical workplace has intensified. The proposal to use entrustable professional activities as a framework of assessment and the related entrustment decision making for clinical responsibilities at designated levels of supervision of learners (e.g., direct, indirect, and no supervision) has become a recent critical innovation of CBME at the “does” level.

Analysis of the entrustment concept reveals that trust in a learner to work without assistance or supervision encompasses more than the observation of “doing” in practice (the “does” level). It implies the readiness of educators to accept the inherent risks involved in health care tasks and the judgment that the learner has enough experience to act appropriately when facing unexpected challenges. Earning this qualification requires qualities beyond observed proficiency, which led the authors to propose adding the level “trusted” to the apex of Miller’s pyramid.

At the 28th Research in Medical Education conference in 1989, George E. Miller (1918–1998), discussing the assessment of learners, stated:

No single assessment method can provide all the data required for judgment of anything so complex as the delivery of professional services by a successful physician. And so let me begin by suggesting a framework within which that assessment might occur. ¹

He then proceeded to present the iconic model that, brilliant in its simplicity, has withstood the test of time. ¹

What Is Miller’s Pyramid?

The pyramid is an organizing framework for assessing learners in the health professions, reflecting phases across the educational trajectory, in which targets of assessment in the lower levels are incorporated within higher levels (see Figure 1).

Figure 1: Miller’s pyramid, with explanatory text from the article in which the pyramid first appeared. ¹ (Used with permission.)

The model merges traditional assessment approaches, often emphasizing standardized, objective testing, with the vocational need of certification for practice. While Miller’s terminology of competence, positioned at a lower level than performance (“knows how”), may no longer align with more current conceptualizations of competence and performance, ^2,3 the core framework has proven to be very useful. In this Perspective, we use the word competence to signify a general medical ability that cuts across the levels of the pyramid. ² The multiple facets of the competence construct represented by the different levels of Miller’s pyramid make valuable and different contributions to assessment that do not perfectly correlate. ⁴ Another argument for distinguishing levels of Miller’s pyramid is that learners adapt their efforts and behavior to meet stated assessment criteria; in other words, assessment drives learning (one of Miller’s assertions ¹). Given this assumption, assessment should be programmed to guide learning ⁵ in an environment that stimulates the behavior aligning most with the desired outcomes of learning. For example, when mastery of knowledge or skills is the focus of assessment, learners should be acknowledged for the deliberate practice that leads to mastery itself. All levels of Miller’s pyramid are important, and the learning and assessment associated with each facet of competence build iteratively throughout the training.

Since 1990, progress has particularly been made in the quality of workplace-based assessment, that is, at Miller’s “does” level. ^6,7 Multiple tools with multiple points of workplace-based observations and assessment over time fit well with what has more recently been called programmatic assessment ^5,8 and a systems approach to assessment embedded in a curriculum. ⁹ The worldwide movement toward competency-based medical education (CBME) has incorporated workplace-based assessment in various forms, ^10,11 which align with the pyramid, ¹² as a major focus. Despite these developments, reliably assessing learners in the clinical workplace remains a huge challenge. The lack of standardized conditions for assessment; rater bias, such as generosity error and halo effects; and the paucity of evidence that relates trainee care quality to health care outcomes are a few of the barriers that have precluded the development of standards of workplace-based assessment at the “does” level. ^13–16 Training raters and using instruments with validity evidence can mitigate some of these challenges but not eliminate them. ¹⁷

The Dual Role of Diplomas in Medicine

While continued efforts to address psychometric inadequacies of workplace-based assessment are needed, something more fundamental also deserves attention. Degrees in medicine, more so than in other programs of higher education, go beyond just a testimony of academic accomplishment. A medical diploma attests (implicitly or explicitly) that medical graduates are prepared to safely care for patients, often from the very day of graduation. This means that examiners and programs have a dual responsibility in assessing learners: (1) to verify that learners have successfully done what they were asked to do (i.e., passed examinations; completed coursework, assignments, and rotations; practiced skills in simulated contexts; and demonstrated expected performance in the clinical environment) and (2) to ensure and protect the quality of patient care (i.e., verifying that learners can assume the responsibilities they face after graduation and assuring the public that the new graduate can provide safe and effective care). A medical license affords a legal responsibility; a specialty certification affords fully unsupervised practice. So, assessing medical learners is not just to verify course completions and progress decisions or award a diploma but also to express confidence that the learner is prepared for professional activities in health care. This dual responsibility should inform the judgments to grant learners the permission to engage in patient care. ¹⁸

Entrustment: Extending Beyond the Assessment of Observed Performance or Action

About 2 decades after Miller’s publication and 1 decade after the widespread adoption of CBME, the medical education community began adopting the concept of entrustable professional activities (EPAs). ¹⁹ An EPA, a term coined in 2005, can be defined as “a unit of professional practice (a task or group of tasks) that can be fully entrusted to a trainee, once he or she has demonstrated the necessary competence to execute this activity unsupervised.” ²⁰ Entrustment decision making is more than another approach or scale to assess medical trainees in the workplace. ²¹ The concept was introduced to operationalize CBME through a stepwise and safe engagement of learners in clinical practice with a progressive decrease of supervision. ²² Implementing EPAs requires a shift in focus from workplace-based assessment of isolated clinical competencies or abilities to decisions on whether to entrust learners with health care delivery tasks and responsibilities on the basis of the learners’ capability to safely and effectively perform those EPAs. ²³ The critical judgment comes into play with respect to the required level of supervision, in response to a learner’s level of readiness to perform a particular health care activity. ²⁴ However, what is less overt is that a summative entrustment decision incorporates trust in the trainee’s ability to handle unknown future situations, including securing adequate support if needed; this (future-facing) trust is based on what was previously observed, learned, and applied.

The Nature of Decisions to Entrust a Learner or Graduate With Health Care Tasks

The variety and breadth of expected clinical competencies are wider than can possibly be observed during training. Subsequent patients will not be the same as those previously diagnosed and treated, and contexts will differ from the ones in which the medical trainee was observed. Any activity in health care involves some level of risk, even for experienced clinicians. Entrustment decisions, ^18,25 therefore, imply an estimation of the learner’s readiness to perform in unknown contexts and the determination that the risks involved in trusting the learner are acceptable going forward. ^26,27 Making a summative entrustment decision that—from moment x on—the learner is permitted to perform EPA y is similar to awarding a driver’s license in continental Europe and trusting the driver will drive safely on the left side of the road when emerging from the Chunnel (underwater tunnel between France and the United Kingdom) on British soil. Even a move from proactive supervision, with a supervisor present (entrustment level 2), to reactive supervision, with a supervisor quickly available (entrustment level 3), may imply accepting risks. In other words, it is impossible to control for all possible situations in which the entrusted learner will act, and it is deceptive to suggest that prior observations have covered every possible context for the enactment of the EPA.

Entrustment decisions should, therefore, include the assessment of a learner’s ability to cope with risks, ask for help, and adapt when the outcomes of health care situations are different than what was expected. More succinctly, entrustment decisions, which focus on the future, should include an estimation of adaptive expertise. Assessing adaptive expertise or adaptive competence requires more than observing performance in the clinical environment. Scales that focus on supervision levels have been created with either a retrospective nature (how much supervision was provided with an activity?) or a prospective nature (do I feel the learner is ready for less supervision for this EPA?). Only the latter represents true entrustment. In radiology, ALARA (as low as reasonably achievable) seeks to provide the lowest level of radiation possible to perform a meaningful test and minimize exposure and its resultant risks. ²⁸ Likewise, the lowest level of acceptable performance risks corresponding with readiness for unsupervised practice may guide summative entrustment decisions and privilege.

Extending Miller’s Pyramid

Since Miller’s seminal address, many approaches to assessment in the clinical workplace have been developed, ^29,30 but all have focused on observing what learners do in the clinical environment, that is, Miller’s level 4 (the “does” level). However, as delineated earlier, there is something beyond the top of Miller’s “does” level. ^31,32 Observed performance in practice is what schools or programs should really deliver to society: graduates that patients, colleagues, and employers can “trust.” This presumptive trust, ^21,33 implicitly conferred at the completion of a physician’s educational program, extends beyond direct observation during training to any workplace physicians find themselves in, suggesting the need for a new pinnacle on Miller’s pyramid. Like all levels, this top level should naturally include and assume sufficient achievement at lower levels, so rather than replacing the “does” level, the new top level assesses something different. This is not a new thought: In 2016, Al-Eraky and Marei proposed incorporating trust in Miller’s pyramid. ³⁴

With these thoughts in mind, we propose an adaptation of Miller’s pyramid by adding a level above the “does” level and making slight adaptations to the current explanatory phrases (Figure 2). The 5 levels may be briefly explained in the focus of the assessment that fits with the level. Table 1 elaborates briefly on the focus and possible methods of assessment, in part inspired by the work of Yudkowsky et al. ³⁵

Table 1: Focus and Methods of Assessment in Medical Education Associated With Each Level of Miller’s Pyramid1 and With a Proposed New Level

Figure 2: Miller’s pyramid extended. A new fifth level (“trusted”) reflects the process for reaching the decision to award a learner an attestation of the completion of training, leading to a medical license or specialty registration or certification, that provides permission to act unsupervised and makes the grantors cognizant of the inherent risks. Note that explanatory phrases have been modified to reflect a more current use of the words “competence” and “performance.”

In this adaptation, the top of the pyramid reflects the process for reaching the decision to award a learner an attestation of the completion of training, leading to a medical license or specialty registration or certification, which provides permission to act unsupervised and makes the grantors cognizant of the inherent risks. Currently, physicians earn their diploma after successfully completing all the required elements of the curriculum, and the ticking clock gets to the preset end of clinical training experience. An administrative body then confirms that the trainee has the right to graduate, and so it happens. In graduate medical education, a program director, along with a single competency committee, oversees learners throughout the course of training, but the step toward unsupervised practice is still often determined by time rather than by observed competence and the future risk that comes with entrustment. In undergraduate medical education, however, the large number of students make it impossible for a single individual or committee to closely follow students’ progress from matriculation to graduation. Thus, the realization of the immense step of being granted the responsibility of a license, and the future risk involved, should drive the decision to graduate based on readiness for entrustment, not time.

The small steps informing entrustment, taken ad hoc in the workplace, seem to be made with more discernment than the big decision to graduate, license, or certify learners. To earn a diploma in higher education, including medical education, the accumulation of passed tests and assessments often suffices, but there are seldom committees or educators who really oversee the significance of the MD diploma for individual graduates. Licensing exams are, at most, an imperfect measure to support the big entrustment decision that the license implies. In small graduate medical education programs, there may be more oversight than in larger programs, but the scheduled end of training and timing of the board exam often dominate the decision to allow a trainee to complete the program.

With the introduction of EPAs, there is a middle ground. EPAs are units of professional practice with boundaries, allowing each EPA to be reasonably overseen, assessed, monitored, documented, and entrusted. Educators or, preferably, a clinical competency committee (CCC), should have the courage—and be empowered—to make thoughtful summative entrustment decisions with sufficient, multifaceted information, and those educators or the committee should then be accountable for those decisions. Some programs make these decisions with a signed certificate form for EPAs (with a statement of awarded responsibility or STAR), which reflects that accountability explicitly. ³⁶ More informally, that accountability can be the answer to a question: “Would I have this learner take care of my own family members?” ³⁷ or “Would I hire this graduate for my department if I could choose among applicants?” The development of a committee’s (or other body’s) trust that a learner can execute an activity unsupervised requires justification and conviction strong enough to enable the deciding body to assure patients the learner has mastered the domain sufficiently to justify a patient’s presumptive trust in him or her. It requires knowledge, experience, and courage to make a summative entrustment decision based on inherently limited previous observations. This reasoning makes “trusted”—with its inherent risks—a new level for Miller’s pyramid, beyond the “does” level.

Assessing Readiness for Entrustment

An entrustment decision requires a sense that the learner understands the risks involved in clinical care and confidence that the individual can deal with the unexpected. ³⁸ There is always the potential for things to go wrong, and it is critical that the learner knows what to do in unexpected circumstances. Both learners and clinical supervisors must know, weigh, and accept these risks when entrustment decisions are made.

In considering what clinicians take into account when making entrustment decisions, when transferring responsibility to a learner, or when titrating the amount of supervision needed, multiple variables emerge. ^39–42 Interestingly, most of these facets of competence are characteristics related to trustworthiness; are not task specific; and have been summarized as integrity, reliability, humility, and agency, in addition to an EPA-specific capability. ^18,21,43 Entrustment seems to require both trustworthiness and previously demonstrated competence in the domain being assessed. Together these facets can be used to weigh the risks involved in an entrustment decision.

Specific assessment approaches that incorporate trust, risk assessment, and adaptive expertise are limited currently. ^44,45 Along with better measures of the quality of care learners deliver, ⁴⁶ these approaches may become more important in the development of assessment in health professions education.

Readiness for entrustment is implied in licensing and certification. However, the top level of the revised pyramid applies not only to those big moments but also in programmatic assessment based on an EPA framework. If a learner is trusted with an individual EPA, he or she is considered able to perform the given EPA safely and effectively without supervision. However, because EPAs in aggregate define the professional activities of a specialty, a learner would be expected to achieve this same level of performance for all EPAs associated with direct patient care at the time of program completion. Over a practitioner’s career, there will be changes in the professional activities he or she is entrusted and entitled to perform. The dynamic nature of an individual’s EPA portfolio, which is based in a physician’s scope of practice, suggests the need for adapting over time what practitioners are entrusted with and entitled to do. ⁴⁷

While CCCs can and should focus on entrustment when making progression decisions for individual EPAs or EPAs in aggregate, ⁴⁸ individual supervisors should consider elements of entrustment in their observations, thinking, and recommendations, whenever possible. These elements may include, but are not limited to, learners exhibiting help-seeking behaviors, learners’ ability to apply previous learning in a new situation, and learners asking for the degree of needed supervision to practice at the edge of their competence. In other words, entrustment thinking, including the non–task-specific features, is not confined to committees (CCC or other) but is part of all workplace-based assessments. At lower levels of the pyramid, judgment based on what a learner actually does or has done is appropriate; at the higher levels, an inference of readiness is needed. Examples of practical approaches to support such inferences are entrustment-based discussions ⁴⁵ and realistic, live semisimulations. ^44,49 A recent report of the use and value of resident-sensitive quality measures in informing entrustment decisions also holds promise. ⁵⁰

Concluding Thoughts

Suggesting an extension of a model as iconic as Miller’s pyramid requires careful consideration. Others have created variations of the pyramid, calling it Miller’s triangle or prism; adding Bloom’s distinctions of knowledge, skills, and attitudes as a dimension ⁵¹; turning the pyramid upside down to stress the importance of the “does” level ³; adding “assessment orbits” around the pyramid ³⁴; and adding professional identity on top of the pyramid. ⁵² We considered such variations but decided to limit the adaptation of the pyramid to a concept that has a clear focus and practical value for the approach used to assess learners and adds to existing approaches already reflected in the pyramid. Cruess et al’s extension, ⁵² calling the top level “is” to reflect the core of a physician’s identity, embraces the personal characteristics we have described earlier (capability, integrity, reliability, humility, and agency).⁵³ Thus, our proposed new pinnacle for Miller’s pyramid combines personal characteristics linked to identity formation (“is”) with the mitigation of risk that is promised in the capacity to adapt from the familiar to the unfamiliar (“trusted”).

“Trusted” can be read in 2 ways. One is related to assessment and to a decision, that is, as a pivotal moment that leads to permission to act without supervision—it is the focus of our contribution. The other is a state of being. Being trusted may be regarded as one of the most crucial characteristics of physicians and other health care professionals and has important implications for continuing professional development. Deserved trust is not a trait but a state that requires ongoing self-awareness, reflection, and maintenance, by all health care professionals.

Acknowledgments:

The concept for this Perspective originated during a 2-day meeting of the International Competency-Based Medical Education Collaborators, Ottawa, Ontario, Canada; July 2019.