Secondary Logo

Journal Logo

Education, simulation and resuscitation

Transforming the learning outcomes of anaesthesiology training into entrustable professional activities

A Delphi study

Wisman-Zwarter, Nienke; van der Schaaf, Marieke; ten Cate, Olle; Jonker, Gersten; van Klei, Wilton A.; Hoff, Reinier G.

Author Information
European Journal of Anaesthesiology: August 2016 - Volume 33 - Issue 8 - p 559-567
doi: 10.1097/EJA.0000000000000474


This original article is accompanied by the following invited commentary:

Savoldelli GL, Van Gessel EF. ‘Entrustable professional activities’: the way to go for competency-based curriculum? Eur J Anaesthesiol 2016; 33:557–558


Many medical training programmes across the world have adopted a competency-based curriculum in undergraduate1 and postgraduate2 medical education. In general, these programmes are based on competency frameworks such as the Canadian Medical Educational Directives for Specialists3 or the framework of the US Accreditation Council for Graduate Medical Education4. Although these competency frameworks are useful, they have also met with criticism. One worry is that separate competencies may never represent the true competence of the physician.5,6 Another worry is that competencies cannot be validly assessed.7 Competency frameworks use an analytic approach as they divide competence into many competencies and sub-competencies.8 The assessment of trainees can result in using largely theoretical checklists of competencies and behaviour that are detached from daily practice. As such, competencies are difficult to observe and assess validly as separate entities.9

Competency-based medical education (CBME) was introduced to produce clinicians who meet recognised standards of competence, in contrast to an existing culture where learners spend a pre-determined time in training without a need to provide much evidence of their competence.10 Increased opportunity for part-time training and fewer duty hours limit the net time spent in the workplace, requiring best use of remaining training time. Instead of dwelling in a rotation for a pre-determined period, trainees should progress to the next rotation when learning outcomes are met. This requires flexibility of the curriculum. Current programmes are poorly suited for an individual approach. To better ground CBME in clinical practice and to allow for more flexibility in medical curricula, postgraduate medical training programmes have begun to consider using entrustable professional activities (EPAs) to redesign their curricula.11–16 The concept of EPAs may contribute to a more flexible, personalised and less time-dependent approach to training programmes.12

EPAs are units of professional practice, profession-specific tasks, intended for clinicians that already possess multiple, integrated competencies together with the knowledge that underpins them and appropriate skills and attitudes.17,18 An EPA is a task that can be entrusted to a trainee with sufficient competence. A curriculum with EPAs can link clinical training and assessment to the work that clinicians actually do in daily practice.19 They are characterised by the fact that learners are not permitted to execute them unsupervised until they have demonstrated their competence to deserve this trust.

Typical examples of EPAs in anaesthesiology would be ‘Epidural analgesia for labour’ or ‘Admitting a critically ill patient to the ICU’.20 Becoming competent in an EPA is a process that can be envisaged as moving through developmental stages that require decreasing levels of supervision, ranging from observing the EPA, to executing it with direct supervision to performing it unsupervised.21,22 In this way, an EPA can provide insight into a trainee's progress regarding a specific unit of work. When competence in an EPA is demonstrated a decision can be made to entrust the trainee to perform the task without direct supervision.10

The aim of our study was to identify EPAs for postgraduate training in anaesthesiology. We also wished to provide an example of how an existing curriculum could be transformed into one based on EPAs. We used a national consensus procedure involving all Dutch anaesthesiology programme directors.


Ethical approval

Ethical approval for this study was provided by the Ethical Review Board of the Netherlands Association for Medical Education (NERB file number 00348, 23 May 2014). Informed consent from the participants was obtained in the first round of the Delphi survey.


A modified Delphi method was used to reach consensus among experts in three rounds on an initial pre-defined list of preliminary EPAs. The process of generating the list of preliminary EPAs and the Delphi method are described in detail below (Fig. 1).

Fig. 1:
Total n = number of participants invited to participate in the specific Delphi round. EPA, entrustable professional activity.

The Delphi technique is an iterative consultation of experts without interaction giving equal weighting to all individual opinions.23,24 During a Delphi, each round presents to the experts a set of questions based on the outcome of the previous round, formulated to seek an opinion. Between rounds, an anonymised summary of the aggregated results is provided to the participants. Successive rounds are carried out until agreement between the experts is reached. The Delphi procedure has been used successfully by other research teams to establish and validate EPAs.25–27


The Netherlands Society of Anaesthesiology sets the curriculum for Dutch anaesthesiology training. The duration of postgraduate specialty training in anaesthesiology in the Netherlands has been 5 years for several decades. For approximately a decade, this training programme in the Netherlands has been competency based. The programme is delivered by training site clusters across the country, each consisting of a primary teaching hospital (nine in total), and several secondary teaching hospitals (31 in total). Each of these hospitals has a programme director responsible for local anaesthesiology training.


All but one programme director in anaesthesiology across the country were invited to participate in the Delphi survey (n = 39); one programme director, being involved as a researcher in the current study, could not participate. Programme directors were the target group because of their expertise in anaesthesiology training.


An overview of the procedure of the Delphi survey is shown in Fig. 1. Three researchers analysed the current national anaesthesiology training programme,28 arranging competency-based educational requirements and subsuming them into EPAs, whereas remaining as close as possible to the original programme. This resulted in 41 preliminary EPAs that were supplemented with 16 EPAs based on recent developments in the field and related to common surgical procedures. Members of the Education Committee of the anaesthesiology department of UMC Utrecht (eight senior anaesthesiologists and three senior trainees) evaluated the list of preliminary EPAs for clarity and completeness (pilot phase). Programme directors were briefed on the concept of EPAs and the purpose of the study during a national meeting of programme directors in anaesthesiology. Thereafter, background information on EPAs, based on the article of Ten Cate,22 and the longlist of preliminary EPAs were distributed among the eligible participants prior to the first round. During the first round, participants were asked to answer four questions for each EPA on a seven-point Likert-type scale (Table 1, upper panel). In addition, for each EPA, an open text box was provided for feedback and elaboration of the answers given. Ultimately, participants had to indicate if the list of EPAs was comprehensive (yes or no) and to elaborate on any EPAs that should be added. If additional EPAs were suggested by more than two experts, they were included in the questionnaire of the second Delphi round.

Table 1:
Overview of questions asked during the Delphi survey

For the second round, all preliminary EPAs were ranked according to their mean score on the statement ‘I would include this EPA in a list describing the current anaesthesiology curriculum’ as high, medium or low scoring. In the second round, three questions were asked about each EPA (Table 1, middle panel). At least 80% of the participants had to agree to retain an high-scoring EPA (question 1) or disagree to delete a low-scoring EPA from the list (question 2). Finally, participants were asked to choose five EPAs among the medium-scoring list as most essential to add to the list of high-scoring EPAs, yielding a near-final list of EPAs to describe the current curriculum.

In the third and final round, a near-final list of EPAs derived from the second round was presented. Participants were asked if they agree with the full list or not (Table 1, lower panel) and had the opportunity to elaborate on the list. Again, an 80% consensus rate was applied to accept or reject the final list.

Data analysis

Mean scores and SDs were calculated for each question in the first round. EPAs were divided into three groups: high, medium and low-scoring EPAs. A high score was defined as a mean of at least 6.0 out of 7 on question 1 (‘I would include this EPA in a list describing the current anaesthesiology curriculum’). A medium score was defined as a mean of 5.0 to 6.0, and a low score as a mean of less than 5.0. In the second and third round, frequencies and percentages were determined for each question. Qualitative data were analysed using a template method in which emerging themes were summarised and organised in a data set.


Twenty-nine programme directors participated in the first and second round (74% response rate) and 27 participated in round three (69% overall response rate). Two programme directors declined to participate, six programme directors did not respond to repeated invitations to take part in round one and four programme directors did not respond during round one (n = 2) or after round two (n = 2). The mean age of responders was 52 years (4.8) and they had 19 years (6.0) of experience in their field.

First Delphi round

Table 2 gives an overview of mean scores and SDs of all EPAs on the four questions in round one.

Table 2:
Scores for preliminary entrustable professional activities on the questions of the first Delphi round
Table 2:
(Continued) Scores for preliminary entrustable professional activities on the questions of the first Delphi round

On the first question, ‘I would include this EPA in a list describing the current anaesthesiology curriculum’, 42% of the EPAs (24/57) showed a high score, 37% of the EPAs (21/57) showed a medium score and 21% of the EPAs (12/57) showed a low score.

The perceived difficulty of the EPAs (mean) varied from 3.6 to 6.6, as rated on a seven-point Likert scale. All six EPAs rated as ‘very difficult’ or ‘extremely difficult’ (mean 6.0 to 7.0) were found among both the medium and low-scoring EPAs on relevance for the curriculum (question 1). None of the proposed EPAs were considered ‘very easy’ or ‘extremely easy’ (mean 1.0 to 3.0).

The fourth question pertained to the moment the EPA should be mastered. The ‘very difficult’ and ‘extremely difficult’ EPAs were expected to be mastered later in training. For 11 EPAs (19%, 11/57), mastery was considered feasible only after an elective rotation in a specific area (Table 2, all EPAs marked with d). For one EPA (‘Perioperative care during heart and lung transplantation’), mastery was not considered feasible at all during the training period of 5 years, but only during a fellowship after certification.

Remarks and missing entrustable professional activities

The majority of the participants (90%, 26/29) used an open text field during the first round to make one or more remarks on the EPAs and to provide feedback on the content of the EPAs. Input from open text boxes was analysed and grouped into 12 different themes. Table 3 gives an overview of the four major themes with more than 35 remarks each.

Table 3:
Overview of the most frequent named themes during round one

Fourteen participants (48%, 14/29) indicated that at least one EPA should be added to the list. ‘Obtaining central venous access’ was the only activity mentioned by more than two participants. This EPA was described in light of the remarks and added to the list. Other suggestions included EPAs on legal procedures, geriatric patients and consultations.

Second Delphi round

Participants were asked to rate the added EPA ‘obtaining central venous access’ using the questions from round one. Scores on this EPA can be found in Table 2.

Applying the consensus rate of 80%, all high-scoring EPAs on relevance for the curriculum, (question 1 of the first round) passed to the near final set of EPAs with at least 90% agreement. All low-scoring EPAs on relevance were deleted from the list with at least 80% agreement. Of the 22 EPAs with medium scores on relevance, all EPAs, except one were endorsed by at least three participants. Based on this outcome, we deleted this particular EPA (‘Perioperative anaesthetic care during trauma neurosurgery’) and let the other 21 medium-scoring EPAs pass to the near-final list.

After two rounds, the near-final list of EPAs describing the core anaesthesiology curriculum consisted of 45 EPAs.

Third Delphi round

Twenty-two participants (82%, 22/27) fully agreed with the near-final set of 45 EPAs obtained from the second round. Two participants commented on two EPAs. It was suggested that EPA ‘perioperative anaesthetic care for American Society of Anesthesiologists (ASA) IV patients undergoing low to medium-risk surgery’ and EPA ‘perioperative anaesthetic care for ASA I to II patients undergoing low to medium-risk surgery’ and EPA ‘perioperative anaesthetic care for ASA III patients undergoing low to medium-risk surgery’ were merged. A medium-scoring EPA, ‘Science and evidence-based medicine (EBM)’ was considered by several participants to be difficult to fit in the curriculum as an EPA (‘Requires courses in EBM/statistics/epidemiology and additional software’ and ‘Mastery at the end of year 4, because of the need of certain amounts of experience to put the data and articles in perspective’). Two EPAs (‘postoperative care during the recovery period’ and ‘perioperative anaesthetic care for ASA III patients undergoing low to medium-risk surgery’) received comments despite being high-scoring EPAs that had passed into the third round with 100% approval. An alphabetical overview of the final list of 45 EPAs after round three, including the descriptions, is provided as supplemental Digital Content (Appendix 1,

Discussion and conclusion

In this study, 27 experts reached consensus on a shortlist of 45 EPAs, based on the pre-defined longlist that describes the desired learning outcomes of a core curriculum in anaesthesiology for the Netherlands. These EPAs constitute a comprehensive range of activities from daily anaesthetic practice, including perioperative anaesthetic care, pain management and resuscitation. With 82% agreement on the final list of EPAs, we reached a high degree of consensus among experts in anaesthesiology training.

Getting curriculum reform proposals accepted by faculty is a recognised challenge in the implementation of competency-based curricula.29 Programme directors are key participants in shaping the curriculum and integrating an EPA-based anaesthesiology curriculum into the workplace in the future. The high level of agreement across national residency programme directors on the final set of EPAs may be indicative of support for such curriculum reform. A next step, to expand acceptance by a broader group of stakeholders, could be to involve other key participants such as attending clinicians and trainees. This could be part of a study validating these EPAs in practice. We chose to limit our sample to programme directors, because of their overview on both the training programme as well as job requirements.

During the progression of the Delphi survey, participants may have developed a growing understanding of the EPA concept, after having received briefing and written information about the study and EPAs. The fact that participants suggested an additional EPA to the list after round one suggests that they had a fair understanding of the concept.

In a traditional Delphi design, participants could have generated ideas on possible EPA titles as a first input for consecutive rounds of consensus finding. Their relative unfamiliarity with the EPA concept and the busy schedules of programme directors made us choose to provide participants with a set of pre-defined EPAs based on the competencies described in the current national training programme.28 This saved the participants from a time-consuming round preceding round one. Although providing pre-defined material to be discussed in the first round is common practice in Delphi studies, it may have influenced the composition of the set of EPAs.

The preliminary list of EPAs was prepared by a relative small group of well informed researchers, who had knowledge of both the current national anaesthesiology training programme28 and the EPA concept. The preliminary EPAs were formed after document analysis and a rearrangement of the required competencies. Even though the EPAs are a representation of the nationally used training programme, another group of individuals involved in such a process might rearrange the competencies slightly differently, resulting in somewhat differently worded EPAs.

The EPA ‘Science and EBM’ was part of the initial list of EPAs and, as with all the other EPAs, only a title and short description were provided. However, this EPA does not meet the definition provided by Ten Cate22 and is more an overarching competency applicable to many EPAs. Given the prominent place, this competency has in the current national framework, and wanting to represent the curriculum as closely and completely as possible, we formulated the EPA ‘Science and EBM’. Participating programme directors decided to include this EPA in the final list, underlining its significance for anaesthesiology training. At the same time, however, several programme directors found it hard to fit this EPA in the curriculum. This result shows the complexity of the transformation toward an EPA-based curriculum.

The suggested advantages of CBME are the incorporation of clear learning outcomes, the inclusion of competencies other than medical expertise alone, independence from time-based training and individualised learning.10 The implementation of an EPA-based curriculum may help ensure that the advantages of CBME are fully exploited. CBME uses clearly defined learning outcomes, describing the desired abilities of graduates, formulated as competencies. These competencies are often generic, that is, not specialty specific, and thus detached from clinical practice. EPAs define learning outcomes as relevant profession-specific activities that require the desired competencies of the physician. EPAs therefore link competencies to medical practice.10,30

With its emphasis on learning outcomes, CBME disengages from time-based training.10 Fixed learning outcomes, that is, mastering an EPA, may be met by trainees at different points in time. As a consequence, allowing trainees to go through the programme at their own pace asks for a flexible length of training, providing extra training time where needed and more rapid progression when possible. This way, EPAs can promote a true individualised approach to medical education. This poses a formidable logistical challenge for a training institution with a time-based programme and transformation to an EPA-based curriculum will only be feasible if more flexibility is inserted into the curriculum.31 In addition, fast progressing trainees may be challenged to continue to develop; this may be fostered by adding elective EPAs to the core set of EPAs.

The pre-defined list of EPAs was based on the current national anaesthesiology training programme28 in the Netherlands and, therefore, the proposed final list covers the learning outcomes of a Dutch national curriculum. Although job requirements of anaesthesiologists, and subsequently the EPAs needed, differ between countries, the process of constructing EPAs based on an existing curriculum remains the same process. When more countries adopt an EPA-based curriculum, a study that compares their differences would be interesting and might provide insight into whether harmonisation of anaesthesiology training in Europe can be achieved.32

This article presents a proposed list of EPAs describing postgraduate medical training in anaesthesiology. Determining a consensus set of EPAs is the first step in the development of an EPA-based curriculum. Next, further refinement of the EPAs is needed by providing a full description for each EPA, mapping the EPAs to competencies and developing assessment tools and performance standards.20 Subsequently, the EPAs need to be validated in actual training practice. Moreover, the use of EPAs has to be compared with the current, more time-based, training, to evaluate if EPA-based curriculum indeed improves postgraduate training in anaesthesiology. With this study, a step is made to move from the current curriculum to a more contemporary approach to postgraduate anaesthesiology training. Although our study considered only the Dutch anaesthesiology programme, we assume that the resulting overview of EPAs will generate a more international discussion on standards for anaesthesiology training.

Acknowledgments relating to this article

Assistance with the study: none.

Financial support and sponsorship: funding was provided by the European Union's Seventh Framework Programme for research, technological development and demonstration under grant agreement 619349. Project name ‘WATCHME: Workplace-based e-assessment technology for competency-based higher multi-professional education’.

Conflicts of interest: none.

Presentation: none.


1. Morcke AM, Dornan T, Eika B. Outcome (competency) based education: an exploration of its origins, theoretical basis, and empirical evidence. Adv Heal Sci Educ 2013; 18:851–863.
2. Frank JR, Mungroo R, Ahmad Y, et al. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach 2010; 32:631–637.
3. Frank JR. The CanMEDS 2005 Physician Competency Framework.; 2005.
4. Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach 2007; 29:648–654.
5. Grant J. The incapacitating effects of competence: a critique. Adv Heal Sci Educ Theory Pract 1999; 4:271–277.
6. Glass JM. Competency based training is a framework for incompetence. BMJ 2014; 348:g2909.
7. Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the accreditation council for graduate medical education: a systematic review. Acad Med 2009; 84:301–309.
8. Pangaro L, ten Cate O. Frameworks for learner assessment in medicine: AMEE Guide No. 78. Med Teach 2013; 35:e1197–e1210.
9. Lurie SJ, Mooney CJ, Lyness JM. Commentary: pitfalls in assessment of competency-based educational objectives. Acad Med 2011; 86:412–414.
10. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach 2010; 32:638–645.
11. Gilhooly J, Schumacher DJ, West DC, et al. The promise and challenge of entrustable professional activities. Pediatrics 2014; 133:S78–S79.
12. ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the healthcare environment. Med Teach 2010; 32:669–675.
13. Hauer KE, Ten Cate O, Boscardin C, et al. Understanding trust as an essential element of trainee supervision and learning in the workplace. Adv Health Sci Educ Theory Prac 2014; 19:435–456.
14. Sterkenburg A, Barach P, Kalkman C, et al. When do supervising physicians decide to entrust residents with unsupervised tasks? Acad Med 2010; 85:1408–1417.
15. Boyce P, Spratt C, Davies M, et al. Using entrustable professional activities to guide curriculum development in psychiatry training. BMC Med Educ 2011; 11:96.
16. Chang A, Bowen JL, Buranosky RA, et al. Transforming primary care training-patient-centered medical home entrustable professional activities for internal medicine residents. J Gen Intern Med 2013; 28:801–809.
17. Swing SR. Perspectives on competency-based medical education from the learning sciences. Med Teach 2010; 32:663–668.
18. Ten Cate O, Chen HC, Hoff RG, et al. Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE Guide No. 99. Med Teach 2015; 99:1–20.
19. van Loon K, Driessen EW, Teunissen PW, et al. Experiences with EPAs, potential benefits and pitfalls. Med Teach 2014; 36:698–702.
20. Jonker G, Hoff RG, Ten Cate OTJ. A case for competency-based anaesthesiology training with entrustable professional activities: an agenda for development and research. Eur J Anaesthesiol 2014; 32:1–6.
21. Ten Cate O. Entrustability of professional activities and competency-based training. Med Educ 2005; 39:1176–1177.
22. Ten Cate O. Nuts and bolts of entrustable professional activities. J Grad Med Educ 2013; 5:157–158.
23. Green RA. The Delphi technique in educational research. SAGE Open 2014; 4:1–8.
24. Mckenna HP. The Delphi technique: a worthwhile research approach for nursing? J Adv Nurs 1994; 19:1221–1225.
25. Shaughnessy AF, Sparks J, Cohen-Osher M, et al. Entrustable professional activities in family medicine. J Grad Med Educ 2013; 5:112–118.
26. Hauer KE, Kohlwes J, Cornett P, et al. Identifying entrustable professional activities in internal medicine training. J Grad Med Educ 2013; 5:54–59.
27. Hauer KE, Boscardin C, Fulton TB, et al. Using a curricular vision to define entrustable professional activities for medical student assessment. J Gen Intern Med 2015; 30:1344–1348.
28. Driessen E, van Kleef J, Knape H, et al.. Project Modernisering Opleidingsplan Anesthesiologie.; 2008. (Dutch).
29. Ebert TJ, Fox CA. Competency-based education in anesthesiology. Anesthesiology 2014; 120:24–31.
30. Ten Cate O. Trusting graduates to enter residency: what does it take? J Grad Med Educ 2014; 6:7–10.
31. Carraccio C, Englander R, Van Melle E, et al. Advancing competency-based medical education: a charter of clinician-educators. Acad Med 2015; (in press).
32. Van Gessel EF, Ostergard HT, Niemi-Murola L. Harmonisation of anaesthesiology training in Europe. Best Pract Res Clin Anaesthesiol 2012; 26:55–67.
33. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990; 65:63–67.

    Supplemental Digital Content

    © 2016 European Society of Anaesthesiology