On the first question, ‘I would include this EPA in a list describing the current anaesthesiology curriculum’, 42% of the EPAs (24/57) showed a high score, 37% of the EPAs (21/57) showed a medium score and 21% of the EPAs (12/57) showed a low score.
The perceived difficulty of the EPAs (mean) varied from 3.6 to 6.6, as rated on a seven-point Likert scale. All six EPAs rated as ‘very difficult’ or ‘extremely difficult’ (mean 6.0 to 7.0) were found among both the medium and low-scoring EPAs on relevance for the curriculum (question 1). None of the proposed EPAs were considered ‘very easy’ or ‘extremely easy’ (mean 1.0 to 3.0).
The fourth question pertained to the moment the EPA should be mastered. The ‘very difficult’ and ‘extremely difficult’ EPAs were expected to be mastered later in training. For 11 EPAs (19%, 11/57), mastery was considered feasible only after an elective rotation in a specific area (Table 2 , all EPAs marked with d). For one EPA (‘Perioperative care during heart and lung transplantation’), mastery was not considered feasible at all during the training period of 5 years, but only during a fellowship after certification.
Remarks and missing entrustable professional activities
The majority of the participants (90%, 26/29) used an open text field during the first round to make one or more remarks on the EPAs and to provide feedback on the content of the EPAs. Input from open text boxes was analysed and grouped into 12 different themes. Table 3 gives an overview of the four major themes with more than 35 remarks each.
Fourteen participants (48%, 14/29) indicated that at least one EPA should be added to the list. ‘Obtaining central venous access’ was the only activity mentioned by more than two participants. This EPA was described in light of the remarks and added to the list. Other suggestions included EPAs on legal procedures, geriatric patients and consultations.
Second Delphi round
Participants were asked to rate the added EPA ‘obtaining central venous access’ using the questions from round one. Scores on this EPA can be found in Table 2 .
Applying the consensus rate of 80%, all high-scoring EPAs on relevance for the curriculum, (question 1 of the first round) passed to the near final set of EPAs with at least 90% agreement. All low-scoring EPAs on relevance were deleted from the list with at least 80% agreement. Of the 22 EPAs with medium scores on relevance, all EPAs, except one were endorsed by at least three participants. Based on this outcome, we deleted this particular EPA (‘Perioperative anaesthetic care during trauma neurosurgery’) and let the other 21 medium-scoring EPAs pass to the near-final list.
After two rounds, the near-final list of EPAs describing the core anaesthesiology curriculum consisted of 45 EPAs.
Third Delphi round
Twenty-two participants (82%, 22/27) fully agreed with the near-final set of 45 EPAs obtained from the second round. Two participants commented on two EPAs. It was suggested that EPA ‘perioperative anaesthetic care for American Society of Anesthesiologists (ASA) IV patients undergoing low to medium-risk surgery’ and EPA ‘perioperative anaesthetic care for ASA I to II patients undergoing low to medium-risk surgery’ and EPA ‘perioperative anaesthetic care for ASA III patients undergoing low to medium-risk surgery’ were merged. A medium-scoring EPA, ‘Science and evidence-based medicine (EBM)’ was considered by several participants to be difficult to fit in the curriculum as an EPA (‘Requires courses in EBM/statistics/epidemiology and additional software’ and ‘Mastery at the end of year 4, because of the need of certain amounts of experience to put the data and articles in perspective’). Two EPAs (‘postoperative care during the recovery period’ and ‘perioperative anaesthetic care for ASA III patients undergoing low to medium-risk surgery’) received comments despite being high-scoring EPAs that had passed into the third round with 100% approval. An alphabetical overview of the final list of 45 EPAs after round three, including the descriptions, is provided as supplemental Digital Content (Appendix 1, http://links.lww.com/EJA/A96).
Discussion and conclusion
In this study, 27 experts reached consensus on a shortlist of 45 EPAs, based on the pre-defined longlist that describes the desired learning outcomes of a core curriculum in anaesthesiology for the Netherlands. These EPAs constitute a comprehensive range of activities from daily anaesthetic practice, including perioperative anaesthetic care, pain management and resuscitation. With 82% agreement on the final list of EPAs, we reached a high degree of consensus among experts in anaesthesiology training.
Getting curriculum reform proposals accepted by faculty is a recognised challenge in the implementation of competency-based curricula.29 Programme directors are key participants in shaping the curriculum and integrating an EPA-based anaesthesiology curriculum into the workplace in the future. The high level of agreement across national residency programme directors on the final set of EPAs may be indicative of support for such curriculum reform. A next step, to expand acceptance by a broader group of stakeholders, could be to involve other key participants such as attending clinicians and trainees. This could be part of a study validating these EPAs in practice. We chose to limit our sample to programme directors, because of their overview on both the training programme as well as job requirements.
During the progression of the Delphi survey, participants may have developed a growing understanding of the EPA concept, after having received briefing and written information about the study and EPAs. The fact that participants suggested an additional EPA to the list after round one suggests that they had a fair understanding of the concept.
In a traditional Delphi design, participants could have generated ideas on possible EPA titles as a first input for consecutive rounds of consensus finding. Their relative unfamiliarity with the EPA concept and the busy schedules of programme directors made us choose to provide participants with a set of pre-defined EPAs based on the competencies described in the current national training programme.28 This saved the participants from a time-consuming round preceding round one. Although providing pre-defined material to be discussed in the first round is common practice in Delphi studies, it may have influenced the composition of the set of EPAs.
The preliminary list of EPAs was prepared by a relative small group of well informed researchers, who had knowledge of both the current national anaesthesiology training programme28 and the EPA concept. The preliminary EPAs were formed after document analysis and a rearrangement of the required competencies. Even though the EPAs are a representation of the nationally used training programme, another group of individuals involved in such a process might rearrange the competencies slightly differently, resulting in somewhat differently worded EPAs.
The EPA ‘Science and EBM’ was part of the initial list of EPAs and, as with all the other EPAs, only a title and short description were provided. However, this EPA does not meet the definition provided by Ten Cate22 and is more an overarching competency applicable to many EPAs. Given the prominent place, this competency has in the current national framework, and wanting to represent the curriculum as closely and completely as possible, we formulated the EPA ‘Science and EBM’. Participating programme directors decided to include this EPA in the final list, underlining its significance for anaesthesiology training. At the same time, however, several programme directors found it hard to fit this EPA in the curriculum. This result shows the complexity of the transformation toward an EPA-based curriculum.
The suggested advantages of CBME are the incorporation of clear learning outcomes, the inclusion of competencies other than medical expertise alone, independence from time-based training and individualised learning.10 The implementation of an EPA-based curriculum may help ensure that the advantages of CBME are fully exploited. CBME uses clearly defined learning outcomes, describing the desired abilities of graduates, formulated as competencies. These competencies are often generic, that is, not specialty specific, and thus detached from clinical practice. EPAs define learning outcomes as relevant profession-specific activities that require the desired competencies of the physician. EPAs therefore link competencies to medical practice.10,30
With its emphasis on learning outcomes, CBME disengages from time-based training.10 Fixed learning outcomes, that is, mastering an EPA, may be met by trainees at different points in time. As a consequence, allowing trainees to go through the programme at their own pace asks for a flexible length of training, providing extra training time where needed and more rapid progression when possible. This way, EPAs can promote a true individualised approach to medical education. This poses a formidable logistical challenge for a training institution with a time-based programme and transformation to an EPA-based curriculum will only be feasible if more flexibility is inserted into the curriculum.31 In addition, fast progressing trainees may be challenged to continue to develop; this may be fostered by adding elective EPAs to the core set of EPAs.
The pre-defined list of EPAs was based on the current national anaesthesiology training programme28 in the Netherlands and, therefore, the proposed final list covers the learning outcomes of a Dutch national curriculum. Although job requirements of anaesthesiologists, and subsequently the EPAs needed, differ between countries, the process of constructing EPAs based on an existing curriculum remains the same process. When more countries adopt an EPA-based curriculum, a study that compares their differences would be interesting and might provide insight into whether harmonisation of anaesthesiology training in Europe can be achieved.32
This article presents a proposed list of EPAs describing postgraduate medical training in anaesthesiology. Determining a consensus set of EPAs is the first step in the development of an EPA-based curriculum. Next, further refinement of the EPAs is needed by providing a full description for each EPA, mapping the EPAs to competencies and developing assessment tools and performance standards.20 Subsequently, the EPAs need to be validated in actual training practice. Moreover, the use of EPAs has to be compared with the current, more time-based, training, to evaluate if EPA-based curriculum indeed improves postgraduate training in anaesthesiology. With this study, a step is made to move from the current curriculum to a more contemporary approach to postgraduate anaesthesiology training. Although our study considered only the Dutch anaesthesiology programme, we assume that the resulting overview of EPAs will generate a more international discussion on standards for anaesthesiology training.
Acknowledgments relating to this article
Assistance with the study: none.
Financial support and sponsorship: funding was provided by the European Union's Seventh Framework Programme for research, technological development and demonstration under grant agreement 619349. Project name ‘WATCHME: Workplace-based e-assessment technology for competency-based higher multi-professional education’.
Conflicts of interest: none.
1. Morcke AM, Dornan T, Eika B. Outcome (competency) based education: an exploration of its origins, theoretical basis, and empirical evidence. Adv Heal Sci Educ
2. Frank JR, Mungroo R, Ahmad Y, et al. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach
3. Frank JR. The CanMEDS 2005 Physician Competency Framework
4. Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach
5. Grant J. The incapacitating effects of competence: a critique. Adv Heal Sci Educ Theory Pract
6. Glass JM. Competency based training is a framework for incompetence. BMJ
7. Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the accreditation council for graduate medical education: a systematic review. Acad Med
8. Pangaro L, ten Cate O. Frameworks for learner assessment in medicine: AMEE Guide No. 78. Med Teach
9. Lurie SJ, Mooney CJ, Lyness JM. Commentary: pitfalls in assessment of competency-based educational objectives. Acad Med
10. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach
11. Gilhooly J, Schumacher DJ, West DC, et al. The promise and challenge of entrustable professional activities. Pediatrics
12. ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the healthcare environment. Med Teach
13. Hauer KE, Ten Cate O, Boscardin C, et al. Understanding trust as an essential element of trainee supervision and learning in the workplace. Adv Health Sci Educ Theory Prac
14. Sterkenburg A, Barach P, Kalkman C, et al. When do supervising physicians decide to entrust residents with unsupervised tasks? Acad Med
15. Boyce P, Spratt C, Davies M, et al. Using entrustable professional activities to guide curriculum development in psychiatry training. BMC Med Educ
16. Chang A, Bowen JL, Buranosky RA, et al. Transforming primary care training-patient-centered medical home entrustable professional activities for internal medicine residents. J Gen Intern Med
17. Swing SR. Perspectives on competency-based medical education from the learning sciences. Med Teach
18. Ten Cate O, Chen HC, Hoff RG, et al. Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE Guide No. 99. Med Teach
19. van Loon K, Driessen EW, Teunissen PW, et al. Experiences with EPAs, potential benefits and pitfalls. Med Teach
20. Jonker G, Hoff RG, Ten Cate OTJ. A case for competency-based anaesthesiology training with entrustable professional activities: an agenda for development and research. Eur J Anaesthesiol
21. Ten Cate O. Entrustability of professional activities and competency-based training. Med Educ
22. Ten Cate O. Nuts and bolts of entrustable professional activities. J Grad Med Educ
23. Green RA. The Delphi technique in educational research. SAGE Open
24. Mckenna HP. The Delphi technique: a worthwhile research approach for nursing? J Adv Nurs
25. Shaughnessy AF, Sparks J, Cohen-Osher M, et al. Entrustable professional activities in family medicine. J Grad Med Educ
26. Hauer KE, Kohlwes J, Cornett P, et al. Identifying entrustable professional activities in internal medicine training. J Grad Med Educ
27. Hauer KE, Boscardin C, Fulton TB, et al. Using a curricular vision to define entrustable professional activities for medical student assessment. J Gen Intern Med
28. Driessen E, van Kleef J, Knape H, et al.. Project Modernisering Opleidingsplan Anesthesiologie
.; 2008. (Dutch).
29. Ebert TJ, Fox CA. Competency-based education in anesthesiology. Anesthesiology
30. Ten Cate O. Trusting graduates to enter residency: what does it take? J Grad Med Educ
31. Carraccio C, Englander R, Van Melle E, et al. Advancing competency-based medical education: a charter of clinician-educators. Acad Med
2015; (in press).
32. Van Gessel EF, Ostergard HT, Niemi-Murola L. Harmonisation of anaesthesiology training in Europe. Best Pract Res Clin Anaesthesiol
© 2016 European Society of Anaesthesiology
33. Miller GE. The assessment of clinical skills/competence/performance. Acad Med