Metacognition refers to a person's ability to regulate their thinking. In medicine, besides the general definition, metacognition includes checking clinical reasoning for possible bias, seeing the illness from a patient's perspective, and assessing what one needs to know about a treatment option.1 Ophthalmo-logists are expected to be self-directed learners throughout their careers, and metacognitive skills are critical to this in addressing what to learn, when to learn, and how to learn. This review covers concepts of metacognition, metacognition in medical education and for the continuing professional development (CPD) learner in particular, and strategies to improve metacognition while undertaking CPD activities. This review functions as a primer for all learners and an early alert for an educational process that will be likely to find its way into the accepted and desired fundamental medical educational experience, of which all those involved in ophthalmology education should be aware of.
We searched the articles in PubMed on April 25, 2018, using the terms “metacognition” and “continuing medical education (CME)” or “continuing professional development (CPD)” or “continuous professional development”. There were no limits to dates or language, and the research resulted in 5 articles,2-6 including a quantitative study of metacognitive skills in early-career pediatricians3; a qualitative study of the benefits of clinical teaching by experienced family practitioners, which included improved metacognition4; reviews and commentaries2,6; and a study pertaining to medical students only.5 There were no ophthalmology-specific papers. Due to the paucity of literature it is not possible to write a comprehensive literature review of metacognition in CPD based on empirical studies. Instead, this review aimed to provide practical applications of metacognition in CPD undertaken by practicing ophthalmologists, and to do so we have extrapolated from relevant literature for medical students and residency training in other medical disciplines, along with other teaching fields, based on reference lists of the 5 returned articles and the authors' experiences.
CONCEPTS IN METACOGNITION AND COGNITION
The concept of metacognition was originated by Flavell.7 Metacognition can be defined as “thinking about thinking”, or the “higher order thinking which involves active control over the cognitive processes engaged in learning”.8 According to Flavell,7 it comprises metacognitive knowledge (acquired knowledge about cognitive processes) and metacognitive regulation,1 which consists of planning, monitoring, and evaluation.9 Metacognition is associated with intelligence10 and academic achievement. Disorders of metacognition are described in multiple psychopathologies.11
Cognition refers to a range of mental processes related to the acquisition, storage, manipulation, and retrieval of information. Figure 1 shows the relationship between cognition and metacognition and external and internal influencing factors.12 Cognitive domains include social cognition, executive function, memory, attention, and psychomotor speed. Cognition and metacognition are highly related and may overlap. An example of the difference is shown when performing an ocular biometry. Cognition is knowledge of the technique and measurement of axial length of 20 mm. Metacognition is necessary to review the result, determine if it is clinically consistent, and rescan the patient if necessary.
Metacognition is closely related to self-directed learning (SDL) and self-regulated learning (SRL). The 2 concepts are compared by Saks and Leijen.13 Self-directed learning is the process in which individuals take the initiative in diagnosing their learning needs, formulating learning goals, identifying human and material resources for learning, choosing and implementing appropriate learning strategies, and evaluating learning outcomes (applying Knowles' adult learning theory).14 Self-regulated learning is a younger concept arising from cognitive psychology and is an active process whereby learners set goals for their learning and attempt to monitor, regulate, and control their cognition, motivation and behavior, guided and constrained by their goals and contextual features on the environment.6,15-17 Although SDL is a fundamental skill for medical practitioners, there may be some learning disadvantages in it (for example, going on tangents due to interest, rather than engaging with curriculum),18 which demonstrates that metacognition can improve ability for SDL but is not automatic to the process.
Metacognition is an attribute necessary for lifelong learning,19 which can be defined as continuous training over the course of a professional career,20 or more broadly in the older literature as a “continuous, constant, unending, lifetime quest, which starts with birth and concludes only at death.”21
Metacognition is also related to self-reflection but can occur at an unconscious level when experienced physicians “act on intuition”. Metacognition is not always associated with good judgments; dissociation between metacognitive judgment-of-performance and actual performance is attributed to impaired self-evaluation. Positive affect (happiness) improves memory search but also results in overconfidence in judgments.22 Other external and internal factors affecting cognition and metacognition are shown in Figure 1.
The 3 domains of learning apply in CPD: cognitive,23,24 psychomotor (physical skills; eg, learning a new surgical technique),25 and affective (beliefs, behaviors, and attitudes).26 Metacognition is not synonymous with the affective domain but is considered the highest level of knowledge acquired using the domains of learning.23Table 1 lists the knowledge dimension with ophthalmology-specific examples.24
The relationship between neuroanatomy and metacognition is poorly understood. There may be different neuroanatomical correlations for the different metacognitive domains.27,28 Metacognitive ability can be improved by dopamine administered 100 mg orally29 and by noradrenaline blockade.30
METACOGNITION IN MEDICAL EDUCATION
Metacognition is important in the learning process in general and particularly in medical school. Excellent reviews include Medina et al12 and Quirk.31 Medical students need to learn the ability to make decisions with partial information and high risk of error and to become independent learners for the duration of their professional career because of expected continuing and rapid expansion in medical knowledge. Metacognitive abilities32 and SRL5 have been demonstrated to improve during medical school and be associated with academic performance, although this is controversial.33,34 Metacognition is important in the development of professionalism.35 Metacognitive skills are embedded in all aspects of CanMEDS (an educational framework for medical training designed by the Royal College of Physicians and Surgeons of Canada),36 particularly Scholar, which requires the individual to be receptive to acquiring, developing, and applying new skills throughout their entire career.37
METACOGNITION AND THE CPD LEARNER
Practicing ophthalmologists undertaking CPD are assumed to be older than medical students and trainees and to have learnt SDL and metacognition during medical school and residency, along with effective study techniques.
In contrast to cognitive skills, which decline at varying rates in the older population,38 metacognitive capabilities are thought to be preserved with age.39 Older ophthalmologists with changing cognitive abilities would be expected to use metacognitive abilities to improve goal-related memory outcomes during CPD activities and performance in practice.
Although medical students, residents, and fellows are assumed to have learnt SDL and metacognitive skills, there is evidence that this may not be the case.3,40,41 Practicing physicians have self-assessed metacognitive abilities.42 However, this self-assessment of metacognition may be inaccurate as medical students and doctors have been shown to be poor at self-assessment,43-46 leading to inflated or pessimistic self-assessments.47 Inaccurate self-assessment is thought to be due to weak metacognitive skills, hence improved metacognition improves the accuracy of self-assessments.48
Regulators of CPD expect CPD learners and providers to do so in a way that implicitly promotes metacognition. For example, metacognition is embedded in CanMEDS36 lifelong learning and the Accreditation Council for Continuing Medical Education criterion 31 states, “The provider creates individualized learning plans for learners.”49
PROMOTING METACOGNITION IN CPD
Awareness of and Instruction in Metacognition
This begins with building awareness that metacognition exists and differs from cognition. Methods to improve metacognition include formal instruction in metacognition,50 the use of metacognitive scaffolds including immediate feedback,51 deliberate consideration of alternatives, and “playback” (or cognitive apprenticeship)12 where one's actions are compared step by step with the optimal actions.52 Physical activity has been shown to improve metacognition in children,53 but there is not yet evidence for this in CPD learners.
Although ideal, it is possible that not all learners will benefit from metacognition instruction. Looking at dyscompetent (under-performing)54 medical practitioners, Hanna et al55 found improvement in only 1 of 5 physicians after a 3-year personalized intensive CME program. It is possible that these doctors lacked metacognitive skills necessary for improvement, despite intensive instruction.
Awareness and Mitigation of Cognitive Errors
Ophthalmologists need to recognize the use of heuristics (simplified thinking strategies used to speed assessment)56 in their clinical reasoning. Use of heuristics is very helpful in busy clinics but is accompanied by cognitive biases (cognitive dispositions to respond) that are thought to be a cause of clinical errors,57 although this is disputed.58 More recent work describes clinical reasoning using a dual processing model,59 with a rapid, automatic, intuitive component (type 1, probably using heuristics) and a slower, logical, and analytic component (type 2), with errors occurring in both components,60 but more often in type 1 decisions. Cognitive biases are summarized in Table 2.61,62
Proposed debiasing strategies to minimize cognitive errors are shown in Table 362,63; it needs to be noted that the effectiveness of these strategies is controversial.2,64 Underlying these strategies is critical self-reflection with a heightened sense of vigilance. Metacognition is one of the strategies; for example, Croskerry65 described using metacognition in universal (acquisition of knowledge), generic (understanding major classes of heuristics), and specific (awareness of specific scenarios in which errors are likely to occur) cognitive forcing strategies.
Hamm2 provides an alternative opinion that using metacognition to reduce error rates is more work than conventionally recognized, and less effective than initially hoped by its proponents, notably Croskerry.65 Instead he argues for better teaching at the level of clinical diagnosis, thus reducing the need for metacognition to reduce clinical errors.
Appropriate Needs Analysis
Needs analysis begins with self-assessment; sources of assessment include standardized tests, personalized surgical audit results, Office Record Reviews (templates for medical record reviews offered by the American Board of Ophthalmology), morbidity-mortality meetings, and interesting patient encounters. However, as noted above, medical practitioners are inaccurate in self-assessment of needs analysis. This was demonstrated in rheumatologists where a self-directed (curriculum-based) program based on perceived needs and interest was shown to be less effective than a personalized program based on responses to multiple-choice questions.66
Ophthalmologists need to be aware of this and seek additional information in preparing needs analysis for CPD activities. External, objective sources of information that may assist in identifying learning needs include patients,67 peers, CPD teachers, medical regulators and hospital administrators, national training bodies,68,69 and the community. Patients give feedback via complaints, patient satisfaction surveys, and as part of multisource feedback. Peers may give feedback during morbidity-mortality meetings, as part of multisource feedback, and informally based on patient outcomes. Teachers of CPD may assist via pre- and post-test questions and give feedback after encounters. Medical regulators and hospital administrators may receive patient complaints and are able to identify outlier ophthalmologists. National training bodies (eg, Royal Australian and New Zealand College of Ophthalmologists) may receive patient complaints, identify outliers, and conduct surveys of trainers. The community is involved through national health data, and in some jurisdictions, community relationships directly with medical schools and training bodies regarding community health priorities.70 It is likely that multiple sources of information will be most beneficial,71 but this has not been extensively studied.
Choosing Appropriate Activities
After identifying learning needs as accurately as possible, the ophthalmologist must select appropriate CPD study methods and activities. Important strategies include self-questioning (Table 4 lists sample self-questions to promote metacognition about learning during a range of typical CPD activities), reading for comprehension, and review of patient's perspective.
Reading for comprehension involves “active reading” including understanding the goals of reading, monitoring while reading (eg, skimming before reading, attending to headings, re-reading selected text), and summarizing learnings, which are all metacognitive skills.
Review of patient's perspective is a useful technique after a patient encounter to promote metacognition. The patient's beliefs, thoughts, and feelings about health and illness are elicited and compared with one's own. Useful questions that can be asked include the following: What concerns you most about your illness? What is your understanding of your medical problem? Do you have any religious or spiritual concerns about your problem?31 Understanding the patient perspective can prompt metacognitive review of the diagnosis and reduction in cognitive errors.
Learning New Surgical Techniques
There are no specific studies of metacognitive influences on learning new ophthalmology procedures, but it has been suggested that setting learning goals (in contrast to performance goals) is optimal for learning new surgical tasks because it engages metacognition.71 Instructors can prompt metacognition in learners by scaffolding (ie, providing a more supportive environment initially, which is gradually withdrawn as ophthalmologists move toward independence) and by providing rubrics for surgical techniques, such as those provided by the International Council of Ophthalmology.72
Recording Activities Using Portfolios
Portfolios have been used widely in medical schools73 and postgraduate medicine,74 although the evidence supporting their use is modest. Portfolios are collections of data related to learning over time; many different types are possible. Reflective portfolios are largely aimed at the development of reflective and metacognitive skills.75 Comprehensive portfolios are embedded in the curriculum and form part of the assessment program in some medical schools, but this broader type of portfolio is not envisaged in CPD.
The content of CPD reflective portfolios includes learning needs, practice goals and expectations, and metrics to demonstrate achievement, thus differing from simple lists of CPD meetings attended or results of self-assessment tests and resulting recommended remediation and learning activities. Portfolios may include metrics from any of the sources listed in Table 4. Portfolios are kept over a career, rather than a CPD cycle mandated by regulators (typically 1-5 years), and thus record repeated learning cycles over time. Portfolios that foster metacognitive development include the following features: a) personal statement of goals for learning that includes awareness of self and others including institutions and national priorities; b) strengths and weaknesses in cognitive, affective, and metacognitive performance including assessment of learning style; c) plans for addressing learning needs; d) self-questions with reflections, plans, and responses compiled during learning experiences; and e) self-evaluation of performance, with reference to self-generated and external sources of data.31
Portfolios may be recorded in paper or online format using personalized76 or templates provided. E-portfolios have been shown to be more successful, possibly due to increased interest in the process.
It should be noted that the use of portfolios is controversial. For CPD learners there is a risk of bureaucracy becoming overwhelming if portfolios are adopted by CPD regulators. A direct learning gain by users and effective mentoring is necessary for portfolios to be beneficial, and this may also limit their use in CPD.
THE ROLE OF THE CPD EDUCATOR
Educators in CPD play a role by modeling metacognitive behavior and teaching using methods designed to promote metacognition among learners. Techniques used by instructors that can increase metacognition among CPD learners12 include planning, with explicit description of learning objectives and pretesting to demonstrate learners' lack of knowledge; and test reviews, where the CPD educator reviews test results and corrects answers with CPD learners. “Thinking out loud” involves the learner and instructor both thinking out loud about a complex problem. This demonstrates to the learner their errors and the thought processes of an expert. “Reflection using the muddiest point” requires the learner to write down the most confusing part of the material to prompt SDL. “Judgments of understanding” asks CPD learners to prospectively make judgments about test scores. Over time judgments move from overconfident to underconfident, with reduction in bias. Immediate feedback involves reviewing performance with the learner shortly after the encounter. “Scaffolding” is useful in learning surgical techniques, where learners start with a more supportive environment and gradually move towards independence. “Guided reflective journaling” requires both the learner and instructor to undertake written review of selected clinical encounters. Teaching using clinical cases in a problem-based learning77 approach is also likely to promote metacognition. The “flipped classroom” is a promising method of instruction involving learners doing prereading and applying the new knowledge during classroom encounters, rather than being taught basic information without application in classroom encounters. Flipped classroom theoretically increases metacognition; however, evidence on its effectiveness is weak.78 Educators in CPD are likely to benefit from their teaching: teachers of clinical reasoning to medical students report greater use of metacognition in their own clinical practice.4
Areas for development in the field can be considered from the perspective of CPD participants, CPD educators delivering subject material, ophthalmology societies designing and managing CPD programs, and medical regulators. We recommend that metacognition be explicitly taught in medical school, ophthalmology training, and that the benefits and risks of using heuristics be explicitly taught to practicing ophthalmologists. We recommend CPD educators design activities to promote higher levels of cognition (eg, flipped classroom methodology) and medical educators design tools which are better able to measure metacognition in practicing ophthalmologists. Ophthalmology societies should encourage the use of personal learning plans, audit of results, and recording in portfolios. We recommend that research be directed towards the hypotheses that 1) improving metacognition in participants, CPD teachers, and CPD activities makes CPD more effective and time-efficient; and 2) improving metacognition will enhance learning of new surgical techniques. Finally, in the interests of best patient care, medical regulators should foster research into metacognition in CPD.
At the end of training, ophthalmologists are expected to be self-directed learners for the rest of their career. This requires metacognitive skills to plan, monitor, and evaluate their learning. The evidence based on metacognition in CPD is very weak, and definite conclusions are not possible. We suggest metacognition in CPD can be promoted by learning about and having explicit training in metacognition, learning about cognitive errors and strategies to avoid them, accurate learning plans informed by external objective information, selection of CPD activities and using metacognitive prompts, and learning new surgical techniques. Instructors of CPD play a pivotal role. Metacognitive skills are likely to vary among ophthalmologists, and unfortunately some may find it very difficult to improve their skills. Our recommendations for future research cover the spectrum of ophthalmologists, CPD educators, and designers and regulators of CPD programs.
1. Gonullu I, Artar M. Metacognition
in medical education. Educ Health (Abingdon)
2. Hamm RM. Figure and ground in physician misdiagnosis: metacognition
and diagnostic norms. Diagnosis (Berl)
3. Burman NJ, Boscardin CK, Van Schaik SM. Career-long learning: relationship between cognitive and metacognitive skills. Med Teach. 2014
4. Bartlett M, Gay SP, List PA, et al. Teaching and learning clinical reasoning: tutors' perceptions of change in their own clinical practice. Educ Prim Care
5. Cho KK, Marjadi B, Langendyk V, et al. Medical student changes in self-regulated learning during the transition to the clinical environment. BMC Med Educ
6. Berkhout JJ, Helmich E, Teunissen PW, et al. Context matters when striving to promote active and lifelong learning in medical education. Med Educ
7. Flavell JH. Metacognition
and cognitive monitoring: a new area of cognitive-developmental inquiry. Am Psychol
9. Schraw G, Moshman D. Metacognitive theories. Educ Psychol Rev
10. Borkowski JG, Carr M, Pressely M. ‘Spontaneous' strategy use: perspectives from metacognitive theory. Intelligence
11. Sun X, Zhu C, So SHW. Dysfunctional metacognition
across psychopathologies: a meta-analytic review. Eur Psychiatry
12. Medina MS, Castleberry AN, Persky AM. Strategies for improving learner metacognition
in health professional education. Am J Pharm Educ
13. Saks K, Leijen A. Distinguishing self-directed and self-regulated learning and measuring them in the e-learning context. Procedia Soc Behav Sci
14. Knowles MS, Holton EF III, Swanson RA. The Adult Learner. The Definitive Classic in Adult Education and Human Resource Development
. 7th ed. Amsterdam: Elsevier; 2011.
15. Pintrich PR. The role of goal orientation in self-regulated learning. In: Boekaerts M, Pintich PR, Zeidner M, eds. Handbook of Self-Regulation
. San Diego: Academic; 2005:451-502.
16. Sandars J, Cleary TJ. Self-regulation theory: applications to medical education: AMEE Guide No. 58. Med Teach
17. Panadero E. A review of self-regulated learning: six models and four directions for research. Front Psychol
18. Mayer RE. Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. Am Psychol
19. Mahajan R, Badyal DK, Gupta P, et al. Cultivating lifelong learning skills during graduate medical training. Indian Pediatr
20. Wojtczak A. Glossary of medical education terms: part 3. Med Teach
21. Cohen WJ. Lifelong learning - a definition and a challenge. Educ Leadersh
22. Sidi Y, Ackerman R, Erez A. Feeling happy and (over)confident: the role of positive affect in metacognitive processes. Cogn Emot
23. Krathwohl DR. A revision of Bloom's toxonomy: an overview. Theory Pract
24. Anderson LW, Krathwohl DR, eds. A Taxonomy for Learning, Teaching and Assessing. A Revision of Bloom's Taxonomy of Educational Objectives, Abridged Edition
. Boston: Addison Wesley Longman; 2001.
26. Krathwohl DR, Bloom BS, Masia BB, eds. Taxonomy of Educational Objectives. Handbook II: Affective Domain
. London: Longman; 1964.
27. Baird B, Cieslak M, Smallwood J, et al. Regional white matter variation associated with domain-specific metacognitive accuracy. J Cogn Neurosci
28. Hu X, Liu Z, Chen W, et al. Individual differences in the accuracy of judgments of learning are related to the gray matter volume and functional connectivity of the left mid-insula. Front Hum Neurosci
29. Joensson M, Thomsen KR, Andersen LM, et al. Making sense: dopamine activates conscious self-monitoring through medial prefrontal cortex. Hum Brain Mapp
30. Hauser TU, Allen M, Purg N, et al. Noradrenaline blockade specifically enhances metacognitive performance. Elife
31. Quirk ME. Intuition and Metacognition in Medical Education: Keys to Developing Expertise
. New York, NY: Springer; 2006.
32. Hong WH, Vadivelu J, Daniel EG, et al. Thinking about thinking: changes in first-year medical students' metacognition
and its relation to performance. Med Educ Online
33. Premkumar K, Pahwa P, Banerjee A, et al. Does medical training promote or deter self-directed learning? A longitudinal mixed-methods study. Acad Med
34. Harvey BJ, Rothman AI, Frecker RC. Effect of an undergraduate medical curriculum on students' self-directed learning. Acad Med
35. Gordon J. Fostering students' personal and professional development in medicine: a new framework for PPD. Med Educ
36. Frank JR, Snell L, Sherbino J, eds. CanMEDS 2015 Physician Competency Framework
. Ottawa, Canada: Royal College of Physicians and Surgeons of Canada; 2015.
38. Park HL, O'Connell JE, Thomson RG. A systematic review of cognitive decline in the general elderly population. Int J Geriatr Psychiatry
39. McGillivray S, Castel AD. Older and younger adults' strategic control of metacognitive monitoring: the role of consequences, task experience, and prior knowledge. Exp Aging Res
40. Nothnagle M, Anandarajah G, Goldman RE, et al. Struggling to be self-directed: residents' paradoxical beliefs about learning. Acad Med
41. Sockalingam S, Soklaridis S, Yufe S, et al. Incorporating lifelong learning from residency to practice: a qualitative study exploring psychiatry learners' needs and motivations. J Contin Educ Health Prof
42. Iannello P, Perucca V, Riva S, et al. What do physicians believe about the way decisions are made? A pilot study on metacognitive knowledge in the medical context. Eur J Psychol
43. Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med
44. Woolliscroft JO, TenHaken J, Smith J, et al. Medical students' clinical self-assessments: comparisons with external measures of performance and the students' self-assessments of overall performance and effort. Acad Med
45. Davis DA, Mazmanian PE, Fordis M, et al. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA
46. Colthart I, Bagnall G, Evans A, et al. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide No. 10. Med Teach
47. Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. J Pers Soc Psychol
48. Greer M. When intuition misfires. Monitor on Psychology
50. Safari Y, Meskini H. The effect of metacognitive instruction on problem solving skills in Iranian students of health sciences. Glob J Health Sci
51. El Saadawi GM, Azevedo R, Castine M, et al. Factors affecting feeling-of-knowing in a medical intelligent tutoring system: the role of immediate feedback as a metacognitive scaffold. Adv Health Sci Educ Theory Pract
52. Feyzi-Behnagh R, Azevedo R, Legowski E, et al. Metacognitive scaffolds improved self-judgments of accuracy in a medical intelligent tutoring system. Instr Sci
53. Álvarez-Bueno C, Pesce C, Cavero-Redondo I, et al. The effect of physical activity interventions on children's cognition and metacognition
: a systematic review and meta-analysis. J Am Acad Child Adolesc Psychiatry
54. Williams BW. The prevalence and special educational requirements of dyscompetent physicians. J Contin Educ Health Prof
55. Hanna E, Premi J, Turnbull J. Results of remedial continuing medical education
in dyscompetent physicians. Acad Med
56. Marewski JN, Gigerenzer G. Heuristic
decision making in medicine. Dialogues Clin Neurosci
57. Graber M, Gordon R, Franklin N. Reducing diagnostic errors in medicine: what's the goal? Acad Med
58. Norman GR, Eva KW. Diagnostic error and clinical reasoning. Med Educ
59. Marcum JA. An integrated model of clinical reasoning: dual-process theory of cognition and metacognition
. J Eval Clin Pract
60. Pelaccia T, Tardif J, Triby E, et al. An analysis of clinical reasoning through a recent and comprehensive approach: the dual-process theory. Med Educ Online
. 2011 Mar 14:16.
61. Croskerry P. Achieving quality in clinical decision making: cognitive strategies and detection of bias. Acad Emerg Med
62. Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med
63. Hussain A, Oestreicher J. Clinical decision-making: heuristics and cognitive biases for the ophthalmologist. Surv Ophthalmol
64. Graber ML, Kissam S, Payne VL, et al. Cognitive interventions to reduce diagnostic error: a narrative review. BMJ Qual Saf
65. Croskerry P. Cognitive forcing strategies in clinical decisionmaking. Ann Emerg Med
66. Mehta N, Geissel K, Rhodes E, et al. Comparative effectiveness in CME: evaluation of personalized and self-directed learning models. J Contin Educ Health Prof
. 2015;35 Suppl 1:S24-S26.
67. Keister DM, Hansen SE, Dostal J. Teaching resident self-assessment through triangulation of faculty and patient feedback. Teach Learn Med
68. Kim S, Dunkin BJ, Paige JT, et al. What is the future of training in surgery? Needs assessment of national stakeholders. Surgery
69. Kjaer NK, Vedsted M, Høpner J. A new comprehensive model for continuous professional development. Eur J Gen Pract
70. Ellaway RH, O'Gorman L, Strasser R, et al. A critical hybrid realist-outcomes systematic review of relationships between medical education programmes and communities: BEME Guide No. 35. Med Teach
71. Gardner AK, Jabbour IJ, Williams BH, et al. Different goals, different pathways: the role of metacognition
and task engagement in surgical skill acquisition. J Surg Educ
73. Buckley S, Coleman J, Davison I, et al. The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11. Med Teach
74. Tochel C, Haig A, Hesketh A, et al. The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No. 12. Med Teach
75. Sandars J. The use of reflection in medical education: AMEE Guide No. 44. Med Teach
76. Avila J, Sostmann K, Breckwoldt J, et al. Evaluation of the free, open source software WordPress as electronic portfolio system in undergraduate medical education. BMC Med Educ
77. Wood DF. Problem based learning. BMJ
78. Chen F, Lui AM, Martinelli SM. A systematic review of the effectiveness of flipped classrooms in medical education. Med Educ