Secondary Logo

Journal Logo

Developing Skills of Unsupervised Practice

The Benefits of Tying Yourself in Knots: Unraveling the Learning Mechanisms of Guided Discovery Learning in an Open Surgical Skills Course

Aagesen, Andreas H. MBBS; Jensen, Rune D. MSc, PhD; Cheung, Jeffrey J.H. MSc, PhD; Christensen, John B. MD; Konge, Lars MD, PhD; Brydges, Ryan MSc, PhD; Thinggaard, Ebbe MD, PhD; Kulasegaram, Kulamakan M. PhD

Author Information
doi: 10.1097/ACM.0000000000003646

Abstract

Optimizing procedural skills training is an ongoing area of investigation in medical education,1,2 with much evidence suggesting that training should include some degree of learner self-regulation blended with judicious instruction and guidance.3 Instead of viewing learner vs instructor control as polarities, a combined approach has been strongly suggested as efficacious in the general education and medical education literature. However, how this blending and balance is accomplished has become the subject of continued research and debate.4–8

One potential approach for blending direct instruction and learner self-regulation is through guided discovery learning (GDL), which reverses the sequencing of traditional instruction. That is, instead of didactic training followed by practice examples or problem-solving attempts, the order is reversed with a discovery phase, consisting of learners engaging in problem solving without instruction, followed by an instructor-guided phase. Typical studies of GDL compare the discovery followed by instruction sequence with the traditional instruction followed by practice sequence. GDL has been shown to be an effective method of facilitating various learning outcomes in higher learning contexts such as mathematics, science, and other concept learning domains.5–7,9,10 Evidence has accumulated to suggest GDL facilitates an experience of productive failure (also known as productive struggle), defined as a meaningful experience of uncertainty and challenge which elicits a number of cognitive processes: activating prior knowledge, elaborating the deep structure of concepts, focusing attention on knowledge deficits, and preparing queries of and attention to the instructor.3,8,11–14

Though GDL has been shown to represent an efficient way of stimulating productive failure, most studies have been in the realm of “concept learning” and in the context of K-12 education.5,6,9,15 For psychomotor skills, the evidence is less clear. Discovery learning has long been explored as a training strategy for simple movements (e.g., goal directed aiming).16 However, whether GDL can be generalized to the training of complex, multistep procedures, such as those in surgery, has yet to be comprehensively studied. One small clarification experiment testing GDL vs instruction followed by practice for the acquisition and transfer of suturing skills demonstrated moderate effects in favor of GDL.4 While those results were promising, they did not generate insights on the mechanisms of how GDL might work to enhance procedural skills learning.

In the present study, we aimed to replicate the benefits of GDL in the context of medical learners engaged in procedural skills training and also expand our understanding of the mechanisms by which GDL impacts learning. To achieve these goals, we conducted a mixed-methods cohort study of surgical learners attending a 6-week open surgical skills course. We compared the final summative skills and written assessments of an intervention cohort (which followed a GDL pedagogical approach) with historical controls who had previously attended a traditional version of the course (didactic instruction before practice attempts). We observed and interviewed learners in the GDL group during the course to yield data on learning mechanisms. Overall, we aimed both to justify GDL’s utility, as well as to clarify how GDL impacts learning of procedural skills.

Method

For the past 2 years (2017–2019), Copenhagen Academy for Medical Education and Simulation (CAMES) has implemented a course in basic open surgical techniques.17 The course is aimed at junior surgeons and accepts 8 participants per course. Participation in the course is voluntary. The course consists of 6 one-and-a-half hour sessions delivered in 6 weeks. Before the first session, each participant receives an at-home training kit and a logbook with detailed descriptions of the different procedures to guide practice at home. The first 2 sessions focus on tying surgical hand-knots. The next 4 sessions include a variety of suturing techniques ranging from cutaneous running sutures to intracutaneous and facial sutures. Basic theory concerning choice of suture, material, and procedure is taught throughout the course.

Participants

Participants were postgraduate physician trainees from hospitals in the capital region of Denmark who were planning to learn to assist before entering a surgical department or had been employed in surgical departments but lacked the basic surgical skills. Our only exclusion criterion was if participants had already acquired their surgical specialist status. To our knowledge, there were no differences in the selection or preparation of participants in the GDL cohort compared with the control.

Study design

Our study followed a concurrent triangulation mixed-methods design in which we compared the numeric performance-based data of an intervention cohort with a historical cohort and used directed thematic content analysis of direct observations and semistructured interviews for the intervention cohort only.18 Our quasi-experimental data were intended to establish efficacy evidence, while our interview and observational data were intended to elaborate previously hypothesized and emerging mechanisms. We also used insights gained during the qualitative data analysis to inform further exploratory quantitative analysis.

For the intervention cohort, participants were taught using the GDL learning approach. Each training session began with a discovery phase, where participants spent 10 minutes exploring the session-specific skill, including accessing relevant surgical instruments and sutures, without the aid of an instructor. Typically, they were provided with an example of the finished product as a guide as in a previous study.4 Participants used the materials to try and replicate the finished suture- or hand-knot. After the initial discovery phase, participants received didactic instruction on how to perform the skill and then were given approximately 1 hour to practice with teacher feedback. In sessions that included more than one skill, participants were shown an example of what was deemed the most important skill of the day’s session. By contrast, the historical cohort was taught using the direct instruction approach, in which each session started with participants being instructed by the teacher followed by practice. The length of each session was the same for both the GDL and control cohorts. After the course, both cohorts of participants completed a summative procedural skills assessment consisting of completing 2 hand-knot and 5 suturing skills (see List 1), as well as a short, written exam on relevant conceptual knowledge.

Cohort and learning environment differences

GDL data were collected from 2 groups of 8 participants (N = 16) taking the course between October and December 2018. The historical control cohort consisted of 7 groups of 8 participants, with a dropout of 5 participants (N = 51), who took the course over the period February to September 2018. Demographic information for the GDL cohort is provided in Table 1. The instructor for both cohorts was the same: a retired surgeon who had taught throughout his career and in this course for 2 years before our study.

Table 1
Table 1:
Demographics of Participants in 2 GDL Groups

Outcomes and data collection

The outcomes for the quantitative comparison of group differences in this study were (1) a procedural skills test where participants had to perform the 7 skills taught in the course on a skill pad (see List 1) and (2) performance on the conceptual knowledge test. For the procedural skills test, each skill was scored using a mixture of Global Rating Scale (GRS) and checklist based on the Objective Structured Assessment of Technical Skills (OSATS)19 previously used in a research study on this course.17 The combined score of the GRS and checklist was summed across all 7 skills and converted to a percentage, and taken as the final outcome. The conceptual knowledge test consisted of 6 short-answer questions relating to the choice of suturing technique and equipment in different hypothetical scenarios; it was scored using GRS with the total score converted to a percentage. Both outcomes were used in previous research studies and were consistent across the intervention and control cohorts. Additionally, we conducted post hoc exploratory statistical analyses for all hand-knot vs suturing skills, as our qualitative observations indicated potential variation in levels of engagement with these 2 skill types during the discovery phase. All quantitative data were collected by J.B.C. but scored by a single experienced blinded rater. We did not include a second rater, as the course traditionally uses a single rater and resource limitations prevented expanding the rater pool.

All qualitative data were collected throughout the GDL cohort sessions by the lead author (A.H.A.) who was supervised by a qualitative health researcher (R.D.J.). A.H.A. is a medical student uninvolved in formal evaluations of the participants, which allowed him to participate and observe participants throughout the course without posing a threat.20 Before the data collection started, a semistructured interview and observational guide were developed using theories of self-regulated learning21 and transfer of learning.22 A.H.A. collected the observational data throughout by noting predetermined areas of focus such as trainees’ behavior with task, instructor, and other trainees, as well as any other interactions.

Interviews were conducted during the course period and until no new data of relevance to the research question were generated by the newest interview transcript. Participants were interviewed for approximately 30 minutes after or before each training session. A total of 13 interviews with 10 participants were conducted. To gain a more complete picture of learning experiences across the different skill types (hand-knots vs sutures), 3 participants were interviewed twice. Having one researcher (A.H.A.) both collecting observational data and conducting interviews allowed for a rich immersion with the course, data, and participants, further enabling him to iteratively refine data collection process for interview questions and observational field notes made during the teaching session.23

Analysis

A one-way ANOVA was used to compare procedural skills performance and conceptual knowledge test scores between the GDL and historical cohorts. These analyses were performed using SPSS 25 (IBM Corporation, Armonk, New York) with an alpha of 0.05. Cohen’s d effect sizes were computed for all differences.

For the qualitative data analysis, we triangulated our observational data with our semistructured interviews. Throughout data collection, the team iteratively discussed observation and interview data to direct subsequent data collection. This also informed the subsequent exploratory post hoc statistical analyses comparing procedural skills performances by skill type (hand-knots vs sutures). If a recurrent theme was identified in the interviews or observations and deemed important, it would be incorporated in the subsequent data collection. The ongoing thematic analysis was both inductive and deductive; the process was guided by previously described learning mechanisms established in the productive failure literature8,13 (see Table 2). After data collection was completed, audio to script transcriptions for the interviews were done using InqScribe (V.2.2.4, Inquirium, LLC, Chicago, Illinois) and subsequently anonymized. Two researchers (A.H.A. and R.D.J.) read through and analyzed a subset of 3 transcripts using directed thematic content analysis. Because there was a strong agreement on which passages to code and on the themes applied, the remaining data were analyzed by one researcher (A.H.A.).24 To ensure methodological quality and rigor, the diversity of the data and the research group was taken into account during analysis and when producing the manuscript.25 After immediate data analysis was completed, researchers A.H.A. and R.D.J. completed the analysis through constant comparison of themes generated from observational and interview data. A condensation of the analysis was done to encapsulate the most representative themes. After the analysis was finalized, quotes were chosen and translated from Danish into English. Examples of analytic procedure are provided in Supplemental Digital Appendices 1 and 2 at https://links.lww.com/ACADMED/B2. This study was submitted to regional ethics committee, which deemed that no approval was needed (VD-2018-356).

Table 2
Table 2:
Interview Quotes Attesting to Knowledge Structures Identified by Previous Research

Results

We collected data from 16 participants in the GDL course and accessed data from 51 participants from the historical cohort. Most participants in the GDL cohort were just starting their surgical careers—that is, not yet entering or newly employed at a surgical department (see Table 1). One participant did not provide demographic data.

Performance-based data

The one-way ANOVA showed that the GDL cohort was superior to the traditional cohort on the written conceptual knowledge test (F [1,65] = 4.96, P = .03, d = .62, mean [standard deviation]; GDL: 80.10% [8.03%] vs historical: 72.08% [13.60%]). GDL was superior on the total score of procedural skills assessment (F [1,65] = 6.23, P = .02, d = .68, mean [SD]; GDL: 69.64% [6.2%] vs historical: 64.50% [9.30%]).

In concurrent triangulation mixed-methods designs, qualitative data can inform quantitative analysis. Based on our observations and subsequent interviews, we hypothesized that GDL would be less efficacious for hand-knots as participants reported having less experience and feeling difficulty in engaging during the discovery phase. Accordingly, we conducted a secondary exploratory analysis of performance on the 2 hand-knots vs suturing task tests. We found the effect of GDL training to be smaller for hand-knots compared with historical controls (d = .15; 73% vs 70%); by contrast, the same analysis for sutures yielded a larger effect of GDL training (d = .60; 80% vs 74%).

Interview and observational data

While our performance data highlight the efficacy of GDL, our qualitative data provide insights into the mechanisms and contextual impacts of our GDL intervention. Our deductive analyses identified many potential mechanisms related to building conceptual knowledge (Theme 1) that have been previously described in the literature. We observed learners engaging in, as well as reflecting on, activities elicited by the GDL intervention that are concordant with the mechanisms hypothesized by previous research.7,8,14,26,27 We provide a summary of these in Table 2, as subthemes 1–5.

Our inductive analyses identified additional effects of GDL such as motivating self-regulated learning (Theme 2). These mechanisms were mediated by contextual elements including the type of skill and psychological safety (Theme 3). We summarize the relationship between these themes in Figure 1. The main themes with examples from collected data are discussed below.

Figure 1
Figure 1:
Relationship between intervention, potential mechanisms, and observed effects.

Theme 1: Building conceptual knowledge

We observed mechanisms such as creating solutions and comparing with the canonical solution as shown in field note 1 (see Table 3). Further, participants spoke about how generating their own solutions helped identify new areas of focus (Table 2, quote 4) and also enabled them to better understand the “deep structure” of the suturing task they were required to learn (Table 2, quote 5). Additionally, learners compared their previous knowledge, practice, and discovery of a procedure during the start of the lesson with that of the instructor’s presentation of a given skill (Table 2, quote 2). Furthermore, observations attested to how reflection on their own learning helped the learners to identify pivotal elements, ask questions, and focus on the structure of the suturing skill (Table 3, field note 2).

Table 3
Table 3:
Field Notes Taken During Direct Observation of Course Participants Throughout the Course

Theme 2: Motivating self-regulated learning

GDL participants expressed an increase in motivation as a result of the discovery phase (Table 2, quote 3). This motivation was expressed in many ways including increased self-regulated learning at home and between classroom sessions in anticipation of future skills learning. For example, some participants reviewed the skill before the session in an effort to ensure their 10 minutes of discovery time in the classroom was maximally beneficial. This may have prepared learners to engage with the mechanisms identified in Theme 1.

In my case, it’s maybe that I’m preparing myself for the lesson. That way I didn’t have to feel stupid those first 10 minutes, because I’d read up on my homework. (A1)

Participants also expressed changing attitudes to how their learning activities aligned with clinical practice:

… I actually think that it’s quite healthy in surgery, or as a doctor in general, that sometimes you need to do things, you haven’t seen done before, or you don’t know exactly how to perform. And then you have to give it a go and try […]. And that’s probably very healthy. Maybe if you had a little more of this as a medical student, then you wouldn’t be so shocked, when you faced reality, and realize there are plenty of things, you have to figure out to do yourself. (B1)

Most participants reported these positive changes in learning behavior and attitude across most of the GDL sessions. Still, a minority of participants did not appear engaged during specific sessions in the discovery phase of the GDL intervention depending on the skill or task, as evidenced by field note 3 (Table 3). These participants primarily had previous specific surgical experience with a skill and indicated that the discovery phase was not sufficiently challenging. Participants who expressed these views were likely to use the GDL phase to do other types of learning, such as practicing other surgical techniques, though they still reported that this was a productive use of their time.

Theme 3: Type of skill and psychological safety

The mechanisms identified in Themes 1 and 2 were moderated by the context of the learning environment. We observed and heard from participants that the type of skill, their previous experience, and the “accessibility” of the task dictated how they engaged with the discovery phase. The most visible example was the difference in participants’ experience with hand-knots and sutures. The majority of participants identified a lack of experience with hand-knots compared with sutures which subsequently affected their motivation to practice (Theme 2). This finding prompted our subsequent exploratory quantitative analyses of these differences described earlier.

I think it made more sense, when we started learning sutures. […] if it’s because my brain just doesn’t do hand-knots, but it just made sense, when you had a finished example next to you, and thought I’ve got to attempt that. […] then you were like; how do we start, how many suture-knots, and so forth. (B6, second interview)

Hence, when participants had little prior knowledge of a skill, they reported less benefit from the discovery phase of training. This may have been due to an increased sense of struggle that prevented them from actively engaging with the task (see Table 3, field note 4).

Another emergent element of the course context prominent in our analysis related to participants’ perceived psychological safety during the course. In the observations, some participants generally experienced difficulties engaging in productive failure due to the social context, expressing discomfort with failing in front of others:

The whole idea of sitting and maybe doing something that might be wrong in front of others, that kind of thing. I think it’s something people find difficult … to start off with at least. (A4)

Over time, however, some participants described an improvement in the psychological safety and comfort with GDL. Most often, this was related to their growing competence, as well as their increasing experience with the educational approach.

I think the educational structure frustrated me a lot more the first couple of times, because […] you don’t know a whole lot about surgery. […] you felt you were completely out of your comfort zone […]. Whereas, at the end of the 6 weeks, when you’re presented a new technique, you now have an impression of surgical knot-techniques in general, and then it’s easier to deduct or discover how the technique is performed. And then actually attempt it. (B2)

The increasing psychological safety was observed and perceived from participants’ willingness to engage their task during the first 10 minutes, as described under Theme 2 (see Table 3, field note 5).

Discussion

We tested and explored how surgical trainees exposed to a 6-week open surgical course using GDL principles performed compared with a historical cohort taught under the traditional sequence (i.e., lecture followed by practice). Our assessment data replicated previous studies reporting on the benefits of GDL. Our qualitative data establish potential mechanisms for GDL’s efficacy in our unique longitudinal simulation-based course context. The observed activities and the perceptions of our participants reflected on the knowledge structures identified by Kapur and others.8,26,27 We provide supportive evidence for previously identified mechanisms (e.g., activating prior knowledge), as well as evidence for hypothesized mechanisms such as the promotion of self-regulated learning outside of formal training.14 We also identified potential contextual elements, including participants’ perception of psychological safety that appeared to moderate the effects of GDL. Thus, our study adds evidence that supports the efficacy and underlying mechanisms for why GDL can be a valuable approach to instructional design.2,8

One potentially novel mechanism of GDL that we observed involved learners reporting increased motivation to prepare for future training sessions and to review previous material before engaging in the GDL intervention. This motivation for self-regulated learning may have had a synergistic effect with other mechanisms that helped learners build conceptual knowledge, as learners arrived with some conceptual knowledge that could be activated during the productive failure elicited by the discovery phase. If this finding can be replicated in other settings, it would represent a key goal of all preparatory training in health care, like simulation-based training, which is to design initial training such that it enhances learners’ motivation to engage in future learning.21,28,29

Our results speak to the importance of considering the prior experience of learners as well as the nature of the learning task. We observed and heard from participants that the practice of hand-knots could be daunting during GDL. An image of a completed hand-knot offers no guidance on how to initiate and little to no intrinsic feedback on how to complete the knot. This task may be too difficult, and previous research suggests that these types of tasks would fail to facilitate the productive struggle that occurs when GDL is successful.26 Other researchers have also noted that for GDL to be beneficial, learners must have some prior knowledge to activate during the discovery phase.7 Therefore, both researchers and teachers must carefully choose which tasks are suitable for discovery learning by considering alignment with the prior experience of learners.

We also found that psychological safety was a key theme for our participants, who pointed to the learning environment of the specific course and surgical training more broadly as driving both engagement with GDL as well as discomfort with “failing” in front of others. Similar findings have been shown in studies of the operating room as a learning environment, which aligns with some participants likening GDL to their clinical learning environments.30 While continued experience with GDL attenuated their negative experiences, psychological safety remained a key concern for participants. We are not aware of psychological safety and the culture of the learning context receiving attention in the GDL literature, though health professions simulation has long recognized its importance.31–33

Future implications

Our results confirm that GDL can be efficacious for procedural skill training. Allowing some degree of learner self-regulation balanced by instructor guidance and feedback appears to be a feasible and impactful choice for simulation-based procedural skills training. By eliciting productive failure, GDL seems to help learners elaborate and build more conceptual knowledge compared with traditional teaching, which previous work has shown to be related to improved procedural skill transfer and retention.34,35 However, our results show there is no “one-size-fits-all” approach. The type of skill, learner’s previous knowledge and experience, and psychological safety will influence the efficacy of GDL. Educators should attend to how these mechanisms play out in their own setting by looking for evidence of the behaviors and mechanisms identified in this study.

Whilst our study managed to provide supportive evidence for previous mechanisms, future studies should probe the exact influence that motivation for self-regulated learning has on the efficacy of GDL. Furthermore, clarifying to what extent psychological safety moderates the “productive” portion of struggling during GDL is necessary. Building on previous research on the role of psychological safety in simulation may be helpful in this regard.

As with all nonrandomized cohort studies, we cannot rule out the possibility of confounders. Our qualitative data in relation to Theme 1 and Theme 2 are limited in that learners discussed their perceptions of learning, activities, and changes in their mastery of skills; we cannot directly infer changes to conceptual knowledge. Moreover, we cannot weigh the relative importance of the mechanisms identified in our data; further clarification research is warranted to understand the specific contributions of the numerous mechanisms identified. We did not confirm whether participants engaged in self-regulated learning outside of the classroom, though we have no reason to suspect they falsely reported doing so.

Conclusions

We found that GDL can be an efficacious approach to teaching procedural skills. By eliciting productive failure, it appears to prompt participants to engage in strategies that may help elaborate conceptual knowledge for the skill and promote motivation to engage in further self-regulated learning. However, educators implementing GDL should be mindful of the context of the skill, the learning environment, and the perceived psychological safety of the participants.

List 1

Skills Taught and Assessed

Hand-knots

  • One handed
  • Two handed

Suture techniques

  • Far–far–near–near
  • Fascial
  • Subcutaneous
  • Inverted mattress
  • Intracutaneous

Acknowledgments:

The authors wish to thank Dr. Maria Mylopoulos for her critical comments on the paper, the Copenhagen Academy for Medical Education and Simulation (CAMES), and the Capital Region for their surgical training program.

References

1. Clark RE, Kirschner PA, Sweller J. Putting students on the path to learning: The case for fully guided instruction. Am Educ. 2012; 36:6–11
2. Mayer RE. Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. Am Psychol. 2004; 59:14–19
3. Loibl K, Roll I, Rummel N. Towards a theory of when and how problem solving followed by instruction supports learning [published online ahead of print July 2016]. Educ Psychol Rev. doi:10.1007/s10648-016-9379-x
4. Kulasegaram K, Axelrod D, Ringsted C, Brydges R. Do one then see one: Sequencing discovery learning and direct instruction for simulation-based technical skills training. Acad Med. 2018; 93(11 suppl):S37–S44
5. Kapur M. Productive failure in learning math. Cogn Sci. 2014; 38:1008–1022
6. Kapur M. Productive failure in learning the concept of variance. Instr Sci. 2012; 40:651–672
7. Schwartz DL, Martin T. Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cogn Instr Hatano Ina Bransford Schwartz. 2004; 22:129–184
8. Kapur M. Examining productive failure, productive success, unproductive failure, and unproductive success in learning. Educ Psychol. 2016; 51:289–299
9. DeCaro MS, Rittle-Johnson B. Exploring mathematics problems prepares children to learn from instruction. J Exp Child Psychol. 2012; 113:552–568
10. Likourezos V, Kalyuga S. Instruction-first and problem-solving-first approaches: Alternative pathways to learning complex tasks. Instr Sci. 2017; 45:195–219
11. Lee HS, Anderson J. Student learning: What has instruction got to do with it?. Annu Rev Psychol. 2013; 64:445–469
12. Bransford JD, Schwartz DL. Chapter 3: Rethinking transfer: A simple proposal with multiple implications. Rev Res Educ. 1999; 24:61–100
13. Schwartz DL, Bransford JD. A time for telling. Cogn Instr. 1998; 16:475–5223
14. Belenky DM, Nokes-Malach TJ. Motivation and transfer: The role of mastery-approach goals in preparation for future learning. J Learn Sci. 2012; 21:399–432
15. Loehr AM, Fyfe ER, Rittle-Johnson B. Mathematics problem solving: A classroom study. J Probl Solving. 2014; 7:36–50
16. Wulf G, Weigelt C. Instructions about physical principles in learning a complex motor skill: To tell or not to tell. Res Q Exerc Sport. 1997; 68:362–367
17. Christensen JB, Nodin E, Zetner DB, Fabrin A, Thingaard E. Basic open surgical training course. Dan Med J. 2018; 65:A5519
18. Creswell J, Clark V, Gutmann M, Hanson W. Advance mixed methods research designs. Tashakkori A, Teddlie C, eds. In: Handbook of Mixed Methods in Social and Behavioral Research. Thousand Oaks, CA: SAGE Publications. 2003209–240
19. Hatala R, Cook DA, Brydges R, Hawkins R. Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): A systematic review of validity evidence. Adv Health Sci Educ Theory Pract. 2015; 20:1149–1175
20. Pope C. Conducting ethnography in medical settings. Med Educ. 2005; 39:1180–1187
21. Brydges R, Manzone J, Shanks D, et al. Self-regulated learning in simulation-based training: A systematic review and meta-analysis. Med Educ. 2015; 49:368–378
22. Kulasegaram KM, McConnell M. When I say … transfer-appropriate processing. Med Educ. 2016; 50:509–510
23. Pink S, Morgan J. Short-term ethnography: Intense routes to knowing. Symb Interact. 2013; 36:351–361
24. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006; 15:77–101
25. Tracy SJ. Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qual Inq. 2010; 16:837–851
26. Ashman G, Kalyuga S, Sweller J. Problem-solving or explicit instruction: Which should go first when element interactivity is high? [published online ahead of print October 2019]. Educ Psychol Rev. doi:10.1007/s10648-019-09500-5
27. Schwartz DL, Chase C, Chin DB, et al. Interactive metacognition: Monitoring and regulating a teachable agent. Hacker DJ, Dunlosky J, Graesser AC, eds. In: Handbook of Metacognition in Education. The Educational Psychology Series. New York, NY: Routledge/Taylor & Francis Group. 2009340–358
28. Hmelo CE, Lin X. Becoming self-directed learners: Strategy development in problem-based learning. Probl Learn A Res Perspect Learn Interact. 2000227–250
29. Brydges R, Dubrowski A, Regehr G. A new concept of unsupervised learning: Directed self-guided learning in the health professions. Acad Med. 2010; 85(10 suppl):S49–S55
30. Swendiman RA, Edmondson AC, Mahmoud NN. Burnout in surgery viewed through the lens of psychological safety. Ann Surg. 2019; 269:234–235
31. Turner S, Harder N. Psychological safe environment: A concept analysis. Clin Simul Nurs. 2018; 18:47–55
32. Rudolph JW, Raemer DB, Simon R. Establishing a safe container for learning in simulation: The role of the presimulation briefing. Simul Healthc. 2014; 9:339–349
33. Kolbe M, Eppich W, Rudolph J, et al. Managing psychological safety in debriefings: A dynamic balancing act [published online ahead of print August 2019]. BMJ Simul Technol Enhanc Learn. doi:10.1136/bmjstel-2019-000470
34. Cheung JJH, Kulasegaram KM, Woods NN, Brydges R. Why content and cognition matter: Integrating conceptual knowledge to support simulation-based procedural skills transfer. J Gen Intern Med. 2019; 34:969–977
35. Cheung JJH, Kulasegaram KM, Woods NN, Moulton CA, Ringsted CV, Brydges R. Knowing how and knowing why: Testing the effect of instruction designed for cognitive integration on procedural skills transfer. Adv Health Sci Educ Theory Pract. 2018; 23:61–74
36. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007; 19:349–357
    37. American Anthropological Association. Code of Ethics of the American Anthropological Association. Am Sci. 2009;(February):1–8

      References cited only in Table 2

      38. Slamecka NJ, Graf P. The generation effect: Delineation of a phenomenon. J Exp Psychol Hum Learn Mem. 1978; 4:592–604
      39. Hirshman E, Bjork RA. The generation effect: Support for a two-factor theory. J Exp Psychol Learn Mem Cogn. 1988; 14:484–494
      40. Reber Arthur S. Implicit learning and tacit knowlegde. J Exp Psychol Gen. 1989; 118:219–235

        Supplemental Digital Content

        Copyright © 2020 by the Association of American Medical Colleges