The observable changes in the attainable performance, as well as the associated changes in the structure of the representations, as illustrated, respectively, in Figures 1 and 2, need to be described for each domain of expert performance. The detailed cognitive processes and practice activities will, of course, differ across domains of expertise (e.g., chess, music, sports, teaching, surgery), but the general principle remains: Performers aspiring to expert levels must engage in training activities that are not only designed to improve particular aspects of performance but are also integrated with all the other aspects of performance. When aspiring experts have improved and mastered one aspect of performance, they must then direct their attention to other improvable aspects.
My colleagues and I proposed a framework for explaining or illustrating the development of expert performance in music.20 Consistent with the findings of Bloom,22 our framework includes the start of playing an instrument. We explicitly identify the initial playing of an instrument because playing an instrument is nearly always linked to, at least in the beginning, instruction by a parent or professional teacher.20 Producing enjoyable music without any prior training is difficult, whereas even a child can learn to kick a ball by trial and error without any instruction. The duration of focused practice for children learning an instrument is recommended to be relatively short, around 10 to 20 minutes per day, allowing the child to engage in play outside of practice. During the beginning phases of regular practice, parents help the children to detect errors and make corrections and, thus, improve their performance. Eventually the aspiring musicians will acquire their own mental representations that permit them to hear the sounds of their own playing and detect any problems by themselves.2 Initially, however, the majority of young musicians lack the ability to hear the sounds of their own music. Without an instructor to help them identify and correct problems, beginners end up just playing the same mistakes over and over, as shown by studies analyzing videotaped practice sessions.23 Additionally, the ability to listen to the music that one produces is critical for the motivation to keep striving to improve. Music students who can hear or image representations of how the music is supposed to sound often enjoy hearing themselves play their favorite pieces, and they experience joy when producing new sounds associated with increasingly complex music pieces.
To summarize, deliberate practice20 (see middle part of Figure 1) occurs when advanced students (such as those at an internationally acclaimed music school) follow their teachers’ recommendations for practice activities (training tasks)—assuming, that is, that the students practice with full concentration toward the current practice objectives (training goals); that the students receive or self-generate immediate feedback; and that the training tasks offer opportunities to make repetitions with gradual improvements (structure of practice).
Several articles2,24–28 have described the particular forms of teacher-guided practice (deliberate practice) in domains other than music. Importantly, an excellent teacher in any domain helps his or her students develop their own mental representations such that the students can eventually take on most of the teacher roles, evaluating their own performance and even, eventually, designing their own practice goals and being able to increasingly image, monitor, and refine their own performance.
Expert performers will continue for decades in their professional domains after they have completed their training and education. Less research covers this phase, when performers transition to other roles (teacher, coach, manager, judge) within the domain of expertise. In the domain of music, it is relatively common for older musicians to perform professionally in public, which allows the audience and music critics to directly compare their performances with those of their students and/or younger musicians. When my colleague and I29 compared expert pianists ranging in age from 50 to 70 years old with young expert pianists in their 20s, we found two interesting results. Using laboratory tasks designed for research on aging (e.g., speeded choices, speeded substitution of digits for letters), the older experts performed much worse than the young experts and matched the performance of a group of amateur pianists matched for age.29 Most interestingly, when we tested all young and old pianists on musically relevant tasks, we again observed a reduction in performance among the older expert pianists; we found that, for some older pianists, this decrement was associated with stopped or at least reduced engagement in weekly practice. The age differences were no longer significant when we controlled (statistically) for the amount of maintained practice (weekly hours of practice alone).29
More generally, a review of performance shows that skill in activities, such as typing and flying airplanes, decays as a function of the length of time since the cessation of practice.30 Evidence from one recent study shows that taking a break from solving crosswords for a year or more reduces a puzzler’s performance in crossword competitions.31 Also, a large body of evidence from sports shows that reductions in the intensity of practice, as well as complete cessation of training, lead to decreased physiological adaptations and decrements in sports performance.32 The high level of performance attained by master athletes over the age of 60 and even 70 who, importantly, engage in high levels of practice weekly is remarkable.33 Collectively, these findings imply that continued practice during one’s life is very important for maintaining a high level of performance, and that the age-related declines in expert performance are mediated by reduced engagement in practice.34
Applying the Expert-Performance Approach and Deliberate Practice to Medicine
How to successfully transfer the theoretical framework of expert performance with deliberate practice from music to medicine is not obvious. Children are often introduced to the domain of music with the idea that they might become expert musicians. A curriculum for each instrument, designed to gradually improve students’ skills, governs the first 10 to 15 years of instruction. From the beginning, music students are encouraged to perform in front of family and friends; at higher levels of skill, they perform more complex pieces in front of audiences. Opportunities for feedback from not only teachers, but also other musicians who listen to public performances, abound. In stark contrast, in the United States and Canada, future medical doctors typically spend the first 13 years of learning in general K–12 education, and then they spend another 4 years acquiring a bachelor’s degree, often studying natural science. Traditionally, full clinical training does not begin until the third year of medical school with clerkship, and only at this point do students begin to make decisions about specializing (e.g., in surgery, psychiatry, radiology, or pediatrics). This clinical training and specialization corresponds to the beginning of acquiring specialized skills in diagnosis and the execution of specific medical procedures.
Training in medicine has traditionally focused on acquiring, first, theoretical knowledge and, then, actual experience in real-world situations, where performance has consequences for patients. In the domain of medicine, the traditional focus on knowledge and neglect of gradual skill acquisition through deliberate practice is exemplified by the well-known saying about learning and performing medical procedures, “See one, do one, teach one.” This adage implies virtually instantaneous mastery of new procedures among medically trained individuals; however, objective performance measures gathered when new laparoscopic techniques became available in the mid-1980s have invalidated this conventional wisdom.35 In spite of extensive training in classical surgery, the learning curves of the experienced surgeons were typical: the reduction of significant errors in laparoscopic surgery was a function of the number of completed procedures.36
Just as simply watching a procedure is ineffective, so too are continuing medical education (CME) lectures and accumulated hours of professional experience. A review showed that attending CME lectures did not effect any meaningful changes in participating doctors’ actual practice.37 Similarly, other research38,39 has not shown sustained benefits of longer professional experience by health care professionals after completing supervised training. Beyond some gains from the initial experience during the first years of independent practice, benefits for improved judgment from additional professional experience are very limited.40
There are several reasons that additional professional experience does not seem to improve performance. Availability of particular types of experience in real-world settings is typically not under the control of the learner; days, weeks, or months may pass between encounters of a particular type of patient with similar symptoms and problems—an interval hardly ideal for learning and improving skills. Medical residents enjoy the possibility of seeking advice and feedback from their supervisors, but clinicians in independent practice are less likely to have the same opportunity. Actively seeking feedback needs to be supported by better-designed learning environments41—for trainees and those in practice alike. Supervisors should be encouraged and trained to give specific individualized feedback that allows the resident to make appropriate changes to his or her performance through designed practice42; however, one challenge for supervisors is that they cannot give trainees completely accurate feedback until patients have been assigned a final diagnosis and treatments have been completed, which could, regrettably, take days, weeks, months, or even years.
One innovation through which experts can give feedback to trainees is simulation. There has been a strong interest in creating, for some professional domains of expertise, learning environments that simulate real-world environments where practice may lead to real improvements in performance. These environments allow instructors the ability not only to provide high levels of control over the situations but also to offer the trainee immediate, informative, and accurate feedback.
Developments in technology during the 20th century afforded the possibility of designing simulators for training airplane pilots and others with critical jobs (e.g., operators of nuclear reactors, drivers of trains). Newer developments have allowed for the creation of flight simulators that can reproduce the entire flight mission, including planning the flight, communicating with the airport flight controllers, and managing emergency events. One of the first meta-analyses of training in airplane simulators showed that the average effect sizes of adding simulator training to regular flight training (all trainees proceed at the same pace) were relatively small (r pb around 0.1–0.2), but that effects of mastery-based training (trainees proceed only after attaining a predetermined level of performance) were substantially higher (r pb around 0.5).43 (p72) Subsequent reviews emphasized the interactions between level of skill and the fidelity of the simulator (i.e., beginners benefit more from low-fidelity simulators than higher-fidelity ones).44 Other researchers found large benefits of brief simulator training for new procedures, but no significant benefits for general skills controlling the airplane45,46 or landing it, especially on aircraft carriers.47 These findings indicate that the target skill must be analyzed carefully to ensure that the simulator functionally represents the critical perceptual and control characteristics of real-world situations.
The last several decades have witnessed remarkable advances in the use of simulators in medicine. Most of the research in training has focused on preparing medical students, interns, and residents for their first medical procedures with human patients. The simulators provide trainees with an opportunity to execute a particular procedure using a device that, as much as possible, replicates a particular medical situation. Some computer-based simulations of the body (or its parts) provide feedback on how successfully trainees have inserted needles or manipulated targets. In a groundbreaking review, Issenberg and colleagues48 analyzed learning outcomes of simulation-based training in an effort to assess which particular training conditions were transferable (i.e., associated with improved performance of the procedures outside the simulator). The most important element for effective, transferable learning was linked to having explicit performance goals. Receiving immediate, accurate feedback and repeatedly performing the assigned task were also vital elements. In a subsequent review,49 these authors argued that these elements, shown to be essential for effective learning, corresponded to the elements of deliberate practice. They also argued that training in the simulator must be extended until each trainee reaches a predefined level of performance associated with mastery.49
One influential study of simulation-based medical training involved skilled surgeons performing tasks on simulators. The researchers used these experts’ simulator performance as guidelines to determine mastery goals for students training with the same virtual reality simulators.50 Authors of another, comprehensive review determined that medical education using simulators embraces best teaching practices: “distributed, structured, and deliberate practice,”51 (p336) “appropriate mechanisms for feedback,”51 (p336) and objective training goals. Authors of a very recent review52 found that this type of simulation-based mastery learning is associated with increased learning compared with traditional medical education (i.e., classroom instruction and supervised performance in clinics) and results in significant transfer to clinical outcomes, such as improved patient care and health. Studies of mastery learning show that trainees’ skill in performing medical procedures can be greatly improved over traditional medical education by providing the trainees with simulator training that provides immediate feedback and opportunities to repeatedly perform until they reach an objective criterion.
The Expert-Performance Approach With Deliberate Practice as Distinct From Mastery Learning
This current Perspective focuses on the expert-performance approach with deliberate practice, which is distinct from mastery learning. The concept and practice of mastery learning has a long history, beginning in the 1920s and 1930s.53 The central idea is that K–12 students are able to attain the same learning goals, but require different amounts of time to do so.54 Giving all students enough time to master a topic or skill (as determined by meeting some predefined test score) ensures that all students have the same or similar mastery of the prerequisites when they move on to the next educational topic.
The mastery concept makes sense within a sequential curriculum for learning skills of increasing complexity, such as in general K–12 education, which requires students to master specified knowledge and skill prerequisites to be prepared for more advanced courses in mathematics, sciences, and the humanities. The main goal of education is mastery of general knowledge and skills that are likely applicable in any profession or path available to high school graduates. In the case of medicine, the training in medical school prepares students for further training in any medical specialty. Given that continued professional education has only a very modest impact on clinical practice37 and the accumulation of professional experience beyond the first years has only a small effect of performance,38–40 one might question whether the effectiveness of current medical education is optimal for developing performance of medical professionals.
The expert-performance approach with deliberate practice is in many respects the opposite of general education because it starts by focusing specifically on the particular desired end product of training and experience—namely, the representative target performance of medical specialists, such as surgeons or radiologists who have patient outcomes that are superior to their peers. By analyzing the superior target performance, the expert-performance approach with deliberate practice identifies the mental structures and representations that expert specialists have acquired and refined during the extended period of their training and professional practice.
Several of the criteria for mastery learning55 are consistent with some of the prerequisites for deliberate practice; these include the importance of an explicit goal for training; access to immediate, accurate, and detailed feedback; and opportunities for repetition and practice until a prespecified level of performance has been attained. In mastery learning, learning is evaluated almost entirely by the learner’s attained performance on the specified criterion test. Mastery learning was developed within the theoretical framework of behaviorism and thus does not entail assessing the cognitive processes mediating the acquired performance. In direct contrast, the expert-performance approach includes an attempt to assess participants’ thought processes and involves evaluating how the improved performance is mediated through and integrated with other skills and knowledge related to the final or target superior professional performance. For the expert-performance approach, the successful integration and continued refinement of different skills provide the keys to the development of high (expert) levels of complex performance.
Surgery as an Example of Expert Performance Mediated by Acquired Cognitive Representations
In this section, I describe how the expert-performance framework with deliberate practice can provide additional insights into superior performance in surgery and the associated cognitive representations mediating performance in surgery.
According to the expert-performance approach, the first goal in the scientific study of expert performance in a particular domain is to identify reproducibly superior performance in authentic contexts. The next step is either to study this performance in that context or, ideally, to capture the performance through representative tasks in the laboratory so that the superior performance can be repeatedly reproduced so as to identify its mediating characteristics and, in particular, its acquired mental representations. The final step involves creating training methods that can develop the associated cognitive representations in potential experts effectively.
Identifying expert performance in surgery
It is not easy to find activities for specific medical specialties, for which differences in objective, uncontroversial measures of patients’ outcomes can be directly linked to the performance of individual clinicians. One of the best examples, however, where this link is possible concerns outcomes of cancer surgery.
In two particularly relevant studies, Vickers and colleagues13,14 show that patient outcomes gradually improve as a function of the surgeons’ experience with a particular procedure. They examined the surgical removal of the prostate, an especially good measure of surgical skill because “adjuvant therapy is not commonly given for prostate cancer and recurrence is not substantially affected by other aspects of postoperative care”13 (p1171) According to their research, less experienced surgeons who completed fewer than 10 procedures were almost twice as likely to have patients with a recurrence of the cancer as compared with experienced surgeons with more than 250 completed procedures.13 The gradual extended improvement is even more striking in another review that examined outcomes of cancers restricted to a single organ.14 In these cases, the recurrence of cancer declined as a function of surgeons’ increasing experience for the first 1,500 to 2,000 procedures, at which point the recurrence of cancer was essentially eliminated. These improvements in surgical outcomes as a function of more experience are likely related to unique characteristics of surgery. Unlike many other medical activities, the surgeons receive immediate feedback from mistakes and other unexpected problems during surgery. Further, during the subsequent hours or days the patient is in postoperative care, the surgeon often has the opportunity to diagnose problems, which might even lead to the need for immediate corrective surgery and feedback about the cause of the associated problems.
Cognitive processes that mediate surgeons’ superior performance
Because the second step in the expert-performance approach is to identify how expert surgeons’ thought processes differ from those of less accomplished ones, here I review these processes, showing how these thoughts provide evidence for acquired mental representations.
A common view of experts’ performance, in line with traditional theories of skill acquisition, is that the experts have automated their performance56,57; thus, experts’ performance, according to some traditional theories, is guided primarily by intuition.58,59 This idea is testable: Simply interviewing experts provides an initial rough estimate of how much intuition drives performance. In one recent study,60 investigators asked eight surgeons general questions about their decision making before, during, and after laparoscopic surgeries. Probably the most interesting conclusion was that surgeons reported that they could execute only the most straightforward cases according to simple rules. The investigators found that most laparoscopic surgeries are too unpredictable and that “even expert surgeons find themselves in situation[s] in which they must thoughtfully reevaluate their approach during surgery, evaluating alternative actions, such as the selection of different instruments or changing the position of the patient.”60 (p1036) In other words, intuition did not drive the surgeons’ performance.
In another study, investigators asked 12 surgeons to recall one critical incident involving a particular surgery that had happened within a long time span (as many as two years prior to the interview).61 Half of the surgeons (n = 6) did not report remembering any deliberation of alternatives; they remembered considering only one approach. The remaining 6 surgeons reported analyzing and comparing more than one solution. Given that the decisions were recalled from the past, an obvious danger is that the surgeons might not have correctly recalled all of their thoughts. A group of investigators62 has addressed this methodological problem—essentially avoiding the problem of recall and forgetting—by observing the surgeries as they occurred. These investigators asked surgeon volunteers to predict which of their planned surgeries would likely be challenging, and the surgeons allowed the researchers to be present to observe these surgeries. The investigators interviewed the surgeons immediately after the surgery. The questions they asked focused on situations in which the surgeons were not sure what to do next.62 The investigators identified one nonroutine decision in every case submitted to analysis.62 The primary cognitive mechanism surgeons used to detect problems involved noticing a mismatch between their expectations based on the preoperative plan and the actual surgical situation (see Representations 1 and 3 in Figure 3). After they recognized the problem, the surgeons actively generated alternative actions and weighed their relative benefits.62 The findings—based on information about thought processes collected immediately after a completed surgery62—clearly support the hypothesis that experienced surgeons have acquired refined representations for planning surgery, implementing the plan, and monitoring the surgery so they can detect mismatches. Consistent with these findings, reports of recalled thinking during past surgeries after a delay of months or years are unreliable—most likely because of forgetting in the interim period.
A related body of evidence supporting the premise that experts immediately recognize patterns, rather than engage in thinking, comes from some experts themselves, who report that they were not thinking while performing.56–59 These reports are particularly frequent in sports, where athletes, who are interviewed after a competition, often report simply doing what felt right.63 One method to empirically assess the experts’ thoughts during the actual performance is to examine whether their performance relies on perceptual access to (i.e., actually seeing) the current situation while executing a particular action or whether the information is mentally represented and thus accessible from memory without the aid of perception. For example, in one study, squash players wore goggles that could instantly obstruct all vision, and their vision was occluded just after the opposing player had completed their hitting action.64 More-skilled players were more accurate in their anticipation of the ball trajectory than less-skilled players.64 In another study, investigators “occluded” soccer players’ vision while the players were watching a video of a soccer game by unexpectedly stopping the tape and blanking the screen.65 They found that the players’ ability to accurately recall where the other relevant soccer players were on the field and where they were heading was significantly superior for more-skilled soccer players than less skilled.65 Investigators in both of these studies showed that more-skilled and expert performers had extracted more useful and reportable information about the given situation.
Investigators have used this methodology to assess experts’ ability to recall salient details of a dynamically changing environment (situation awareness) or to assess their mental representations of a situation after key data have been removed in domains other than sports.66 For example, in simulator training, researchers have removed all relevant information and asked expert and novice air traffic controllers to recall information relevant to managing the arrival of airplanes67 and fighter pilots to execute specific missions.68
Studies of situation awareness in surgery show that more-skilled surgeons are better able to access the ideal representation of information (Representation 2) relevant to the current state of the surgery. Situation awareness should decrease when external interference, such as a telephone call, interrupts a surgery. One study found that such interruptions were associated with an increased probability of errors in residents, but residents’ situation awareness allowed them to discover most of their own mistakes during surgery, thus avoiding negative patient outcomes.69 Another source of evidence for the need to monitor mental representations during critical situations (i.e., to maintain situation awareness) is based on observations of surgeons whose surgeries had successful or unsuccessful outcomes. An analysis of surgical errors during a particular laparoscopic procedure showed that the injuries were due to misperception of the anatomical structures rather than technical errors, indicating that surgeons had developed an inadequate mental representation.70 According to another study,71 surgeons of different specialties reported slowing or even halting action at critical points during the surgery when they increased their attention. This study also reported evidence that surgeons decreased their attention and situation awareness during so-called “easy” operations and that these surgeries were associated with near misses or errors.71 Similarly, Bann and colleagues72 (p414) have argued that “senior surgeons are more prone to slips and lapses.” In sum, findings from these studies70,72 support the idea of automatic habitual processing (low situation awareness) in some standard surgeries by experienced surgeons; however, I argue that this type of processing is not a sign of expertise but, rather, a sign of reduced attention that may be leading to an increased risk of error. Consistent with research on expert performance, the superior performance of experienced surgeons is associated with refined representations to plan, to execute, and to monitor surgical states, which allows these surgeons to be prepared for unexpected outcomes and carefully consider the best solutions to problem situations.2,24
Developing and refining mental representations: Implications for training
The evidence for developing mental representations to refer to in planning for, executing, and monitoring surgeries raises questions about how training might be designed to help learners develop these representations more effectively. Several findings support the potential benefits of training outside the operating room. Superior skill in identifying relevant anatomical structures during laparoscopic procedures is associated with reduced risks of injuring adjacent tissues, ducts, and vessels.70 Recently, a researcher presented surgeons of differing experience with pictures from laparoscopic surgeries taken just prior to making a surgical cut.73 The surgeons were instructed to mark the spot where they would cut for the surgery. Although the author observed systematic differences between the groups (more experienced surgeons recommended different initial cuts than less experienced surgeons), no independent gold standard was available to demarcate the best location for the proposed cut. Still, the findings indicate that the skill of deciding where to make cuts during surgery should be taught directly during training, especially for less experienced surgeons. Finally, research on supportive skills for laparoscopic surgery is available: Experienced laparoscopic surgeons were interviewed about their methods for manipulating tissues and generating superior views via the camera to determine the tissue planes.74 In sum, these findings support the existence of mental representations and the ability to access them outside the operating room, which has implications for designing medical education to support their development.
If clinical instructors were able to review videos of prior surgeries, extract recordings of particularly relevant situations, and store these recordings in a library, they would be able to develop in trainees the ability to identify anatomical structures. For example, a trainee could view a frozen screen and draw the most appropriate cut as rapidly as possible without sacrificing accuracy. A computer could analyze responses, providing trainees with immediate feedback and, later, opportunities to perform similar tasks at the same or higher levels of difficulty.
Creating feedback loops that allow for improvements of current surgery performance
One of the most interesting developments in medicine is the current effort to carefully document a trainee’s behavior during surgery to identify near misses and mistakes. In one case study, a Canadian neurosurgeon named Mark Bernstein worked with his team to record and enter all errors into a computer after every elective surgery he performed (n = over 1,000) from 2000 to 2006.75 He even included minor mistakes, “such as dropping of a sponge.”75 (p1076) These descriptions were analyzed to relate them to complications so as to identify preventable errors that caused the complications. Following this published analysis, Bernstein continued to record his errors from August 2006 to May 2013,76 and the error rates associated with this latter period were compared with the earlier period. The average number of errors and error-related complications fell by over 50% during the second period. This reduction in errors, especially during the first years of recording (2001–2002), suggests that the mere act of attending to errors to record them may have an effect on their subsequent occurrence.
Perhaps one of the most exciting developments in the measurement of behavior during surgery is the systematic video recording of surgeries followed by detailed analyses of the videotapes.77 This method—review by, ideally, an independent expert blind to the identity of the particular surgical team, to avoid any potential bias in the coding of errors77—can be used to assess weaknesses in a surgeon’s current performance, and can thus serve as the starting point for training surgeons using deliberate practice. An essential prerequisite for communication between teachers and students is the creation of detailed coding schemes, as illustrated by one for laparoscopic gastric bypass surgery that has been shown to be reliable and valid.78 Using such a coding scheme should allow the identification of particular technical problems so that trainees may practice avoiding these and receive accurate feedback to improve—before entering an actual operating room.
This type of video recording, as a means to identify training goals for improvement, is not limited to less experienced surgeons. In a particularly interesting study, Birkmeyer and colleagues79 collected a video of a single bariatric surgery from each of the participating surgeons. They rated each surgeon’s surgical skill and discovered post hoc that the ratings were related to the complication rates for the same surgeon’s normal surgical practices.79 The findings of this study indicate that it may be possible to reduce complication rates for patients by training the surgeons to increase their skill in performing their surgeries.
The methodology of using video recordings and their independent assessment seems to offer a potential feedback loop through which weaknesses and potential problems can be identified. These areas requiring improvement could then be addressed through targeted training focused on the relevant technical skills, the perceptual skills necessary to sense and understand the critical anatomical structures, the ability to plan the surgery, and/or the capacity to detect and deal with unexpected deviations or events.80 Recognizing the development of refined mental representations in skilled surgeons may have implications for learning and teaching, for introducing and acquiring, the skills necessary for a particular surgical procedure.
How to introduce learning of surgical procedures
When surgical trainees are trained through the mastery learning approach, they receive general instruction about a particular procedure and then are allowed to perform the procedure with a simulator. The simulator provides feedback about the accuracy of trainees’ actions and, often, the amount of time they took to complete the procedure. The trainees then repeat the procedure until they have reached a predetermined proficiency level. A recent review shows that simulator-trained participants perform better than the control participants when tested on the simulator, on animal models, and even on human patients who lack complicating factors.81 These findings show that simulator training leads to superior performance when tested with conditions similar to training or simple clinical cases, but so far this type of training has not been designed to build and shape the superior surgical performance of independent surgeons.
The expert-performance approach with deliberate practice demands that the required performance on the simulator be an extremely close approximation of the relevant aspects of the procedure on an actual patient.2 The trainee’s initial performance should establish a good representation of the procedure, and each subsequent performance can be reviewed and, with more training, refined until the trainee reaches highly skilled levels. To illustrate, children do not spontaneously adopt the best postures and techniques when playing the piano, so their teachers instruct them and closely monitor their playing until the children acquire the correct fundamental actions and postures. Failure to adopt the correct fundamental actions will limit the individual’s ability to perform technically difficult music pieces (also, incorrect fundamental technique often leads to overuse injuries among adult professional musicians). Similarly, surgical trainees using simulators should receive supervised instruction about acquiring the correct fundamental technique to maximize their future skill. Currently, trainees are allowed to execute the procedure on the simulator in whatever manner feels most natural to them; they typically receive no information about more advanced techniques that might be useful in successfully completing more challenging future surgeries.
In a recent study,82 investigators examined the possibility of giving trainees feedback so they could attain the correct fundamental techniques. This study compared two groups of inexperienced surgeons, all of whom trained on a simulator to perform a laparoscopic cholecystectomy (LC). One group received feedback on their weakness and experienced 30 minutes of training targeting that weakness (i.e., deliberate practice) before they completed a second LC, whereas the other group watched surgical tutorials unrelated to laparoscopy or cholecystectomy for 30 minutes. Although both groups showed improved performance on the simulator and on a porcine model, the deliberate practice group attained a higher quality of simulator performance and showed superior transfer to real tissue.82
Expert performance in the surgical environment requires the ability to execute a wide range of emergency procedures. For some of these procedures, the primary emphasis is on the speed in which simple sequences of actions are executed; in such procedures, entrenched action sequences are desirable. Elizabeth Hunt and colleagues83 developed a training procedure that they named “rapid cycle deliberate practice” for resuscitation. After a team of trainees received instruction on performing the procedure, their time to initiate heart compression and other critical events was recorded. Rather than having the team discover more effective methods through practice and discussion, the instructor next provided step-by-step guidance on the best procedure for initiating heart compressions as fast as possible. After completed “rapid cycle deliberate practice,” in which the team practiced the procedure over and over and the instructor provided guidance on improving weaknesses, the trainees were almost four times as likely to start heart compressions within one minute of loss of pulse, a factor related to successful resuscitation.83
An important question to ask when considering the expert-performance perspective is whether the cognitive representations used to perform surgical procedures in the simulator match mental representations when working on actual patients in the operating room. Research so far is inconclusive. Recent tests comparing experienced and less-skilled surgeons’ performances on simulated cases have shown that performance in simulation centers correlates to performance both on other simulated cases and with actual patients in operating rooms.84 Another group of researchers have found a significant difference in performance on simulated catheterization cases in a laboratory between novices and experienced interventional cardiologists, but not between two groups of experienced cardiologists even when the cardiologists in the experienced group had large differences in the number of completed therapeutic procedures.85 Other researchers, studying coronary angiography, rated videos of catheterization procedures performed on actual patients and on simulated cases in the laboratory.86 They found that experience improved performance on actual patients, but not on the simulator. Collecting think-aloud protocols and immediate retrospective reports24,87 from surgeons performing both in the operating room and in the simulation lab should enable the comparison of thought processes and mental representations used in the two situations. In turn, these comparisons should enable the refinement of simulators such that training tasks in the simulator require the trainee to conduct the procedure in the simulator with the same or very similar actions used on actual patients.
At least one general training approach minimizes the problems associated with acquiring representations during simulator training that differ from those used in the operating room. Palter and Grantcharov88 centered their training on an analysis of videotapes of individual’s surgical performance on actual patients. After each completed surgery, an instructor reviewed the videotape to identify weaknesses and then assigned targeted training in the simulator for remediating these weaknesses. To assess the effects of this type of training, the investigators compared two groups of novice surgical residents performing LCs in the operating room. One group of residents was assigned practice tasks in the simulator based on an analysis of their videos of their first surgery so they could focus on improving the weakest aspects of their performance. Another group of residents, who served as the control group, were given informal feedback as is traditionally done in surgery education. After this experimental intervention, the two groups performed a second (also videotaped) LC in the operating room. Blind analysis of the videotaped performance showed that the deliberate practice group was now superior to the control group; the deliberate practice group in fact improved to such an extent that the distributions of performance for the two groups did not even overlap.88 This intervention is a particularly effective demonstration of how all aspects of deliberate practice can be applied in the surgical domain. Instructors applying this training method would assess the actual surgery in the operating room for weaknesses and design the training in the simulator to improve those weaknesses.
Maintaining surgical performance
One of the most consistent predictors of surgical outcomes is the volume of surgeries completed by a given surgeon. Recent studies of surgical outcomes have shown that the length of time between consecutive surgeries of a given type is significantly related to patient outcomes—the longer the gap, the worse the patient outcomes.89–91 This pattern is consistent with the earlier reviewed studies29–34 on other types of skills.
Cognitive Processes That Mediate Improvements in Performance in Other Medical Activities
The permitted length of this Perspective does not permit a comprehensive review of all the different types of medical activities and tasks; therefore, I have focused the remainder of my Perspective on two medical tasks—namely, interpreting X-rays and interviewing patients, for which adapting the expert-performance approach with deliberate practice is markedly different. An important difference between the two tasks is related to their different levels of complexity. Interpretation of a variety of static X-ray images entails examining fixed images that can be easily removed from the original radiology clinic and presented to experts or trainees, including medical students, interns, residents, and radiologists, with few or no changes, thus creating a standardized means of capturing the essential elements of the task. In contrast, patient–doctor meetings involve an extended interaction that depend on the particular patient and his or her problems as well as the associated responses of each doctor.
Performance of X-ray interpretation
Measuring the performance of interpreting X-rays is relatively easy for mammograms. The general method is to collect a number of X-rays from actual patients and then wait a sufficient amount of time to procure a final diagnosis for each. Given the low rate of cancer cases, collecting X-ray images from mammograms requires waiting long enough to amass a sufficient number of pathological cases and then mixing in a number of normal cases.
A recent review shows that the accuracy of interpreting mammograms as screens for breast cancer increases on average as a function of the number of completed interpretations; specifically, radiologists who have not completed a fellowship incorrectly call fewer and fewer women in for further testing (false positives) over their first three years of practice (33 in year 1; 19 in year 3)—without any changes in their rate of missing cancers.92 Of particular relevance for training effectiveness is the finding that radiologists who completed a radiology fellowship already performed at the expert level during their first year of independent practice.92 Although this finding is only correlational, it suggests that the period of fellowship training in radiology affords the fellows the opportunity to improve their diagnostic performance prior to the start of independent practice.
Interestingly, while both this study92 and another93 show that experienced mammographers’ performance is reproducibly superior to that of less experienced mammographers and that experienced mammographers meet the criteria for expert performance, both also report that even experienced mammographers have large individual differences in the accuracy of their diagnoses. Researchers have not yet been able to specify the nature of these individual differences, which would help clinical teachers individualize training in mammography.94
Using a particularly promising approach to describe the processes mediating superior performance, investigators asked highly experienced and less experienced individuals to think aloud while making their diagnoses.95,96 In the more recent study,96 investigators asked 10 radiologists and 10 radiology residents to think aloud while diagnosing the same mammograms. The analysis of the think-aloud protocols indicates that cognitive processes associated with more experience are associated with superior mental representations of normal cases, which allow experts to carefully analyze findings in all mammograms. The authors of this study96 also found that individuals with higher levels of expertise were more able to self-regulate and apply successful search and reasoning strategies. These findings indicate that it would be fruitful to study if and how experienced mammographers engage in learning with immediate feedback, and what their thinking and learning processes are, especially when they make a mistake.
Another group of researchers97 analyzed the incorrect diagnoses of a relatively large group of radiologists (n = 92). Higher-performing mammographers identified the types of cases and lesions that lower-performing mammographers missed. In another study,98 investigators examined the effect of feedback on how accurately mammographers detected cancer; these investigators also analyzed the results of biopsies ordered based on an original reading of the X-ray. A higher frequency of workups was associated with a significantly higher cancer detection rate, but the frequency of women being asked to endure an unnecessary biopsy (i.e., of false positives) was also higher. These two findings have helped to establish not only review procedures that are associated with higher accuracy in diagnosis but also means of targeted practice that allow lower-performing radiologists to improve their performance by identifying cancer in a variety of cases, even at the level of their higher-performing peers.
The traditional training of radiologists is based on the apprentice model, through which the apprentice, typically a resident, examines submitted X-rays to generate a preliminary diagnosis. Subsequently, the resident’s supervisor examines the X-ray and gives an official diagnosis, which serves as the gold standard. Nodine and colleagues93 administered a test to radiology residents and fellows and their supervisors (mammographers) on interpreting a number of mammograms. The accuracy of the breast cancer diagnoses increased as a logarithmic function of the number of mammograms that the individuals had encountered during their professional experience and reached a stable level of accuracy (though this level is far from perfect) at around 10,000 mammograms, which is generally the number of mammograms completed by the supervising mammographers.
In the current training system, supervisors must make their decisions without knowing the correct diagnosis because a couple of years may pass before they can infer with a high degree of confidence whether the patient had cancer or was cancer free at the time of the submitted X-ray. By establishing a library with old X-rays, relevant patient information, and final diagnoses (to serve as the gold standard), radiology instructors may be able to design a learning environment that provides cases and immediate feedback on proposed diagnoses.99 Such a library could then be computerized, such that X-rays and diagnoses are indexed. The assembled library of cases could thus serve the dual purpose of measuring performance for interpreting X-rays and teaching trainees about different types of X-rays, such as mammograms or bone lesions. If instructors observe weaknesses for certain features or particular types of X-rays, they could (or a computer program could) generate training sets in which practice items are organized by difficulty to make deliberate practice possible. In fact, one group of researchers, Pusic and colleagues, developed a case bank of 234 digital items in an initial study,100 and they were able to show that the accuracy of diagnosis for the items in the library was significantly superior for radiologists compared with pediatric fellows, and stepwise, for pediatric fellows compared with medical students. In a related study101 focused on training, they designed a practice environment in which residents were able to get immediate feedback on each completed diagnosis. With practice, the residents increased their diagnostic accuracy as a function of the number of radiographs that they had studied. On the basis of this practice curve, Pusic and colleagues estimated the number of additional practice X-rays necessary to reach the accuracy demonstrated by the attending pediatricians.
Another recent study102 demonstrated the effectiveness of receiving immediate feedback on diagnoses of mammograms displayed through a DVD. The study randomly assigned trainees to getting training with the DVD or being members of a control group with no additional activity. The DVD group performed significantly better than the control group on a subsequent test with different mammograms. It should be possible to go beyond simply presenting all trainees with the same sequence of mammograms and giving immediate feedback on each case. For example, clinical instructors should be able to assess weaknesses in the trainees’ performances by examining their cognitive representations collected either through an analysis of their think-aloud protocols or through their sketches of recalled mammograms. This type of individualized coaching would contain all the essential elements of deliberate practice.
Clinical interactions with patients
An important aspect of most professional medical doctors’ everyday activity involves communication with patients. These interactions with patients103 include interviews to elicit information about patients’ medical problems, discussions to educate patients about how their problems might be improved by treatment, and of particular importance, conversations to jointly develop a plan for care. Clear evidence shows that adherence to the recommended treatment plan, including taking prescribed medications, influences patient outcomes for chronic diseases.104,105 Improving a doctor’s communication skills, therefore, likely increases his or her patients’ adherence to treatment plans and, in turn, improves outcomes. Similarly, research has shown that cancer patients’ understanding of their disease and its prognosis, which is tied to doctors’ communication skills, is related to better decisions about end-of-life treatments.106 A review examining patients treated by professional palliative care specialists who experienced extended training in, specifically, communicating with cancer patients demonstrated significant beneficial effects on patient outcomes.107 At the same time, other recent studies have failed to demonstrate increased patient satisfaction or improved patient outcomes after physicians experienced either short-term training in communicating with patients with serious illness108 or general training in hospitalist communication skills.109 In one of the studies109 the training consisted only of three 90-minute workshops, but in the other108 the training lasted four days and even included skills practice with standardized patients. Notably, only with the more extended training did the investigators observe significant effects on behavior during the training,108 but these effects did not transfer to the clinical environment. An insightful comment110 proposed the need for new and innovative ways to teach communication skills and argued for a tighter connection between measured clinical performance and designed practice—just as I have advocated earlier in this Perspective.
The idea of effective communication with patients as a set of teachable skills implies training adults (i.e., making changes to preexisting adult behaviors); therefore, it seems plausible that workshops or even a four-day training will be insufficient for attaining substantial improvement in everyday performance.2 These findings are consistent with the earlier observations that continued professional medical education courses did not influence behavior in the clinic,37 and that mere professional experience after the first couple of years of independent practice or deliberate practice rarely improves objective performance.38–40,111,112 Wouda and van de Wiel103 make the important point that medical students already enter medical school with a number of previously acquired habits and skills for communicating with other people; these habits are unlikely to change with mere experience, and they are difficult to modify even through instruction or modeling. These authors103 argue that the most effective methods of influencing communication skills involve video recording a physician’s or trainee’s interactions with peers, relatives, and real patients. Instructors can then view the videotapes and identify key weaknesses and problems that they can then address directly with the physician or trainee. In this way, the teacher’s one-on-one guidance may not differ much from piano teachers’ and tennis coaches’ directed work with individuals. In all three scenarios, the instructor identifies an area to be improved and either instructs the trainee to engage independently in deliberate practice of appropriate tasks or spends an hour interacting with the trainee. At first the clinical instructor may provide simple situations, so the trainee can anticipate what will happen. For example, during the training of backhand volleys, the trainee might be standing at the net waiting for an easy hit that the coach delivers in a consistent manner. With increased success, the coach will make serves more challenging, and eventually the player and the coach will engage in a volley in which the trainee must be prepared for backhand volleys during regular playing. This method—that is, integrating a deliberately practiced skill into normal execution of the activity—can be applied to improving communication with patients. After understanding and practicing to eliminate their key weaknesses, students would engage in more (tape-recorded) patient interviews with the goal of gaining information or preparing the patient for the plan of care. Importantly, the improved communication skills must be integrated into the trainee’s increasingly refined mental representations not only for encoding and combining patient information but also for perceiving and responding to patients’ reactions and expectations. Consequently, the development of superb communication skills is an extended process, and the recommendation is that doctors design a plan for continued education and training throughout their careers to improve the effectiveness of their communication skills. A recent report113 describes efforts to implement communication-skills training for residents based on video recordings of outpatient consultations. This process of reshaping the medical education system will be long, and the first step involves training the supervisors so they can serve the residents as qualified and effective teachers.
The research on expert performance differs from that on general education, which focuses on the acquisition of new knowledge and general rules that educators hope will be widely applicable in many professional domains. The theoretical framework of expert performance also differs from the theories of expertise that focus on the knowledge and rules acquired prior to or during active practice. According to expertise theories, pattern matching and effortlessly retrieved memories of previously executed actions eventually, naturally replace knowledge and consideration of rules. A finding that is inconsistent with these theories indicates that new knowledge encountered at CME seminars and conferences is not effortlessly converted into changes in habitual behavior in the clinic.37 Furthermore, research shows that additional professional experience with familiar tasks does not consistently improve accuracy of performance38–40 but, rather, primarily makes the associated action sequences consistent, efficient, and nearly effortless. Instead, improved performance is related to goal-directed training with immediate feedback, as suggested by the expert-performance approach with deliberate practice. Effective practice involves the refinement of mental representations during training activities through which individuals attempt to go beyond their current habitual performance by trying to attain higher performance goals. To facilitate these gradual improvements in performance, a long-term commitment to monitor performance in the clinic is necessary for integrating and transferring the skills and knowledge gained through corrective training and feedback. This training process will be more effective and reliable with the help of a supervising teacher.
This understanding of how to improve medical performance could have implications for the selection of medical students and resident applicants; that is, performance on basic tests of perceptual-motor and spatial performance for medical school applicants and initial performance on relevant simulators for residency applicants could help determine admission or placement.114 This proposal depends on the assumption that tests of basic abilities measure prerequisite abilities for attaining more advanced motor skills. The results of one of the first studies115 to demonstrate this relationship between simulator performance and spatial ability cast doubt on this assumption. The surgical residents with lower scores on the spatial ability test were able to eventually achieve a simulated performance comparable to that of the high-ability residents. A subsequent study by the same research group examining dental students showed that “after 10 minutes of supervised practice and feedback,”116 (p756) students with the lower visual–spatial scores performed as well as those with higher test scores.
Although performance on basic tests measuring spatial ability,117 as well as video game experience,118 has been found to significantly predict initial simulator performance, individual differences in performance on simulators decrease with extended training, and, likewise, the correlations between basic abilities and simulator performance for novices change as a function of level of acquired skills.119 Even more importantly, individual differences in simulator performance do not seem to transfer significantly to the operating room.120,121 Investigators tested the spatial abilities of experienced surgeons and found that their scores did not significantly correlate or account for individual differences in surgical performance,116 which led the investigators to conclude that “practice and surgical experience appear to obviate the impact of innate abilities.”116 (p757) These findings from surgery align with those (discussed earlier) that fail to show a tight relationship between simulator performance and clinical performance at the individual level.84–86 They also support the general pattern observed across a wide range of domains of expertise.122
When novices begin a new activity in any new domain, they must rely on their existing skills and abilities, but as their skills are constructed during teacher-led practice and as they acquire appropriate representations to support their superior performance, the correlations between actual performance and basic ability fall to insignificance. Future research on the acquisition of the representation of superior performance in the operating room will help medical educators understand the apparent nonrelationship between initial performance in the simulator and eventual expert surgical performance.
The focus of the expert-performance framework is on the highest levels of observed performance in a domain, such as a medical specialty. Unfortunately, only a small number of studies have analyzed the performance of medical specialists through the methods commonly used to assess the structure of either expert performance or the training history of elite performers in other domains of expertise (e.g., music, sports). I believe that such studies, if/when they are conducted, will show that elite performers have acquired mental representations and fundamental skills in a manner that allows them to use the foundational skills to integrate improvements and build toward the mental representation of the ideal performance. Traditional professional education has not designed sufficient opportunities for this type of slow acquisition and refinement of skills. For example, in preparing for their first surgical procedure in the clinic, trainees may have spent thousands of hours acquiring knowledge and general rules, but only a handful of hours performing the actual procedure. In other domains of expertise, such as music, ballet, chess, and sports, the ratio between practical or technical training and theoretical training is reversed.
Once academic medicine recognizes that medical interns and residents may need to practice in designed training environments considerably longer than they currently do in order to attain the superior skills that are desirable—necessary even—for independent clinical practice, the community might begin to reform medical training. For example, initial training in fundamental skills (e.g., in the execution of surgical procedures, in interviewing patients, and in making perception-based diagnoses) may begin much sooner. The availability of videos and simulators should enable the identification of procedures for which training could begin early. Without risks to patients, medical students and residents could benefit from hundreds of hours of practice with fundamental techniques in simulators. They could develop perceptual and spatial skills from training with deliberate practice using videos of procedures collected in a library. Another benefit of articulating the particular skills required for medical students and residents is that objective tests for these skills would need to be specified, which, in turn, would permit the design of training environments that would provide rich opportunities for motivated trainees to acquire the fundamental skills at their own pace (whether before or during medical school or residency).
The new interest in improving the effectiveness of medical education and in simulation training offers a timely opportunity to motivate the collection and analysis of objective and detailed data on medical performance by individuals and teams. The expert-performance framework offers a general approach for monitoring and facilitating the development of mental representations and acquired skills which trainees can use to mediate superior individual performance. Identifying and analyzing reproducibly superior clinical performance should enable medical educators to design or “reverse engineer” training that will effectively develop this performance and the associated representations. When medical students, interns, and residents have acquired the necessary set of mental representations, they will have become self-regulated learners—that is, members of the medical community who have the tools to improve their and their team members’ performances over their entire professional careers.
Acknowledgments: Dr. Leonard Hill, Elizabeth Karlin, and Dr. Martin Pusic gave valuable comments and suggestions on an earlier draft of this Perspective.
1. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79(10 suppl):S70–S81
2. Ericsson KAEricsson KA, Charness N, Feltovich PJ, Hoffman RR. The influence of experience and deliberate practice on the development of superior expert performance. The Cambridge Handbook of Expertise and Expert Performance. 2006 New York, NY Cambridge University Press:683–703
3. Ericsson KA, Smith JEricsson KA, Smith J. Prospects and limits in the empirical study of expertise: An introduction. Toward a General Theory of Expertise: Prospects and Limits. 1991 Cambridge, UK Cambridge University Press:1–38
4. Ericsson KA, Charness N, Feltovich PJ, Hoffman RR The Cambridge Handbook of Expertise and Expert Performance. 2006 New York, NY Cambridge University Press
5. Hanushek EA, Rivkin SJ. Generalizations about using value-added measures of teacher quality. Am Econ Rev. 2010;100:267–271
6. Saxon D, Barkham M. Patterns of therapist variability: Therapist effects and the contribution of patient severity and risk. J Consult Clin Psychol. 2012;80:535–546
7. Tracey TJ, Wampold BE, Lichtenberg JW, Goodyear RK. Expertise in psychotherapy: An elusive goal? Am Psychol. 2014;69:218–229
9. Waljee JF, Curtin C. Quality assessment in hand surgery. Hand Clin. 2014;30:329–334, vi
10. Lyu H, Wick EC, Housman M, Freischlag JA, Makary MA. Patient satisfaction as a possible indicator of quality surgical care. JAMA Surg. 2013;148:362–367
11. Lyu H, Cooper M, Freischlag JA, Makary MA. Patient satisfaction as a possible indicator of quality surgical care—reply. JAMA Surg. 2013;148:986–987
12. Yadlapati R, Gawron A, Keswani RN. Patient satisfaction does not correlate with established colonoscopy quality metrics. Am J Gastroenterol. 2014;109:1089–1091
13. Vickers AJ, Bianco FJ, Serio AM, et al. The surgical learning curve for prostate cancer control after radical prostatectomy. J Natl Cancer Inst. 2007;99:1171–1177
14. Vickers AJ, Bianco FJ, Gonen M, et al. Effects of pathologic stage on the learning curve for radical prostatectomy: Evidence that recurrence in organ-confined cancer is largely related to inadequate surgical technique. Eur Urol. 2008;53:960–966
15. Elo AE The Rating of Chess Players: Past and Present. 19862nd ed. New York, NY Arco
16. de Groot AD Thought and Choice in Chess. 19782nd English ed. The Hague, The Netherlands Mouton Publishers
17. van der Maas HL, Wagenmakers EJ. A psychometric analysis of chess expertise. Am J Psychol. 2005;118:29–60
18. Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills. Acad Med. 1995;70:194–201
19. Guid M, Bratko I. Using heuristic-search based engines for estimating human skill at chess. ICGA J. 2011;34:71–81
20. Ericsson KA, Krampe RT, Tesch-Römer C. The role of deliberate practice in the acquisition of expert performance. Psychol Rev. 1993;100:363–406
21. Ericsson KA, Lehmann AC. Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annu Rev Psychol. 1996;47:273–305
22. Bloom BSBloom BS. Generalizations about talent development. Developing Talent in Young People. 1985 New York, NY Ballantine Books:507–549
23. McPherson G, Renwick J. Longitudinal study of self-regulation in children’s musical practice. Music Educ Res. 2001;3:169–186
24. Ericsson KAEricsson KA, Charness N, Feltovich PJ, Hoffman RR. Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts’ performance on representative task. The Cambridge Handbook of Expertise and Expert Performance. 2006 New York, NY Cambridge University Press:223–242
25. Ericsson KA. Deliberate practice and the modifiability of body and mind: Toward a science of the structure and acquisition of expert and elite performance. Int J Sport Psychol. 2007;38:4–34
26. Ericsson KA. Deliberate practice and the modifiability of body and mind: A reply to the commentaries. Int J Sport Psych. 2007;38:109–123
27. Ericsson KA, Roring RW, Nandagopal K. Giftedness and evidence for reproducibly superior performance: An account based on the expert-performance framework. High Abil Stud. 2007;18:3–56
28. Ericsson KA, Roring RW, Nandagopal K. Misunderstandings, agreements, and disagreements: Toward a cumulative science of reproducibly superior aspects of giftedness. High Abil Stud. 2007;18:97–115
29. Krampe RT, Ericsson KA. Maintaining excellence: Deliberate practice and elite performance in young and older pianists. J Exp Psychol Gen. 1996;125:331–359
30. Winfred A Jr, Winston B Jr, Stanush PL, McNelly TL. Factors that influence skill decay and retention: A quantitative review and analysis. Hum Perf. 1998;11:57–101
31. Moxley JH, Ericsson KA, Scheiner A, Tuffiash M. The effects of periods of disuse on performance in a skilled activity. Appl Cogn Psychol. 2014;29:73–80
32. Mujika I, Padilla S. Detraining: Loss of training-induced physiological and performance adaptations. Part II: Long term insufficient training stimulus. Sports Med. 2000;30:145–154
33. Tanaka H, Seals DR. Endurance exercise performance in masters athletes: Age-associated changes and underlying physiological mechanisms. J Physiol. 2008;586:55–63
34. Krampe RT, Charness NEricsson KA, Charness N, Feltovich PJ, Hoffman RR. Aging and expertise. The Cambridge Handbook of Expertise and Expert Performance. 2006 New York, NY Cambridge University Press:723–742
35. Spaner SJ, Warnock GL. A brief history of endoscopy, laparoscopy, and laparoscopic surgery. J Laparoendosc Adv Surg Tech A. 1997;7:369–373
36. Moore MJ, Bennett CL. The learning curve for laparoscopic cholecystectomy. The Southern Surgeons Club. Am J Surg. 1995;170:55–59
37. Davis DAEricsson KA. How to help professionals maintain and improve their knowledge and skills: Triangulating best practices in medicine. The Development of Professional Expertise: Toward Measurement of Expert Performance and Design of Optimal Learning Environments. 2009 New York, NY Cambridge University Press:180–202
38. Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: The relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142:260–273
39. Ericsson KA, Whyte J 4th, Ward P. Expert performance in nursing: Reviewing research on expertise in nursing within the framework of the expert-performance approach. ANS Adv Nurs Sci. 2007;30:E58–E71
40. Spengler PM, White MJ, Ægisdóttir S, et al. The meta-analysis of clinical judgment project: Effects of experience on judgment accuracy. Couns Psychol. 2009;20:350–399
41. Crommelinck M, Anseel F. Understanding and encouraging feedback-seeking behaviour: A literature review. Med Educ. 2013;47:232–241
42. Gauthier S, Cavalcanti R, Goguen J, Sibbald M. Deliberate practice as a framework for evaluating feedback in residency training. Med Teach. 2015;37:551–557
43. Hays RT, Jacobs JW, Prince C, Salas E. Flight simulator training effectiveness: A meta analysis. Mil Psychol. 1992;4:63–74
44. Noble C. The relationship between fidelity and learning in aviation training and assessment. J Air Transp. 2002;7:33–54
45. Roessingh JJM. Transfer of manual flying skills from PC-based simulation to actual flight—Comparison of in-flight measured data and instructor ratings. Int J Aviat Psychol. 2005;15:67–90
46. Rantanen EM, Talleur DA. Incremental transfer and cost effectiveness of ground-based flight trainers in university aviation programs. Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting. 2005 Thousand Oaks, Calif Sage:764–767
47. Lintern G, Roscoe SN, Koonce JM, Segal LD. Transfer of landing skills in beginning flight training. Hum Factors. 1990;32:319–327
48. Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review. Med Teach. 2005;27:10–28
49. McGaghie WC, Issenberg SB, Petrusa ER, Scalese RJ. Effect of practice on standardised learning outcomes in simulation-based medical education. Med Educ. 2006;40:792–797
50. Van Sickle KR, Ritter EM, McClusky DA 3rd, et al. Attempted establishment of proficiency levels for laparoscopic performance on a national scale using simulation: The results from the 2004 SAGES Minimally Invasive Surgical Trainer–Virtual Reality (MIST-VR) learning center study. Surg Endosc. 2007;21:5–10
51. Tsuda S, Scott D, Doyle J, Jones DB. Surgical skills training and simulation. Curr Probl Surg. 2009;46:271–370
52. McGaghie WC, Issenberg SB, Barsuk JH, Wayne DB. A critical review of simulation-based mastery learning with translational outcomes. Med Educ. 2014;48:375–385
53. Block JHBlock JH. Introduction to mastery learning: Theory and practice. Mastery Learning: Theory and Practice. 1971 New York, NY Holt, Rinehart, and Winston:2–12
54. Carroll JB. A model of school learning. Teach Coll Rec. 1963;64:723–733
55. Issenberg SB, McGaghie WCMcGaghie WC. Looking to the future. International Best Practices for Evaluation in the Health Professions. 2013 New York, NY Radcliffe Publishing:341–359
56. Anderson JR. Acquisition of cognitive skill. Psychol Rev. 1982;89:369–406
57. Fitts PM, Posner MI Human Performance. 1967 Belmont, Calif Brooks/Cole
59. Dreyfus HL, Dreyfus SE, Athanasiou T Mind Over Machine: The Power of Human Intuition and Expertise in the Era of the Computer. 1986 New York, NY Free Press
60. Craig C, Klein MI, Griswold J, Gaitonde K, McGill T, Halldorsson A. Using cognitive task analysis to identify critical decisions in the laparoscopic environment. Hum Factors. 2012;54:1025–1039
61. Pauley K, Flin R, Azuara-Blanco A. Intra-operative decision making by ophthalmic surgeons. Br J Ophthalmol. 2013;97:1303–1307
62. Cristancho SM, Vanstone M, Lingard L, LeBel ME, Ott M. When surgeons face intraoperative challenges: A naturalistic model of surgical decision making. Am J Surg. 2013;205:156–162
63. Beilock SL Choke: What the Secrets of the Brain Reveal About Getting It Right When You Have To. 2010 New York, NY Free Press
64. Abernethy B, Gill DP, Parks SL, Packer ST. Expertise and the perception of kinematic and situational probability information. Perception. 2001;30:233–252
65. Ward P, Ericsson KA, Williams AM. Complex perceptual–cognitive cognitive expertise in a simulated task environment. J Cogn Eng Decis Mak. 2013;7:231–254
66. Endsley MREricsson KA, Charness N, Feltovich PJ, Hoffman RR. Expertise and situation awareness. The Cambridge Handbook of Expertise and Expert Performance. 2006 New York, NY Cambridge University Press:633–652
67. O’Brien KS, O’Hare D. Situational awareness ability and cognitive skills training in a complex real-world task. Ergonomics. 2007;50:1064–1091
68. Endsley MR, Sulistyawati K, Wickens CD, Chui YP. Prediction in situation awareness: Confidence bias and underlying cognitive abilities. Int J Aviat Psychol. 2011;21:153–174
69. Wiegmann DA, ElBardissi AW, Dearani JA, Daly RC, Sundt TM 3rd. Disruptions in surgical flow and their relationship to surgical errors: An exploratory investigation. Surgery. 2007;142:658–665
70. Way LW, Stewart L, Gantert W, et al. Causes and prevention of laparoscopic bile duct injuries: Analysis of 252 cases from a human factors and cognitive psychology perspective. Ann Surg. 2003;237:460–469
71. Moulton CA, Regehr G, Lingard L, Merritt C, MacRae H. Slowing down to stay out of trouble in the operating room: Remaining attentive in automaticity. Acad Med. 2010;85:1571–1577
72. Bann S, Khan M, Datta V, Darzi A. Surgical skill is predicted by the ability to detect errors. Am J Surg. 2005;189:412–415
73. Ali S Validation of a Novel Methodology for Evaluating Surgical Tissue Plane Identification [unpublished dissertation]. 2014 London, Ontario, Canada University of Western Ontario
74. Mentis HM, Chellali A, Schwaitzberg S. Learning to see the body: Supporting instructional practices in laparoscopic surgical procedures. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2014). 2014 New York, NY Association for Computing Machinery:2113–2122
75. Stone S, Bernstein M. Prospective error recording in surgery: An analysis of 1108 elective neurosurgical cases. Neurosurgery. 2007;60:1075–1080
76. Oremakinde AA, Bernstein M. A reduction in errors is associated with prospectively recording them. J Neurosurg. 2014;121:297–304
77. El Ahmadieh TY, Harrop J, Batjer HH, Resnick DK, Bendok BR. Blinded peer assessment of surgical skill is feasible and can predict complication rates: A step toward measuring surgical quality. Neurosurgery. 2014;74:N12–N14
78. Zevin B, Bonrath EM, Aggarwal R, Dedy NJ, Ahmed N, Grantcharov TPATLAS Group. . Development, feasibility, validity, and reliability of a scale for objective assessment of operative performance in laparoscopic gastric bypass surgery. J Am Coll Surg. 2013;216:955–965.e8
79. Birkmeyer JD, Finks JF, O’Reilly A, et al.Michigan Bariatric Surgery Collaborative. Surgical skill and complication rates after bariatric surgery. N Engl J Med. 2013;369:1434–1442
80. Ericsson KA. Necessity is the mother of invention: Video recording firsthand perspectives of critical medical procedures to make simulated training more effective. Acad Med. 2014;89:17–20
81. Buckley CE, Kavanagh DO, Traynor O, Neary PC. Is the skillset obtained in surgical simulation transferable to the operating theatre? Am J Surg. 2014;207:146–157
82. Crochet P, Aggarwal R, Dubb SS, et al. Deliberate practice on a virtual reality laparoscopic simulator enhances the quality of surgical technical skills. Ann Surg. 2011;253:1216–1222
83. Hunt EA, Duval-Arnould JM, Nelson-McMillan KL, et al. Pediatric resident resuscitation skills improve after “rapid cycle deliberate practice” training. Resuscitation. 2014;85:945–951
84. Thijssen AS, Schijven MP. Contemporary virtual reality laparoscopy simulators: Quicksand or solid grounds for assessing surgical trainees? Am J Surg. 2010;199:529–541
85. Lipner RS, Messenger JC, Kangilaski R, et al. A technical and cognitive skills evaluation of performance in interventional cardiology procedures using medical simulation. Simul Healthc. 2010;5:65–74
86. Räder SB, Abildgaard U, Jørgensen E, Bech B, Lönn L, Ringsted CV. Association between endovascular performance in a simulated setting and in the catheterization laboratory. Simul Healthc. 2014;9:241–248
87. Ericsson KA, Simon HA Protocol Analysis: Verbal Reports as Data. 1993Rev ed Cambridge, Mass Bradford Books
88. Palter VN, Grantcharov TP. Individualized deliberate practice on a virtual reality simulator improves technical performance of surgical novices in the operating room: A randomized controlled trial. Ann Surg. 2014;259:443–448
89. Hockenberry JM, Lien HM, Chou SY. The impacts of task repetition and temporal breaks in production on human capital and productivity. J Hum Capital. 2008;2:303–335
90. Hockenberry JM, Lien HM, Chou SY. Surgeon and hospital volume as quality indicators for CABG in Taiwan: Examining hazard to mortality and accounting for unobserved heterogeneity. Health Serv Res. 2010;45(5 pt 1):1168–1187
91. Huesch MD, Sakakibara M. Forgetting the learning curve for a moment: How much performance is unrelated to own experience? Health Econ. 2009;18:855–862
92. Miglioretti DL, Gard CC, Carney PA, et al. When radiologists perform best: The learning curve in screening mammogram interpretation. Radiology. 2009;253:632–640
93. Nodine CF, Kundel HL, Mello-Thoms C, et al. How experience and training influence mammography expertise. Acad Radiol. 1999;6:575–585
94. Taylor PM. A review of research into the development of radiologic expertise: Implications for computer-based training. Acad Radiol. 2007;14:1252–1263
95. Lesgold A, Rubinson H, Feltovitch P, Glaser R, Klopher D, Wang YChi MTH, Glaser R, Farr MJ. Expertise in a complex skill: Diagnosing X-ray pictures. The Nature of Expertise. 1988 Hillsdale, NJ Lawrence Erlbaum Associates:311–342 http://csjarchive.cogsci.rpi.edu/proceedings/2007/docs/p65.pdf
. Accessed August 5, 2015
96. Azevedo R, Faremo S, Lajoie SPMcNamara DS, Trafton JG. Expert–novice differences in mammogram interpretation. Proceedings of the 29th Annual Cognitive Science Society. 2007 Nashville, Tenn Cognitive Science Society:65–70 http://csjarchive.cogsci.rpi.edu/proceedings/2007/docs/p65.pdf
. Accessed August 5, 2015
97. Mello-Thoms C, Trieu PD, Rawashdeh MA, Tapia K, Lee WB, Brennan PCFujita H, Hara T, Muramatsu C. Understanding the role of correct lesion assessment in radiologists’ reporting of breast cancer. Breast Imaging: 12th International Workshop, IWDM 2014, Gifu City, Japan, June 29–July 2, 2014. Proceedings. 2014 Cham, Switzerland Springer:341–347
98. Buist DS, Anderson ML, Smith RA, et al. Effect of radiologists’ diagnostic work-up volume on interpretive performance. Radiology. 2014;273:351–364
99. Ericsson KA. Deliberate practice and acquisition of expert performance: A general overview. Acad Emerg Med. 2008;15:988–994
100. Boutis K, Pecaric M, Seeto B, Pusic MV. Using signal detection theory to model changes in serial learning of radiological image interpretation. How much practice is enough? Using learning curves to assess the deliberate practice of radiograph interpretation. Adv Health Sci Educ Theory Pract. 2010;15:647–658
101. Pusic M, Pecaric M, Boutis K. How much practice is enough? Using learning curves to assess the deliberate practice of radiograph interpretation. Acad Med. 2011;86:731–736
102. Geller BM, Bogart A, Carney PA, et al. Educational interventions to improve screening mammography interpretation: A randomized controlled trial. AJR Am J Roentgenol. 2014;202:W586–W596
103. Wouda JC, van de Wiel HB. Education in patient–physician communication: How to improve effectiveness? Patient Educ Couns. 2013;90:46–53
104. Horne R. Compliance, adherence, and concordance: Implications for asthma treatment. Chest. 2006;130(1 suppl):65S–72S
105. Haughney J, Price D, Kaplan A, et al. Achieving asthma control in practice: Understanding the reasons for poor control. Respir Med. 2008;102:1681–1693
106. Temel JS, Greer JA, Muzikansky A, et al. Early palliative care for patients with metastatic non-small-cell lung cancer. N Engl J Med. 2010;363:733–742
107. Higginson IJ, Evans CJ. What is the evidence that palliative care teams improve outcomes for cancer patients and their families? Cancer J. 2010;16:423–435
108. Curtis JR, Back AL, Ford DW, et al. Effect of communication skills training for residents and nurse practitioners on quality of communication with patients with serious illness: A randomized trial. JAMA. 2013;310:2271–2281
109. O’Leary KJ, Darling TA, Rauworth J, Williams MV. Impact of hospitalist communication-skills training on patient-satisfaction scores. J Hosp Med. 2013;8:315–320
110. Chi J, Verghese A. Improving communication with patients: Learning by doing. JAMA. 2013;310:2257–2258
111. Ericsson KA. An expert-performance perspective of research on medical expertise: The study of clinical performance. Med Educ. 2007;41:1124–1130
112. Ericsson KAFry H, Kneebone R. The surgeon’s expertise. Surgical Education: Theorising an Emerging Domain. 2011 Berlin, Germany Springer:107–121
113. Wouda JC, van de Wiel HB. The effects of self-assessment and supervisor feedback on residents’ patient-education competency using videoed outpatient consultations. Patient Educ Couns. 2014;97:59–66
114. Bishawi M, Pryor AD. Should technical aptitude evaluation become part of resident selection for surgical residency? Surg Endosc. 2014;28:2761–2762
115. Wanzel KR, Hamstra SJ, Anastakis DJ, Matsumoto ED, Cusimano MD. Effect of visual–spatial ability on learning of spatially-complex surgical skills. Lancet. 2002;359:230–231
116. Wanzel KR, Hamstra SJ, Caminiti MF, Anastakis DJ, Grober ED, Reznick RK. Visual–spatial ability correlates with efficiency of hand motion and successful surgical performance. Surgery. 2003;134:750–757
117. Buckley CE, Kavanagh DO, Nugent E, Ryan D, Traynor OJ, Neary PC. The impact of aptitude on the learning curve for laparoscopic suturing. Am J Surg. 2014;207:263–270
118. Nomura T, Miyashita M, Shrestha S, et al. Can interview prior to laparoscopic simulator training predict a trainee’s skills? J Surg Educ. 2008;65:335–339
119. Keehner M, Lippa Y, Montello DR, Tendick F, Hegarty M. Learning a spatial skill for surgery: How the contributions of abilities change with practice. Appl Cogn Psychol. 2006;20:487–503
120. Hogle NJ, Widmann WD, Ude AO, Hardy MA, Fowler DL. Does training novices to criteria and does rapid acquisition of skills on laparoscopic simulators have predictive validity or are we just playing video games? J Surg Educ. 2008;65:431–435
121. Hogle NJ, Chang L, Strong VE, et al. Validation of laparoscopic surgical skills training outside the operating room: A long road. Surg Endosc. 2009;23:1476–1482
© 2015 by the Association of American Medical Colleges
122. Ericsson KA. Why expert performance is special and cannot be extrapolated from studies of performance in the general population: A response to criticisms. Intelligence. 2014;45:81–103