Antonoff, Mara B. MD; Swanson, Jennifer A. MD; Green, Courtney A. MS; Mann, Barry D. MD; Maddaus, Michael A. MD; D'Cunha, Jonathan MD, PhD
On a daily basis, surgical interns encounter complex patients in acute care situations, which they may be independently managing for the first time in their medical careers. This transition to independence poses a significant dilemma, emphasizing the challenging balance between the educational benefits of graded, autonomous patient care and the indisputable necessity of patient safety. Further, because of the nature of the surgical service and its hierarchy, senior residents tend to accompany surgical faculty into the operating room during the day, while interns provide immediate, “hands-on” representation of the care teams on the surgical ward. In addition, the enforcement of work hour restrictions has significantly decreased the number of surgical residents who are in-house at nights and on weekends, resulting in a greater sense of isolation for the surgical intern on call.1 These issues result in a significant problem that is not unique to surgical training; important features of this dilemma resonate throughout graduate medical education.
To provide timely, safe, and effective patient care, interns need to be prepared from day one for the daily tasks that they will encounter, both on their own services and in cross-coverage. Although we postulate that most interns will ultimately assimilate the necessary skills and confidence to perform adequately at some time during their first year of residency, their position requires them to begin July with competence and confidence sufficient to be first-responders to serious problems, attributes that are only questionably attainable in the current medical school setting. Fourth-year medical students are exposed to widely differing educational experiences, with no standardized formal preparation for internship.2 This poses a significant dilemma to patient care and identifies an obvious gap in surgical education.
To address this need, in 2008, we, as members of the Division of Education in the Department of Surgery at the University of Minnesota Medical School, developed a four-week, competency-based course for senior medical students (SMSs) entering surgical specialties, specifically intended to prepare them for matriculation into internship. This course consisted of interactive didactics, simulation of emergencies on the surgical ward, and procedural instruction with skills practice. The objective of the course was to prepare students for the tasks commonly encountered in surgical internship, including response to emergent events as well as daily patient care responsibilities. The need for this type of preparatory course has been recognized nationally, as evidenced by the recent establishment of the Entering Surgery Resident Preparation Curriculum Committee, a joint endeavor of the American College of Surgeons, the Association of Program Directors in Surgery, and the Association for Surgical Education.3 Unfortunately, existing courses are in their infancies, are limited in number, and tend to focus primarily on the acquisition of technical (rather than cognitive) skills, and their outcomes as the trainees matriculate into residency have yet to be rigorously evaluated.
Initiatives directed toward formalizing preparation for surgical internship are gaining momentum, as the importance of improving interns' readiness is largely well accepted and intuitively makes sense. However, the task of demonstrating the impact of such courses remains to be addressed. The intent of the present study was to analyze the outcomes of the first year of our educational intervention (which continues today), with the ultimate goal of furthering national as well as local curriculum development and evaluation efforts. In this study, we hypothesized
* that participation in a four-week comprehensive preparatory course for SMSs entering surgical residencies could improve trainee confidence levels in executing specific job-related tasks of surgical internship;
* that such confidence gains would translate to improved performance in these specific tasks when course participants were compared with appropriate controls; and
* that the internship experience itself allows those not exposed to the course to attain competence and confidence commensurate with those exposed to our preparatory course, such that measured differences in competence and confidence would dissipate after a specific period of training.
With this report, we highlight the utility of our novel curriculum while further correlating trainee task-specific confidence to task-specific performance. In this respect, we believe our findings provide insight into the validity of study instruments that measure confidence levels, a relationship that may be useful to future researchers investigating the readiness of SMSs for residency training.
Participants and course structure
We developed a four-week comprehensive surgery preparatory course for incoming first-year residents and, in 2010, first offered it to SMSs matched into residencies in surgical disciplines. A total of 22 SMSs enrolled in the course. This group was composed of 4 individuals who had matched into residency positions in general surgery (18.2%), 5 in urology (22.7%), 4 in orthopedic surgery (18.2%), 2 in anesthesiology (9.1%), 1 in neurosurgery (4.6%), 1 in integrated plastic surgery (4.6%), and 1 in obstetrics–gynecology (4.6%), as well as 4 (18.2%) nondesignated preliminary surgical interns. Ten (45.5%) individuals later became residents at our home institution, and 17 (77.2%) were men. Students enrolled voluntarily, through standard registration practices at our medical school, the University of Minnesota Medical School. This new elective course was offered on a pass–fail basis, taking place as half-days, Monday through Friday, during April of the final year of medical school.
Instruction for the course was delivered via several techniques. Strategies employed included interactive didactics, moderated small-group discussions, team-based problem solving, dynamic simulations of postoperative emergencies, instructed technical skills practice, and mock page exercises.4 The mock page program consisted of a validated set of simulated pages from trained clinical nurse educators, presenting the students with common postoperative calls, followed by immediate one-on-one feedback and weekly classroom-based debriefings. Learning objectives for the course were generated by a committee consisting of clinical surgical faculty members and residents, with input from our previously established internal curriculum for first-year residents.5 Special attention was directed toward structuring the objectives on a foundation of the core competencies as defined by the Accreditation Council for Graduate Medical Education (ACGME). These objectives were refined and expanded upon review of proposed guidelines outlining educational objectives for trainees in their first year of surgical residency,6 as well as the published experiences of outside institutions.7–10 Review of course learning objectives revealed 32 key specific tasks in which the students were instructed.
In all, the competency-based course was comprised of 55 sessions. These included 33 interactive lectures, covering a breadth of topics, ranging from daily tasks such as preoperative assessment and electrolyte replacement to the intense management of situations such as the evaluation of the unstable trauma patient. We allotted time for several small-group discussions regarding topics such as professionalism, communication, ethics, and self-directed adult learning strategies. Participants attended four technical skills practice sessions, using both high- and low-fidelity models in our simulation center, to receive instruction and have opportunities for proctored skills practice in several areas, including airway management, suturing and knot tying, tube thoracostomy, placement of invasive monitoring catheters, and endoscopy and bronchoscopy. Our course included three sessions specifically devoted to the Critical Skills curriculum, a previously validated curriculum developed within our department to teach trainees to competently manage emergent postoperative events.3,11 These unique sessions employed one day of classroom-based software simulation, followed by two days of team-based interactive, dynamic case simulations (one observed case, five practice cases, performance assessment, and debriefing). We also incorporated four “Meet-the-Resident” forums, allowing the students to engage in question-and-answer sessions with current individuals at all levels of surgical training, to seek their guidance as well as their insight regarding expectations and keys to success. Our students participated in a multiinstitutional mock-page program, which included four mock-page debriefings.4 Our sessions also included three order-writing exercises and more than 20 sample case discussions.
Instructors for the course consisted primarily of clinical surgical faculty members, representing key divisions from the Department of Surgery, with guest lectures additionally delivered by expert faculty members from other university departments, including anesthesiology, radiology, hematopathology, and integrative care medicine. Surgical residents participated as moderators for small-group discussions and simulation instruction where appropriate.
We obtained institutional review board approval for this study prior to any data collection, and participants signed all required consent forms. We obtained data at four separate time points: (1) precourse, (2) immediately postcourse, (3) one-month follow-up, and (4) two-month follow-up. An overview of all assessment tools and outcome variables is shown in Table 1. Data collection took place between April and September of 2010.
Measures of improvement
Data obtained from course participants at the first two time points (precourse; immediately postcourse) included surveys, knowledge tests, and performance examinations, which we administered on the first and last days of the class. We subsequently tracked participants as well as the matched peers into their third month of residency, obtaining performance reviews from supervising clinicians at one-month (end of July) and two-month (end of August) points of follow-up.
To determine whether students' task-specific confidence improved through participation in the course, we asked participants to complete self-assessment surveys on both the first and final days of the class. On the precourse survey, for each of the 32 designated tasks, students ranked their own abilities to carry out the tasks. Participants also scored their own perceptions of the importance of each of those tasks. On the last day of the class, students again provided self-assessments of their own abilities on all 32 tasks. Additionally, we asked them to rank the extent to which the course contributed to any change in confidence upon graduation for each task. All items were written as five-point Likert scales. Topic importance was scaled as 1 = “not at all important” to 5 = “extremely important.” Confidence was scaled as 1 = “extremely nervous” to 5 = “quite confident.” The perceived contribution of the course to skill level on graduation was scaled as 1 = “not helpful” to 5 = “completely responsible.”
As we used self-reported measurements of confidence and anxiety as markers of readiness for internship, we administered a Spielberger State-Trait Anxiety Inventory (STAI) questionnaire to all interns at our own institution during the first month of residency. This tool consists of two, 20-item scales designed to measure both state and trait anxiety levels.12,13 The purpose of this assessment was to determine whether there was an intrinsic difference in general anxiety levels among individuals who took the intern preparatory course compared with nonparticipants. This was used as a critical control in order to rule out any selection bias in terms of the intrinsic characteristics of those individuals who might be more likely to register for such a preparatory course.
We measured baseline knowledge in core curricular concepts as well as the impact of the course on knowledge levels using a 50-point multiple-choice written exam, administered both on the first and final day of the course. This exam covered material from nine representative domains, questioning the students on details of handling both everyday and emergent situations on the surgical ward.
We utilized an objective structured assessment of technical skills (OSATS) exam both pre- and postcourse to measure student mastery of specific technical skills, in order to obtain a baseline value as well as track improvement. The OSATS examination involved a realistic scenario not unlike a situation that might be faced by a typical intern on a busy surgical service. The participants were asked to perform several stepwise tasks on a simulated patient, requiring skills in antiseptic technique, suturing, placement of invasive monitoring catheters, and documentation of operative procedures. We graded participants using itemized checklists in addition to providing global scores in areas such as maintenance of a sterile field and smoothness of skill execution.
We gathered data regarding participants' performance in each of the 32 tasks on matriculation into surgical internship, seeking feedback from the senior residents supervising these individuals. Senior residents were used as evaluators rather than faculty members because we sought to obtain the most accurate evaluations possible, from those individuals who were directly supervising performance of these tasks. We purposely included variability in the breadth of the 32 tasks, with the recognition that highly detailed topic specification might decrease the likelihood of a supervising clinician's having witnessed execution of that task by the individual in question, while trying to avoid any topics that would be too broad to be meaningful. Performance evaluations were obtained at the completion of one month and two months of internship. Supervising residents were asked to rank the first-year resident's ability to perform each task relative to his or her peers using a five-point Likert scale, where 1 and 2 = “below peers,” 3 = “same as peers,” and 4 and 5 = “above peers.” They were also asked the extent to which they felt that the intern's confidence correlated with his or her ability level in each task, again using a Likert scale (1 and 2 = “overconfident;” 3 = “ability congruent with confidence;” 4 and 5 = “modest, unwarranted insecurity”). For those tasks in which the evaluator had not observed the performance of the particular trainee in question, the evaluator was instructed to indicate “not applicable” rather than providing a score on the Likert scale. We collected all follow-up data using scripted interview forms, and evaluators were blinded as to the participation status of the first-year residents in a surgery preparatory course.
Within our own institution, incoming residents who did not participate in an intern preparatory course served as controls (n = 12) for graduates of our course who had matriculated into our residency program (n = 10). Senior residents supervising all interns (both course graduates and controls) were identified, and performance evaluation was completed through live interviews. For those participants who graduated from our course and matriculated into residencies elsewhere (n = 12), the participants themselves provided contact information regarding their supervising residents, who were subsequently contacted for performance evaluation completion via telephone interviews. We asked all supervising evaluators from the outside institutions to provide performance assessments of the interns in question (n = 12); we also asked those evaluators to each assess two additional interns whom they supervised closely during the same time period, to serve as matched controls (n = 24) within the same institution.
Thus, our follow-up data included four main groups of individuals: (1) graduates of our preparatory course who matriculated into our own institution for residency, (2) first-year residents in our surgical training program who did not participate in our course or in any other intern preparatory course, (3) graduates of our course who entered residencies elsewhere, and (4) control first-year residents at outside institutions (see Figure 1 for a diagram describing the study participants).
We made comparisons between performance levels of course graduates and those of controls with regard to all 32 queried tasks, and we used t tests to assess significance. Subgroup analysis was employed to assess the performance of first-year residents at our own home institution versus elsewhere. Additionally, care was taken to identify any individuals who participated in preparatory courses at other medical schools who subsequently matriculated into internship at our institution, and to analyze this subgroup separately.
We also sought to assess the impact of improved preparation for surgical internship on individual compliance with the duty hours regulations set forth by the ACGME. Within our own institution, we reviewed duty hours reports for the months of July and August, evaluating violations of course participants versus those of nonparticipants. We assessed total weekly hours as well as number and type of duty hours violations. Intergroup differences were compared using t tests and Fisher's exact test.
We analyzed all items in terms of descriptive statistics and reported data as means with standard errors (SEs) of the means. For all statistical testing, significance was determined with α = 0.05. We performed statistical testing using SAS 9.1 (SAS Institute, Cary, North Carolina) and Microsoft Excel for Mac 14.1.2 (Microsoft Corporation, Redmond, Washington).
We observed a mean attendance rate of 96%, SE 1.4%, across sessions for the month. We achieved 100% data completion with regard to the pre- and postcourse surveys, knowledge tests, and performance examinations. Our response rate for performance assessments as the individuals matriculated into residency was 90% (response from 20 participants, with 2 graduates lost to follow-up). For the 2 individuals lost to follow-up, data from only the precourse and immediate postcourse time points were included in the analyses. Nonparticipant controls at our home institution were not significantly different from course graduates with regard to sex or surgical specialty designation.
When queried regarding their perceptions of the importance of specific tasks prior to course participation, students placed the greatest weight on skills necessary for ward management of surgical patients, particularly those needed for stabilization in the setting of cardiopulmonary decompensation. The topic receiving the singular highest score for importance was “handling emergent situations in postoperative patients,” with a mean rank of 4.91, SE 0.06. Also scoring highly were “cross coverage of other surgical services,” “management of common postoperative issues,” and “management of arrhythmias” with scores of 4.64 (SE 0.10), 4.55 (SE 0.13), and 4.45 (SE 0.13), respectively. Topics related to professionalism, communication, and personal balance were felt to be the least important, with “stress management/emotional shifting” receiving the lowest score (3.45, SE 0.26). Technical skills received intermediate rankings in this realm.
Students provided numeric assessments of their confidence levels in all 32 tasks at the beginning and conclusion of the preparatory course. Not surprisingly, they felt the least confident in many of the areas that they had deemed to be most important (see Table 2). “Handling emergent situations in postoperative patients” received the lowest precourse confidence score (1.55, SE 0.11). Importantly, this was also the topic in which the students demonstrated the greatest gain in confidence through course participation (postcourse 3.64, SE 0.18; gain of 2.09, SE 0.21), supporting the utility of preparatory courses in augmenting intern readiness for common as well as challenging clinical scenarios. We saw improvements in task-specific confidence in all 32 identified areas, with the smallest (and only nonsignificant) gains in the four professionalism-oriented topics. These were the areas in which the students expressed the greatest precourse confidence and, thus, had the least room for improvement.
When asked about the extent to which participation in the course contributed to confidence levels upon graduation, students indicated that instruction provided through the course was largely to completely responsible for reported task-specific confidence. This was most significant for the tasks for which the students started the course largely unprepared (e.g., “responding to pages,” 4.86, SE 0.07) and less true for those types of skills in which the students felt fairly comfortable prior to starting the course (“emotional shifting/stress management,” 3.18, SE 0.26).
Concerns could be raised that the results of measurements of confidence and anxiety in this group of individuals may be the reflection of a selection bias. In other words, one might suppose that students who choose to enroll in a residency preparatory course may intrinsically be more anxious, swaying the precourse confidence levels downward until those individuals receive thorough preparation. To rule out any selection bias in this regard, we administered the STAI to all new first-year surgical residents at our institution in July of the intern year. We found no significant difference between scores of course participants versus nonparticipants (participants 45.8, nonparticipants 43.0, P = .334).
Knowledge and skill levels
On the written exam, students demonstrated objective improvements in all nine representative knowledge domains from pre- to postcourse (see Table 3). We saw remarkable gains in the mean scores for the multiple-choice component (precourse, 58.5%, SE 2.2%; postcourse, 77.3%, SE 2.0%), order-writing scores (precourse, 50.4%, SE 2.8%; postcourse, 90.3%, SE 2.5%), and overall knowledge test scores (precourse, 54.8%, SE 2.1%; postcourse 82.2%, SE 1.9%). Each of these gains was statistically significant. To measure the reliability of this knowledge test, we calculated the Kuder-Richardson reliability coefficient, finding KR20 = 0.70, a satisfactory value for this type of knowledge assessment. Evaluation of technical skills showed equally noteworthy gains, with a mean precourse exam performance of 82.7%, SE 2.5%, and a mean postcourse score of 93.3%, SE 1.3%. This gain, too, was statistically significant.
Competency upon follow-up
Our initial follow-up assessment occurred at the end of July of the intern year of our course graduates (one month into residency). At this time point, we found that, for each task, an average of 76% of the evaluators (40 of 56) had obtained adequate observations to provide a Likert-based score. Analysis of these findings revealed a performance advantage among course participants compared with their peers in all 32 tasks (see Table 4). This advantage ranged from 0.02 to 1.03 and was significant in 16 (50%) of the tasks. (Course graduates additionally outperformed their peers in the other 16 tasks, though this could not be demonstrated to a level of statistical significance.) Subgroup analysis revealed no difference in performance after one month of residency among all individuals matriculating into residency at our own residency program versus elsewhere. Additionally, there was no advantage measurable when residents were categorized by designated surgical specialty, gender, or number of surgical electives in which the resident participated as a medical student.
Following two months of surgical internship, we found that course graduates no longer demonstrated a significant advantage in any of the 32 skills compared with their matched peers (see Table 5). For each task, an average of 82% of the evaluators (46 of 56) had obtained adequate observations in the month of August to provide a Likert-based score for the trainees under consideration. There was a trend toward better performance in 20 (62.5%) tasks, but the advantage gained through enrollment in the surgery preparatory course had clearly begun to dissipate as the residents progressed into their third month of training. These findings support the hypothesis that, through on-the-ground experience, underprepared trainees do eventually catch up to course graduates. Further, this would suggest that there was no intrinsic difference in the aptitude for clinical performance between individuals who chose to participate in such a course and those who do not; rather, that there was a marked and measurable benefit to targeted preparation that allows for optimization of training and assurance of excellent patient care during the midsummer months, when new residents begin.
Relationship between confidence and competence
We additionally investigated the correlation between confidence and competence, using two different assessments. We utilized the supervising residents' assessments of task-specific performance as our marker of competence in those tasks. We then compared those competency levels with participants' self-reports of task-specific confidence on course completion as well as the supervisors' perceptions of the trainees' confidence in those skills. Mean scores of self-reported confidence obtained in each task upon course completion were plotted versus the supervising residents' assessments of abilities in these areas at the end of July. (Both of these scores were defined using Likert scales, 1–5.) We found a linear relationship between reported confidence and assessed competence with nearly a 1:1 relationship (y = 0.9667x + 0.3776, R2 = 0.8835). Despite the linear relationship, we found that, for all tasks, course graduates were likely to slightly underassess their own abilities, a phenomenon that was more pronounced among the few outlying tasks that did not fit the curve.
We also asked the supervising residents to provide an assessment of the confidence–ability correlation in each task for all course graduates and controls, where low scores indicated overconfidence, high scores indicated unwarranted insecurity, and a score of 3 indicated a close correlation between ability and confidence. Mean scores for the confidence–ability correlation of all 32 tasks were tightly clustered around 3.0, with a range from 2.96, SE 0.03 to 3.06, SE 0.04.
Weekly duty hours were tracked for all interns based in the Department of Surgery, which included 8 course participants and 17 nonparticipants. Average total weekly hours between these two groups were not statistically different in July or August. However, we saw trends toward more frequent violations among nonparticipants in both July and August, with experience resulting in overall improvement in duty hours compliance among all individuals. Course graduates committed an average of 2.62, SE 0.13 violations in July (versus 3.13, SE 0.09 for nonparticipants) and 1.67, SE 0.10 in August (versus 1.73, SE 0.12 for nonparticipants). A greater proportion of nonparticipants violated several duty hours regulations over the first two months of internship. Nonparticipants committed more frequent 10-hour violations (60.0% versus 37.5% in July; 13.3% versus 12.5% in August) as well as 80-hour violations (80.0% versus 62.5% in both July and August). The 30-hour rule was violated regularly by nonparticipants (30.0% in July, 20.0% in August), whereas it was never violated by course participants (0%, P = .037 when compared with nonparticipants). As expected, these findings dissipated by the third month of residency.
We designed, implemented, and delivered a comprehensive, competency-driven course to SMSs entering surgical internships, intended to provide these individuals with the skills necessary for the transition from medical school to residency. We identified 32 specifically instructed skills, and we found that course participation improved student confidence in execution of these skills from the beginning to conclusion of the course. The students' performance on objective assessments of the skills improved in a classroom setting (as measured via written and technical skills exams), and, more importantly, these gains in confidence translated to gains in competence on matriculation into residency. Although reports from other authors have suggested the potential utility of surgical intern preparatory courses, this is the first study of its kind to provide evidence documenting the significant impact of such a course in improving subsequent intern performance in the hospital setting.
Fourth-year medical students receive minimal specific training for daily job-related tasks, and the training that they do receive varies widely among medical schools.2,14 Consequently, new surgical interns may be underprepared for acute postoperative patient management, relying to a great extent on “trial-by-fire” learning experiences on the job, with a significant learning curve taking place during the first few months of internship.1,14 This pattern may potentially result in high levels of intern anxiety, a greater number of adverse events in patient care, and greater difficulty with adherence to duty hours restrictions during the early part of internship due to lack of efficiency.
As new medical school graduates begin residency, they are faced with unending potential stressors: heightened personal responsibility for patient care, excessive work load, and sleep deprivation, in addition to personal pressures such as financial debt and changes in social support networks.15 There are ample data to support that residency remains a very stressful time for young physicians, particularly during the first year.16,17 In fact, numerous instruments measuring emotional stress have demonstrated an inverse relationship between extent of workplace stress and level of training, with interns on hospital inpatient services experiencing the highest extent of stress.18 Investigators have found that tension–anxiety scores in interns are greatest at orientation and begin to decline after the first month of intern year.19 Feelings of inadequate preparation for new responsibilities most certainly contribute to this pattern of emotional stress. Physiologic studies evaluating surgical residents have demonstrated elevation of physiologic parameters of stress in trainees who are on call, with the most profound effects seen among first-year residents.20 By focusing on preparation for the tasks and stressors encountered by surgical interns in daily patient care and during on-call shifts, an educational intervention has the potential to decrease stress levels and ease the transition into residency. Further, in addition to the positive generalized impact of thorough preparation on stress levels, we specifically included the concepts of stress management and emotional shifting as topics within our instructed curriculum. We believe that preparing medical school graduates for the duties that they will be expected to accomplish in their future job positions directly minimizes job-related stress; for those residual stressors that cannot be eliminated, we have given students the tools that they need to respond to difficult stimuli in a healthy manner, building their own resilience and learning from the challenging experiences.
In our increasingly litigious society, public attention to adverse events is at a high point, and preventable medical errors are, more than ever, deemed unacceptable. In one study, surveyed interns attributed prescribing errors to lack of knowledge, and the likelihood of such events correlated with earlier time points in the intern year and lack of specific training during schooling.21 Other authors have similarly shown increased incidence of adverse events earlier in residency, identifying the common themes of inadequate handoffs and insufficient training as the inciting factors leading to mistakes.22–24 A recent national consensus conference on patient safety recommended specific curricular components for surgical residency programs, directed at ensuring safe care delivery.25 Further, as we propose strategies to improve the safety and quality of care delivered by trainees, it is imperative that we acknowledge new interns as a key target of these efforts. It has long been suspected by health care insiders that the introduction of new interns each summer, making their first independent patient management decisions, may be a source of adverse events. Recently, this concept has gained attention both in the medical literature and popular media.26,27 In this social and political setting, the dangers of inadequately preparing our trainees for competent patient care from the first day of residency are palpable. As we face these societal pressures, which are further compounded by media scrutiny, it has become even more important for us to recognize the clear potential of residency preparatory courses to prevent adverse patient events and heighten patient safety. In fact, we can anticipate that the public will soon demand such educational reformation in our training institutions.
In our study, we demonstrated a clear performance benefit among course participants upon matriculation into residency. Our preparatory course graduates outperformed their peer interns in all 32 measured tasks when evaluated at the end of the first month of residency. Thus, we seem to have countered the “July Effect,” the concept that the quality of inpatient hospital care experiences a dip following the inauguration of new medical school graduates into the physician workforce.26,27 It is important to note that the performance advantage experienced by course graduates had essentially dissipated by the time that these individuals were compared with their peers at the end of August. At the onset of this study, we hypothesized that the day-to-day experience of residency provides both knowledge and skill acquisition, allowing those individuals who are not exposed to the preparatory course to attain competence and confidence commensurate with course participants, such that measured performance by members of these two groups would converge after a specific period of training.
In other words, we found that “trial-by-fire” education does work; it just takes time. This lends itself to an obvious critique: Why, then, do we bother with such a time-, resource-, and energy-consuming investment, if all trainees will end up in the same place eventually? We return to the key underlying concept: to provide consistent quality of care for our patients (i.e., to eliminate the “July Effect”26), trainees must be prepared from day one. Providing new surgical interns with the skills they need in order to provide excellent patient care and to carry out the daily tasks of their job is of paramount importance, and by providing them with these skills prior to medical school graduation, we ensure the safety of our patients year-round.
In considering the components of excellence in patient care, efficiency has not traditionally been a mark of distinction. However, in the modern training era, this is a concept that can no longer be ignored. As new interns struggle with attaining job-specific skills during their first few rotations, their performance efficiency may begin quite low, improving with ongoing experience. Noncompliance with the ACGME duty hours standards has been reported commonly among interns, and it has been documented that interns are subject to unique influences affecting such noncompliance.28,29 We believe that job-specific, advanced preparatory training may augment surgical intern efficiency, thereby potentially improving compliance with duty hours restrictions. In this study, we found that course participants were less likely to commit duty hours violations than were nonparticipants. In particular, we found the most significant difference in comparing the frequency of violations of the 30-hour rule, which we feel most directly reflects individual intern efficiency. The 80-hour and 10-hour rules are frequently affected by the overall team workload, team efficiency, and operative schedule. The 1-day-off-in-7 as well as the 80-hour rules are both significantly affected by call schedules. The 30-hour rule, however, usually reflects the individual's efficiency in completing a set number of tasks prior to leaving the hospital postcall.
We recognize that there are a number of confounding systematic issues that may contribute to an individual's duty hours compliance, as highlighted above. However, both course participants and nonparticipants were subject to the same systematic conditions. Despite potential limitations of this outcome measurement, these results are of particular interest in the setting of recent implementation of further restrictions to the hours worked by first-year residents.30 We found that we could improve adherence to duty hours regulations by heightening trainee readiness for daily job-related tasks, another substantial benefit to preparatory course participation.
In light of increasingly stringent duty hours restrictions for first-year residents, the findings from this study have become more relevant than ever. As educators, we have even less time to provide on-the-job instruction to our trainees. We suspect that these changes will result in an extended learning curve during the intern year for underprepared trainees, likely widening the gap between those individuals who have participated in preparatory courses and those who have not. We are in the process of collecting data from subsequent classes, and we are pleased to find that preliminary results appear quantitatively congruent with this study's findings (data not shown). However, as we follow these individuals into residency, it will be of key interest to observe whether the nonparticipants will take longer to catch up to the course graduates, in the setting of the most recent duty hours changes effective July 2011.
Courses specifically intended to prepare SMSs for surgical residency are in limited existence, but they are high in demand and expanding in number.31 Authors who have published descriptions of their previous experiences with preparatory courses have reported improved confidence and performance in execution of technical skills, as well as score increases on knowledge tests.7–10,32,33 Although these courses have been successful at laying a foundation for future curricular developments, their emphases have primarily been on procedural skills. Further, their impact has not been measured objectively and longitudinally as the medical student transitions into residency.
Previous attempts to improve surgical intern readiness through preparatory courses and “boot camps” should be applauded for their forward thinking and progressive approaches to curricular innovation. We believe that we can learn from these experiences and build on them to bridge the critical gap between medical school and residency. A three-week “Resident Readiness” elective conducted at the University of Southern California Keck School of Medicine covered only technical skills.8 A seven-session course at the Washington University in Saint Louis School of Medicine devoted a three-hour small-group discussion to ward management issues, yet this remained the area in which students felt the least prepared after completion of the course.10 A monthlong elective at Southern Illinois University School of Medicine included instruction on patient management that was ranked “most valuable” by the students in the course, yet it occupied only a small fraction of total course hours.7 From a curricular standpoint, our course is unique compared with those that have been previously described: although we teach key technical and procedural skills, we have placed a much greater emphasis on skills needed for management of routine as well as life-threatening events in surgical inpatients. The resulting course includes supervised practice opportunities for technical procedural skills and simulation-based team performance skills as contained in several of the aforementioned courses. Our curriculum, however, includes a unique package of interactive didactic sessions intended to help prepare students for daily tasks of inpatient management as well as emergent, life-threatening events commonly encountered in surgical care.
Typically, interns receive the greatest amount of direct faculty supervision during execution of procedural skills, with much less supervision during pre- and postoperative care management. The intent of our design was to prepare students not only for technical procedures but also for those tasks for which they may have the least guidance on starting internship. On review of specific tasks among the 32 for which we collected data, this concept pertains in particular to the task “handling emergent situations in postoperative patients.” This was the topic felt to be of greatest importance to the students, yet it caused them the most substantial anxiety and was the area in which they felt the least prepared prior to our course. We have previously shown the utility of high-fidelity, dynamic simulation to teach trainees to manage life-threatening events on the postoperative ward.3,11 We have found this to be an efficacious means of teaching this skillset at our own home institution and on transfer to other training programs as well.11 We continue to advocate for the inclusion of these types of emergency simulations in all intern preparatory courses.
As indicated above, “handling emergent situations in postoperative patients” was a topic that was particularly emphasized within our monthlong course. Although all 32 tasks were instructed to some extent, there existed some variability in the amount of time devoted to each of these topics because of their differing levels of importance to a surgical intern. In this regard, topics such as “ICU management” served as internal controls in our study, in that we would not expect our course graduates to portray as significant of a performance advantage over their peers in these areas. On examination of our data, we found this phenomenon to hold true. This finding contrasts substantially from the performance data regarding those topics that we felt to be key skills that our graduates needed early in residency.
Although the delivery of curricular content related to ward management tasks has been largely underaddressed in previous preparatory courses for surgical interns, it has not been entirely ignored. Surgical educators conducted a four-week course at the University of Texas Health Science Center at San Antonio, which included didactics on day-to-day patient management, consisting of several topics contained in our own curriculum.9 However, while the authors did demonstrate an improvement in participant confidence for a number of specific tasks, the study was limited in that (1) gains in confidence were not correlated with effect on performance, and (2) the study lacked a control group. Our study has been designed specifically to overcome these two limitations.
There has been a significant recent movement to reform U.S. medical education, as leaders in academic medicine attempt to respond to the current economic climate and expectations from accreditation bodies. Medical educators are encouraged to help transform our nation's health care delivery into a system focused on patient-centered care, quality improvement, and resource conservation.34 In these discussions, it has been acknowledged that the fourth year of medical school is commonly unstructured and in need of significant curricular reorganization.35–37 Residency program directors have recognized that the final year of medical school must play a critical role in preparing new interns, and they support curricular innovation in the fourth year to transition students to the needs of residency.38 In this setting, a course specifically designed to prepare graduating medical students for the daily tasks of surgical internship fits well into plans for curricular reform, as this identifies an important niche in which our efforts have the potential to make a significant impact. Further, the direct application of this concept to other specialties such as internal medicine, pediatrics, and obstetrics–gynecology is immediately translatable.
Our study is subject to some limitations. Although the supervising clinicians who rated the first-year residents' performance were blinded with regard to course participation, we could not control for the fact that some course graduates may have discussed their experiences during other interactions such as daily conversation. Additionally, as mentioned above, duty hours compliance is not a perfect surrogate for an individual's on-the-job efficiency. In considering limitations, one must also rule out any additional educational exposures—for example, capstone courses or preparatory workshops—that could have led to improved performance by the course participants and thus affected the outcomes. However, participants and controls were questioned about such potential exposures on multiple occasions in order to exclude the possibility of any additional influences on our results. In a very rigorous way, we queried and tracked all 62 study subjects to find out whether they had additional instruction prior to initiation of residency preparation. During subanalysis, there was no difference between these groups. In fact, if nonparticipants had additional training, this would have only narrowed the significance of our data.
With this study, we demonstrated that providing SMSs with specific instruction and practice in the daily tasks of surgical internship resulted in improved confidence and, more important, improved task-related performance compared with controls. To date, the ability of a fourth-year preparatory course to improve performance objectively has not yet been demonstrated in the published literature. Further, we demonstrated a clear correlation between self-perceived confidence and competence measured by others, a relationship that may prove highly useful for future investigators in this area.
The authors wish to thank Dr. Robert Sweet, Troy Reihsen, and John Raymond for their contributions on behalf of the Simulation PeriOperative Resource for Training and Learning (SimPORTAL) at the University of Minnesota. The authors are indebted to faculty, residents, and the Surgical Education Office at the University of Minnesota Department of Surgery for their support of this course. Additionally, the authors wish to thank faculty and residents of the Departments of Radiology and Anesthesiology at the University of Minnesota for their contributions. Portions of the research described in this paper were conducted as part of the Association for Surgical Education's Surgical Education Research Fellowship (SERF) Program (Dr. Kim Schenarts, SERF Director).