Skip Navigation LinksHome > March 2012 - Volume 87 - Issue 3 > The Significant Impact of a Competency-Based Preparatory Cou...
Academic Medicine:
doi: 10.1097/ACM.0b013e318244bc71
Medical Education

The Significant Impact of a Competency-Based Preparatory Course for Senior Medical Students Entering Surgical Residency

Antonoff, Mara B. MD; Swanson, Jennifer A. MD; Green, Courtney A. MS; Mann, Barry D. MD; Maddaus, Michael A. MD; D'Cunha, Jonathan MD, PhD

Free Access
Article Outline
Collapse Box

Author Information

Dr. Antonoff is chief resident in general surgery, Department of Surgery, University of Minnesota Medical School, Minneapolis, Minnesota.

Dr. Swanson is senior resident in general surgery, Department of Surgery, University of Minnesota Medical School, Minneapolis, Minnesota.

Ms. Green is a medical student, Department of Surgery, University of Minnesota Medical School, Minneapolis, Minnesota.

Dr. Mann is program director, Lankenau Surgical Residency Program, Department of Surgery, Lankenau Hospital, Wynnewood, Pennsylvania, and professor of surgery, Jefferson Medical College of Thomas Jefferson University, Philadelphia, Pennsylvania.

Dr. Maddaus is program director, General Surgery Residency Program, and Garamella Lynch Jensen Professor of Surgery, Division of Thoracic and Foregut Surgery, Department of Surgery, University of Minnesota Medical School, Minneapolis, Minnesota.

Dr. D'Cunha is associate program director, General Surgery Residency Program, associate program director, Thoracic Surgery Training Program, and assistant professor of surgery, Department of Surgery, University of Minnesota Medical School, Minneapolis, Minnesota.

Please see the end of this article for information about the authors.

Correspondence should be addressed to Dr. D'Cunha, 420 Delaware St, SE, Mayo Mail Code 207, Minneapolis, MN 55455; telephone: (612) 624-3277; fax: (612) 625-9657; e-mail: jdcunha@umn.edu.

First published online January 25, 2012

Collapse Box

Abstract

Purpose: Previous data suggest that formal, structured preparation might improve knowledge and skills of senior medical students (SMSs) as they transition to surgical residency. However, subsequent impact on clinical performance has not been demonstrated.

Method: The authors developed a comprehensive course for SMSs entering surgical residencies and studied the impact of the course on the subsequent performance of 2010 graduates (n = 22) compared with matched peers (16 nonparticipant controls at authors' home institution and 24 nonparticipant peer controls at outside institutions; total n = 62). Through pre- and postcourse surveys, knowledge tests, and technical examinations, they measured confidence and skill acquisition in 32 specific, job-related tasks. They followed participants and matched peers into internship and collected performance evaluations from supervising senior residents to determine whether course graduates would display performance advantages in these same tasks. The authors used t tests for all comparisons, α = 0.05.

Results: Participants demonstrated marked improvement in task-specific confidence in all 32 tasks from course beginning to end, with improved scores on written and technical skill examinations. Further, course participants outperformed peers in all 32 tasks in July, with their performance advantage predictably dissipating into the third month of residency. There was a marked correlation between confidence and competence in all tasks.

Conclusions: Competency-based preparation for surgical internship resulted in objective gains in task-specific confidence and test performance at course conclusion, translating to improved performance and better patient care upon residency matriculation. These data emphasize the significant impact of formally preparing SMSs before graduation.

On a daily basis, surgical interns encounter complex patients in acute care situations, which they may be independently managing for the first time in their medical careers. This transition to independence poses a significant dilemma, emphasizing the challenging balance between the educational benefits of graded, autonomous patient care and the indisputable necessity of patient safety. Further, because of the nature of the surgical service and its hierarchy, senior residents tend to accompany surgical faculty into the operating room during the day, while interns provide immediate, “hands-on” representation of the care teams on the surgical ward. In addition, the enforcement of work hour restrictions has significantly decreased the number of surgical residents who are in-house at nights and on weekends, resulting in a greater sense of isolation for the surgical intern on call.1 These issues result in a significant problem that is not unique to surgical training; important features of this dilemma resonate throughout graduate medical education.

To provide timely, safe, and effective patient care, interns need to be prepared from day one for the daily tasks that they will encounter, both on their own services and in cross-coverage. Although we postulate that most interns will ultimately assimilate the necessary skills and confidence to perform adequately at some time during their first year of residency, their position requires them to begin July with competence and confidence sufficient to be first-responders to serious problems, attributes that are only questionably attainable in the current medical school setting. Fourth-year medical students are exposed to widely differing educational experiences, with no standardized formal preparation for internship.2 This poses a significant dilemma to patient care and identifies an obvious gap in surgical education.

To address this need, in 2008, we, as members of the Division of Education in the Department of Surgery at the University of Minnesota Medical School, developed a four-week, competency-based course for senior medical students (SMSs) entering surgical specialties, specifically intended to prepare them for matriculation into internship. This course consisted of interactive didactics, simulation of emergencies on the surgical ward, and procedural instruction with skills practice. The objective of the course was to prepare students for the tasks commonly encountered in surgical internship, including response to emergent events as well as daily patient care responsibilities. The need for this type of preparatory course has been recognized nationally, as evidenced by the recent establishment of the Entering Surgery Resident Preparation Curriculum Committee, a joint endeavor of the American College of Surgeons, the Association of Program Directors in Surgery, and the Association for Surgical Education.3 Unfortunately, existing courses are in their infancies, are limited in number, and tend to focus primarily on the acquisition of technical (rather than cognitive) skills, and their outcomes as the trainees matriculate into residency have yet to be rigorously evaluated.

Initiatives directed toward formalizing preparation for surgical internship are gaining momentum, as the importance of improving interns' readiness is largely well accepted and intuitively makes sense. However, the task of demonstrating the impact of such courses remains to be addressed. The intent of the present study was to analyze the outcomes of the first year of our educational intervention (which continues today), with the ultimate goal of furthering national as well as local curriculum development and evaluation efforts. In this study, we hypothesized

* that participation in a four-week comprehensive preparatory course for SMSs entering surgical residencies could improve trainee confidence levels in executing specific job-related tasks of surgical internship;

* that such confidence gains would translate to improved performance in these specific tasks when course participants were compared with appropriate controls; and

* that the internship experience itself allows those not exposed to the course to attain competence and confidence commensurate with those exposed to our preparatory course, such that measured differences in competence and confidence would dissipate after a specific period of training.

With this report, we highlight the utility of our novel curriculum while further correlating trainee task-specific confidence to task-specific performance. In this respect, we believe our findings provide insight into the validity of study instruments that measure confidence levels, a relationship that may be useful to future researchers investigating the readiness of SMSs for residency training.

Back to Top | Article Outline

Method

Participants and course structure

We developed a four-week comprehensive surgery preparatory course for incoming first-year residents and, in 2010, first offered it to SMSs matched into residencies in surgical disciplines. A total of 22 SMSs enrolled in the course. This group was composed of 4 individuals who had matched into residency positions in general surgery (18.2%), 5 in urology (22.7%), 4 in orthopedic surgery (18.2%), 2 in anesthesiology (9.1%), 1 in neurosurgery (4.6%), 1 in integrated plastic surgery (4.6%), and 1 in obstetrics–gynecology (4.6%), as well as 4 (18.2%) nondesignated preliminary surgical interns. Ten (45.5%) individuals later became residents at our home institution, and 17 (77.2%) were men. Students enrolled voluntarily, through standard registration practices at our medical school, the University of Minnesota Medical School. This new elective course was offered on a pass–fail basis, taking place as half-days, Monday through Friday, during April of the final year of medical school.

Instruction for the course was delivered via several techniques. Strategies employed included interactive didactics, moderated small-group discussions, team-based problem solving, dynamic simulations of postoperative emergencies, instructed technical skills practice, and mock page exercises.4 The mock page program consisted of a validated set of simulated pages from trained clinical nurse educators, presenting the students with common postoperative calls, followed by immediate one-on-one feedback and weekly classroom-based debriefings. Learning objectives for the course were generated by a committee consisting of clinical surgical faculty members and residents, with input from our previously established internal curriculum for first-year residents.5 Special attention was directed toward structuring the objectives on a foundation of the core competencies as defined by the Accreditation Council for Graduate Medical Education (ACGME). These objectives were refined and expanded upon review of proposed guidelines outlining educational objectives for trainees in their first year of surgical residency,6 as well as the published experiences of outside institutions.710 Review of course learning objectives revealed 32 key specific tasks in which the students were instructed.

In all, the competency-based course was comprised of 55 sessions. These included 33 interactive lectures, covering a breadth of topics, ranging from daily tasks such as preoperative assessment and electrolyte replacement to the intense management of situations such as the evaluation of the unstable trauma patient. We allotted time for several small-group discussions regarding topics such as professionalism, communication, ethics, and self-directed adult learning strategies. Participants attended four technical skills practice sessions, using both high- and low-fidelity models in our simulation center, to receive instruction and have opportunities for proctored skills practice in several areas, including airway management, suturing and knot tying, tube thoracostomy, placement of invasive monitoring catheters, and endoscopy and bronchoscopy. Our course included three sessions specifically devoted to the Critical Skills curriculum, a previously validated curriculum developed within our department to teach trainees to competently manage emergent postoperative events.3,11 These unique sessions employed one day of classroom-based software simulation, followed by two days of team-based interactive, dynamic case simulations (one observed case, five practice cases, performance assessment, and debriefing). We also incorporated four “Meet-the-Resident” forums, allowing the students to engage in question-and-answer sessions with current individuals at all levels of surgical training, to seek their guidance as well as their insight regarding expectations and keys to success. Our students participated in a multiinstitutional mock-page program, which included four mock-page debriefings.4 Our sessions also included three order-writing exercises and more than 20 sample case discussions.

Instructors for the course consisted primarily of clinical surgical faculty members, representing key divisions from the Department of Surgery, with guest lectures additionally delivered by expert faculty members from other university departments, including anesthesiology, radiology, hematopathology, and integrative care medicine. Surgical residents participated as moderators for small-group discussions and simulation instruction where appropriate.

Back to Top | Article Outline
Data collection

We obtained institutional review board approval for this study prior to any data collection, and participants signed all required consent forms. We obtained data at four separate time points: (1) precourse, (2) immediately postcourse, (3) one-month follow-up, and (4) two-month follow-up. An overview of all assessment tools and outcome variables is shown in Table 1. Data collection took place between April and September of 2010.

Table 1
Table 1
Image Tools
Back to Top | Article Outline
Measures of improvement

Data obtained from course participants at the first two time points (precourse; immediately postcourse) included surveys, knowledge tests, and performance examinations, which we administered on the first and last days of the class. We subsequently tracked participants as well as the matched peers into their third month of residency, obtaining performance reviews from supervising clinicians at one-month (end of July) and two-month (end of August) points of follow-up.

To determine whether students' task-specific confidence improved through participation in the course, we asked participants to complete self-assessment surveys on both the first and final days of the class. On the precourse survey, for each of the 32 designated tasks, students ranked their own abilities to carry out the tasks. Participants also scored their own perceptions of the importance of each of those tasks. On the last day of the class, students again provided self-assessments of their own abilities on all 32 tasks. Additionally, we asked them to rank the extent to which the course contributed to any change in confidence upon graduation for each task. All items were written as five-point Likert scales. Topic importance was scaled as 1 = “not at all important” to 5 = “extremely important.” Confidence was scaled as 1 = “extremely nervous” to 5 = “quite confident.” The perceived contribution of the course to skill level on graduation was scaled as 1 = “not helpful” to 5 = “completely responsible.”

As we used self-reported measurements of confidence and anxiety as markers of readiness for internship, we administered a Spielberger State-Trait Anxiety Inventory (STAI) questionnaire to all interns at our own institution during the first month of residency. This tool consists of two, 20-item scales designed to measure both state and trait anxiety levels.12,13 The purpose of this assessment was to determine whether there was an intrinsic difference in general anxiety levels among individuals who took the intern preparatory course compared with nonparticipants. This was used as a critical control in order to rule out any selection bias in terms of the intrinsic characteristics of those individuals who might be more likely to register for such a preparatory course.

We measured baseline knowledge in core curricular concepts as well as the impact of the course on knowledge levels using a 50-point multiple-choice written exam, administered both on the first and final day of the course. This exam covered material from nine representative domains, questioning the students on details of handling both everyday and emergent situations on the surgical ward.

We utilized an objective structured assessment of technical skills (OSATS) exam both pre- and postcourse to measure student mastery of specific technical skills, in order to obtain a baseline value as well as track improvement. The OSATS examination involved a realistic scenario not unlike a situation that might be faced by a typical intern on a busy surgical service. The participants were asked to perform several stepwise tasks on a simulated patient, requiring skills in antiseptic technique, suturing, placement of invasive monitoring catheters, and documentation of operative procedures. We graded participants using itemized checklists in addition to providing global scores in areas such as maintenance of a sterile field and smoothness of skill execution.

Back to Top | Article Outline
Postcourse follow-up

We gathered data regarding participants' performance in each of the 32 tasks on matriculation into surgical internship, seeking feedback from the senior residents supervising these individuals. Senior residents were used as evaluators rather than faculty members because we sought to obtain the most accurate evaluations possible, from those individuals who were directly supervising performance of these tasks. We purposely included variability in the breadth of the 32 tasks, with the recognition that highly detailed topic specification might decrease the likelihood of a supervising clinician's having witnessed execution of that task by the individual in question, while trying to avoid any topics that would be too broad to be meaningful. Performance evaluations were obtained at the completion of one month and two months of internship. Supervising residents were asked to rank the first-year resident's ability to perform each task relative to his or her peers using a five-point Likert scale, where 1 and 2 = “below peers,” 3 = “same as peers,” and 4 and 5 = “above peers.” They were also asked the extent to which they felt that the intern's confidence correlated with his or her ability level in each task, again using a Likert scale (1 and 2 = “overconfident;” 3 = “ability congruent with confidence;” 4 and 5 = “modest, unwarranted insecurity”). For those tasks in which the evaluator had not observed the performance of the particular trainee in question, the evaluator was instructed to indicate “not applicable” rather than providing a score on the Likert scale. We collected all follow-up data using scripted interview forms, and evaluators were blinded as to the participation status of the first-year residents in a surgery preparatory course.

Within our own institution, incoming residents who did not participate in an intern preparatory course served as controls (n = 12) for graduates of our course who had matriculated into our residency program (n = 10). Senior residents supervising all interns (both course graduates and controls) were identified, and performance evaluation was completed through live interviews. For those participants who graduated from our course and matriculated into residencies elsewhere (n = 12), the participants themselves provided contact information regarding their supervising residents, who were subsequently contacted for performance evaluation completion via telephone interviews. We asked all supervising evaluators from the outside institutions to provide performance assessments of the interns in question (n = 12); we also asked those evaluators to each assess two additional interns whom they supervised closely during the same time period, to serve as matched controls (n = 24) within the same institution.

Thus, our follow-up data included four main groups of individuals: (1) graduates of our preparatory course who matriculated into our own institution for residency, (2) first-year residents in our surgical training program who did not participate in our course or in any other intern preparatory course, (3) graduates of our course who entered residencies elsewhere, and (4) control first-year residents at outside institutions (see Figure 1 for a diagram describing the study participants).

Figure 1
Figure 1
Image Tools

We made comparisons between performance levels of course graduates and those of controls with regard to all 32 queried tasks, and we used t tests to assess significance. Subgroup analysis was employed to assess the performance of first-year residents at our own home institution versus elsewhere. Additionally, care was taken to identify any individuals who participated in preparatory courses at other medical schools who subsequently matriculated into internship at our institution, and to analyze this subgroup separately.

We also sought to assess the impact of improved preparation for surgical internship on individual compliance with the duty hours regulations set forth by the ACGME. Within our own institution, we reviewed duty hours reports for the months of July and August, evaluating violations of course participants versus those of nonparticipants. We assessed total weekly hours as well as number and type of duty hours violations. Intergroup differences were compared using t tests and Fisher's exact test.

Back to Top | Article Outline
Statistical analysis

We analyzed all items in terms of descriptive statistics and reported data as means with standard errors (SEs) of the means. For all statistical testing, significance was determined with α = 0.05. We performed statistical testing using SAS 9.1 (SAS Institute, Cary, North Carolina) and Microsoft Excel for Mac 14.1.2 (Microsoft Corporation, Redmond, Washington).

Back to Top | Article Outline

Results

Participants

We observed a mean attendance rate of 96%, SE 1.4%, across sessions for the month. We achieved 100% data completion with regard to the pre- and postcourse surveys, knowledge tests, and performance examinations. Our response rate for performance assessments as the individuals matriculated into residency was 90% (response from 20 participants, with 2 graduates lost to follow-up). For the 2 individuals lost to follow-up, data from only the precourse and immediate postcourse time points were included in the analyses. Nonparticipant controls at our home institution were not significantly different from course graduates with regard to sex or surgical specialty designation.

Back to Top | Article Outline
Topic importance

When queried regarding their perceptions of the importance of specific tasks prior to course participation, students placed the greatest weight on skills necessary for ward management of surgical patients, particularly those needed for stabilization in the setting of cardiopulmonary decompensation. The topic receiving the singular highest score for importance was “handling emergent situations in postoperative patients,” with a mean rank of 4.91, SE 0.06. Also scoring highly were “cross coverage of other surgical services,” “management of common postoperative issues,” and “management of arrhythmias” with scores of 4.64 (SE 0.10), 4.55 (SE 0.13), and 4.45 (SE 0.13), respectively. Topics related to professionalism, communication, and personal balance were felt to be the least important, with “stress management/emotional shifting” receiving the lowest score (3.45, SE 0.26). Technical skills received intermediate rankings in this realm.

Back to Top | Article Outline
Students' confidence

Students provided numeric assessments of their confidence levels in all 32 tasks at the beginning and conclusion of the preparatory course. Not surprisingly, they felt the least confident in many of the areas that they had deemed to be most important (see Table 2). “Handling emergent situations in postoperative patients” received the lowest precourse confidence score (1.55, SE 0.11). Importantly, this was also the topic in which the students demonstrated the greatest gain in confidence through course participation (postcourse 3.64, SE 0.18; gain of 2.09, SE 0.21), supporting the utility of preparatory courses in augmenting intern readiness for common as well as challenging clinical scenarios. We saw improvements in task-specific confidence in all 32 identified areas, with the smallest (and only nonsignificant) gains in the four professionalism-oriented topics. These were the areas in which the students expressed the greatest precourse confidence and, thus, had the least room for improvement.

Table 2
Table 2
Image Tools

When asked about the extent to which participation in the course contributed to confidence levels upon graduation, students indicated that instruction provided through the course was largely to completely responsible for reported task-specific confidence. This was most significant for the tasks for which the students started the course largely unprepared (e.g., “responding to pages,” 4.86, SE 0.07) and less true for those types of skills in which the students felt fairly comfortable prior to starting the course (“emotional shifting/stress management,” 3.18, SE 0.26).

Concerns could be raised that the results of measurements of confidence and anxiety in this group of individuals may be the reflection of a selection bias. In other words, one might suppose that students who choose to enroll in a residency preparatory course may intrinsically be more anxious, swaying the precourse confidence levels downward until those individuals receive thorough preparation. To rule out any selection bias in this regard, we administered the STAI to all new first-year surgical residents at our institution in July of the intern year. We found no significant difference between scores of course participants versus nonparticipants (participants 45.8, nonparticipants 43.0, P = .334).

Back to Top | Article Outline
Knowledge and skill levels

On the written exam, students demonstrated objective improvements in all nine representative knowledge domains from pre- to postcourse (see Table 3). We saw remarkable gains in the mean scores for the multiple-choice component (precourse, 58.5%, SE 2.2%; postcourse, 77.3%, SE 2.0%), order-writing scores (precourse, 50.4%, SE 2.8%; postcourse, 90.3%, SE 2.5%), and overall knowledge test scores (precourse, 54.8%, SE 2.1%; postcourse 82.2%, SE 1.9%). Each of these gains was statistically significant. To measure the reliability of this knowledge test, we calculated the Kuder-Richardson reliability coefficient, finding KR20 = 0.70, a satisfactory value for this type of knowledge assessment. Evaluation of technical skills showed equally noteworthy gains, with a mean precourse exam performance of 82.7%, SE 2.5%, and a mean postcourse score of 93.3%, SE 1.3%. This gain, too, was statistically significant.

Table 3
Table 3
Image Tools
Back to Top | Article Outline
Competency upon follow-up

Our initial follow-up assessment occurred at the end of July of the intern year of our course graduates (one month into residency). At this time point, we found that, for each task, an average of 76% of the evaluators (40 of 56) had obtained adequate observations to provide a Likert-based score. Analysis of these findings revealed a performance advantage among course participants compared with their peers in all 32 tasks (see Table 4). This advantage ranged from 0.02 to 1.03 and was significant in 16 (50%) of the tasks. (Course graduates additionally outperformed their peers in the other 16 tasks, though this could not be demonstrated to a level of statistical significance.) Subgroup analysis revealed no difference in performance after one month of residency among all individuals matriculating into residency at our own residency program versus elsewhere. Additionally, there was no advantage measurable when residents were categorized by designated surgical specialty, gender, or number of surgical electives in which the resident participated as a medical student.

Table 4
Table 4
Image Tools

Following two months of surgical internship, we found that course graduates no longer demonstrated a significant advantage in any of the 32 skills compared with their matched peers (see Table 5). For each task, an average of 82% of the evaluators (46 of 56) had obtained adequate observations in the month of August to provide a Likert-based score for the trainees under consideration. There was a trend toward better performance in 20 (62.5%) tasks, but the advantage gained through enrollment in the surgery preparatory course had clearly begun to dissipate as the residents progressed into their third month of training. These findings support the hypothesis that, through on-the-ground experience, underprepared trainees do eventually catch up to course graduates. Further, this would suggest that there was no intrinsic difference in the aptitude for clinical performance between individuals who chose to participate in such a course and those who do not; rather, that there was a marked and measurable benefit to targeted preparation that allows for optimization of training and assurance of excellent patient care during the midsummer months, when new residents begin.

Table 5
Table 5
Image Tools
Back to Top | Article Outline
Relationship between confidence and competence

We additionally investigated the correlation between confidence and competence, using two different assessments. We utilized the supervising residents' assessments of task-specific performance as our marker of competence in those tasks. We then compared those competency levels with participants' self-reports of task-specific confidence on course completion as well as the supervisors' perceptions of the trainees' confidence in those skills. Mean scores of self-reported confidence obtained in each task upon course completion were plotted versus the supervising residents' assessments of abilities in these areas at the end of July. (Both of these scores were defined using Likert scales, 1–5.) We found a linear relationship between reported confidence and assessed competence with nearly a 1:1 relationship (y = 0.9667x + 0.3776, R2 = 0.8835). Despite the linear relationship, we found that, for all tasks, course graduates were likely to slightly underassess their own abilities, a phenomenon that was more pronounced among the few outlying tasks that did not fit the curve.

We also asked the supervising residents to provide an assessment of the confidence–ability correlation in each task for all course graduates and controls, where low scores indicated overconfidence, high scores indicated unwarranted insecurity, and a score of 3 indicated a close correlation between ability and confidence. Mean scores for the confidence–ability correlation of all 32 tasks were tightly clustered around 3.0, with a range from 2.96, SE 0.03 to 3.06, SE 0.04.

Back to Top | Article Outline
Duty hours

Weekly duty hours were tracked for all interns based in the Department of Surgery, which included 8 course participants and 17 nonparticipants. Average total weekly hours between these two groups were not statistically different in July or August. However, we saw trends toward more frequent violations among nonparticipants in both July and August, with experience resulting in overall improvement in duty hours compliance among all individuals. Course graduates committed an average of 2.62, SE 0.13 violations in July (versus 3.13, SE 0.09 for nonparticipants) and 1.67, SE 0.10 in August (versus 1.73, SE 0.12 for nonparticipants). A greater proportion of nonparticipants violated several duty hours regulations over the first two months of internship. Nonparticipants committed more frequent 10-hour violations (60.0% versus 37.5% in July; 13.3% versus 12.5% in August) as well as 80-hour violations (80.0% versus 62.5% in both July and August). The 30-hour rule was violated regularly by nonparticipants (30.0% in July, 20.0% in August), whereas it was never violated by course participants (0%, P = .037 when compared with nonparticipants). As expected, these findings dissipated by the third month of residency.

Back to Top | Article Outline

Discussion

We designed, implemented, and delivered a comprehensive, competency-driven course to SMSs entering surgical internships, intended to provide these individuals with the skills necessary for the transition from medical school to residency. We identified 32 specifically instructed skills, and we found that course participation improved student confidence in execution of these skills from the beginning to conclusion of the course. The students' performance on objective assessments of the skills improved in a classroom setting (as measured via written and technical skills exams), and, more importantly, these gains in confidence translated to gains in competence on matriculation into residency. Although reports from other authors have suggested the potential utility of surgical intern preparatory courses, this is the first study of its kind to provide evidence documenting the significant impact of such a course in improving subsequent intern performance in the hospital setting.

Fourth-year medical students receive minimal specific training for daily job-related tasks, and the training that they do receive varies widely among medical schools.2,14 Consequently, new surgical interns may be underprepared for acute postoperative patient management, relying to a great extent on “trial-by-fire” learning experiences on the job, with a significant learning curve taking place during the first few months of internship.1,14 This pattern may potentially result in high levels of intern anxiety, a greater number of adverse events in patient care, and greater difficulty with adherence to duty hours restrictions during the early part of internship due to lack of efficiency.

As new medical school graduates begin residency, they are faced with unending potential stressors: heightened personal responsibility for patient care, excessive work load, and sleep deprivation, in addition to personal pressures such as financial debt and changes in social support networks.15 There are ample data to support that residency remains a very stressful time for young physicians, particularly during the first year.16,17 In fact, numerous instruments measuring emotional stress have demonstrated an inverse relationship between extent of workplace stress and level of training, with interns on hospital inpatient services experiencing the highest extent of stress.18 Investigators have found that tension–anxiety scores in interns are greatest at orientation and begin to decline after the first month of intern year.19 Feelings of inadequate preparation for new responsibilities most certainly contribute to this pattern of emotional stress. Physiologic studies evaluating surgical residents have demonstrated elevation of physiologic parameters of stress in trainees who are on call, with the most profound effects seen among first-year residents.20 By focusing on preparation for the tasks and stressors encountered by surgical interns in daily patient care and during on-call shifts, an educational intervention has the potential to decrease stress levels and ease the transition into residency. Further, in addition to the positive generalized impact of thorough preparation on stress levels, we specifically included the concepts of stress management and emotional shifting as topics within our instructed curriculum. We believe that preparing medical school graduates for the duties that they will be expected to accomplish in their future job positions directly minimizes job-related stress; for those residual stressors that cannot be eliminated, we have given students the tools that they need to respond to difficult stimuli in a healthy manner, building their own resilience and learning from the challenging experiences.

In our increasingly litigious society, public attention to adverse events is at a high point, and preventable medical errors are, more than ever, deemed unacceptable. In one study, surveyed interns attributed prescribing errors to lack of knowledge, and the likelihood of such events correlated with earlier time points in the intern year and lack of specific training during schooling.21 Other authors have similarly shown increased incidence of adverse events earlier in residency, identifying the common themes of inadequate handoffs and insufficient training as the inciting factors leading to mistakes.2224 A recent national consensus conference on patient safety recommended specific curricular components for surgical residency programs, directed at ensuring safe care delivery.25 Further, as we propose strategies to improve the safety and quality of care delivered by trainees, it is imperative that we acknowledge new interns as a key target of these efforts. It has long been suspected by health care insiders that the introduction of new interns each summer, making their first independent patient management decisions, may be a source of adverse events. Recently, this concept has gained attention both in the medical literature and popular media.26,27 In this social and political setting, the dangers of inadequately preparing our trainees for competent patient care from the first day of residency are palpable. As we face these societal pressures, which are further compounded by media scrutiny, it has become even more important for us to recognize the clear potential of residency preparatory courses to prevent adverse patient events and heighten patient safety. In fact, we can anticipate that the public will soon demand such educational reformation in our training institutions.

In our study, we demonstrated a clear performance benefit among course participants upon matriculation into residency. Our preparatory course graduates outperformed their peer interns in all 32 measured tasks when evaluated at the end of the first month of residency. Thus, we seem to have countered the “July Effect,” the concept that the quality of inpatient hospital care experiences a dip following the inauguration of new medical school graduates into the physician workforce.26,27 It is important to note that the performance advantage experienced by course graduates had essentially dissipated by the time that these individuals were compared with their peers at the end of August. At the onset of this study, we hypothesized that the day-to-day experience of residency provides both knowledge and skill acquisition, allowing those individuals who are not exposed to the preparatory course to attain competence and confidence commensurate with course participants, such that measured performance by members of these two groups would converge after a specific period of training.

In other words, we found that “trial-by-fire” education does work; it just takes time. This lends itself to an obvious critique: Why, then, do we bother with such a time-, resource-, and energy-consuming investment, if all trainees will end up in the same place eventually? We return to the key underlying concept: to provide consistent quality of care for our patients (i.e., to eliminate the “July Effect”26), trainees must be prepared from day one. Providing new surgical interns with the skills they need in order to provide excellent patient care and to carry out the daily tasks of their job is of paramount importance, and by providing them with these skills prior to medical school graduation, we ensure the safety of our patients year-round.

In considering the components of excellence in patient care, efficiency has not traditionally been a mark of distinction. However, in the modern training era, this is a concept that can no longer be ignored. As new interns struggle with attaining job-specific skills during their first few rotations, their performance efficiency may begin quite low, improving with ongoing experience. Noncompliance with the ACGME duty hours standards has been reported commonly among interns, and it has been documented that interns are subject to unique influences affecting such noncompliance.28,29 We believe that job-specific, advanced preparatory training may augment surgical intern efficiency, thereby potentially improving compliance with duty hours restrictions. In this study, we found that course participants were less likely to commit duty hours violations than were nonparticipants. In particular, we found the most significant difference in comparing the frequency of violations of the 30-hour rule, which we feel most directly reflects individual intern efficiency. The 80-hour and 10-hour rules are frequently affected by the overall team workload, team efficiency, and operative schedule. The 1-day-off-in-7 as well as the 80-hour rules are both significantly affected by call schedules. The 30-hour rule, however, usually reflects the individual's efficiency in completing a set number of tasks prior to leaving the hospital postcall.

We recognize that there are a number of confounding systematic issues that may contribute to an individual's duty hours compliance, as highlighted above. However, both course participants and nonparticipants were subject to the same systematic conditions. Despite potential limitations of this outcome measurement, these results are of particular interest in the setting of recent implementation of further restrictions to the hours worked by first-year residents.30 We found that we could improve adherence to duty hours regulations by heightening trainee readiness for daily job-related tasks, another substantial benefit to preparatory course participation.

In light of increasingly stringent duty hours restrictions for first-year residents, the findings from this study have become more relevant than ever. As educators, we have even less time to provide on-the-job instruction to our trainees. We suspect that these changes will result in an extended learning curve during the intern year for underprepared trainees, likely widening the gap between those individuals who have participated in preparatory courses and those who have not. We are in the process of collecting data from subsequent classes, and we are pleased to find that preliminary results appear quantitatively congruent with this study's findings (data not shown). However, as we follow these individuals into residency, it will be of key interest to observe whether the nonparticipants will take longer to catch up to the course graduates, in the setting of the most recent duty hours changes effective July 2011.

Courses specifically intended to prepare SMSs for surgical residency are in limited existence, but they are high in demand and expanding in number.31 Authors who have published descriptions of their previous experiences with preparatory courses have reported improved confidence and performance in execution of technical skills, as well as score increases on knowledge tests.710,32,33 Although these courses have been successful at laying a foundation for future curricular developments, their emphases have primarily been on procedural skills. Further, their impact has not been measured objectively and longitudinally as the medical student transitions into residency.

Previous attempts to improve surgical intern readiness through preparatory courses and “boot camps” should be applauded for their forward thinking and progressive approaches to curricular innovation. We believe that we can learn from these experiences and build on them to bridge the critical gap between medical school and residency. A three-week “Resident Readiness” elective conducted at the University of Southern California Keck School of Medicine covered only technical skills.8 A seven-session course at the Washington University in Saint Louis School of Medicine devoted a three-hour small-group discussion to ward management issues, yet this remained the area in which students felt the least prepared after completion of the course.10 A monthlong elective at Southern Illinois University School of Medicine included instruction on patient management that was ranked “most valuable” by the students in the course, yet it occupied only a small fraction of total course hours.7 From a curricular standpoint, our course is unique compared with those that have been previously described: although we teach key technical and procedural skills, we have placed a much greater emphasis on skills needed for management of routine as well as life-threatening events in surgical inpatients. The resulting course includes supervised practice opportunities for technical procedural skills and simulation-based team performance skills as contained in several of the aforementioned courses. Our curriculum, however, includes a unique package of interactive didactic sessions intended to help prepare students for daily tasks of inpatient management as well as emergent, life-threatening events commonly encountered in surgical care.

Typically, interns receive the greatest amount of direct faculty supervision during execution of procedural skills, with much less supervision during pre- and postoperative care management. The intent of our design was to prepare students not only for technical procedures but also for those tasks for which they may have the least guidance on starting internship. On review of specific tasks among the 32 for which we collected data, this concept pertains in particular to the task “handling emergent situations in postoperative patients.” This was the topic felt to be of greatest importance to the students, yet it caused them the most substantial anxiety and was the area in which they felt the least prepared prior to our course. We have previously shown the utility of high-fidelity, dynamic simulation to teach trainees to manage life-threatening events on the postoperative ward.3,11 We have found this to be an efficacious means of teaching this skillset at our own home institution and on transfer to other training programs as well.11 We continue to advocate for the inclusion of these types of emergency simulations in all intern preparatory courses.

As indicated above, “handling emergent situations in postoperative patients” was a topic that was particularly emphasized within our monthlong course. Although all 32 tasks were instructed to some extent, there existed some variability in the amount of time devoted to each of these topics because of their differing levels of importance to a surgical intern. In this regard, topics such as “ICU management” served as internal controls in our study, in that we would not expect our course graduates to portray as significant of a performance advantage over their peers in these areas. On examination of our data, we found this phenomenon to hold true. This finding contrasts substantially from the performance data regarding those topics that we felt to be key skills that our graduates needed early in residency.

Although the delivery of curricular content related to ward management tasks has been largely underaddressed in previous preparatory courses for surgical interns, it has not been entirely ignored. Surgical educators conducted a four-week course at the University of Texas Health Science Center at San Antonio, which included didactics on day-to-day patient management, consisting of several topics contained in our own curriculum.9 However, while the authors did demonstrate an improvement in participant confidence for a number of specific tasks, the study was limited in that (1) gains in confidence were not correlated with effect on performance, and (2) the study lacked a control group. Our study has been designed specifically to overcome these two limitations.

There has been a significant recent movement to reform U.S. medical education, as leaders in academic medicine attempt to respond to the current economic climate and expectations from accreditation bodies. Medical educators are encouraged to help transform our nation's health care delivery into a system focused on patient-centered care, quality improvement, and resource conservation.34 In these discussions, it has been acknowledged that the fourth year of medical school is commonly unstructured and in need of significant curricular reorganization.3537 Residency program directors have recognized that the final year of medical school must play a critical role in preparing new interns, and they support curricular innovation in the fourth year to transition students to the needs of residency.38 In this setting, a course specifically designed to prepare graduating medical students for the daily tasks of surgical internship fits well into plans for curricular reform, as this identifies an important niche in which our efforts have the potential to make a significant impact. Further, the direct application of this concept to other specialties such as internal medicine, pediatrics, and obstetrics–gynecology is immediately translatable.

Our study is subject to some limitations. Although the supervising clinicians who rated the first-year residents' performance were blinded with regard to course participation, we could not control for the fact that some course graduates may have discussed their experiences during other interactions such as daily conversation. Additionally, as mentioned above, duty hours compliance is not a perfect surrogate for an individual's on-the-job efficiency. In considering limitations, one must also rule out any additional educational exposures—for example, capstone courses or preparatory workshops—that could have led to improved performance by the course participants and thus affected the outcomes. However, participants and controls were questioned about such potential exposures on multiple occasions in order to exclude the possibility of any additional influences on our results. In a very rigorous way, we queried and tracked all 62 study subjects to find out whether they had additional instruction prior to initiation of residency preparation. During subanalysis, there was no difference between these groups. In fact, if nonparticipants had additional training, this would have only narrowed the significance of our data.

With this study, we demonstrated that providing SMSs with specific instruction and practice in the daily tasks of surgical internship resulted in improved confidence and, more important, improved task-related performance compared with controls. To date, the ability of a fourth-year preparatory course to improve performance objectively has not yet been demonstrated in the published literature. Further, we demonstrated a clear correlation between self-perceived confidence and competence measured by others, a relationship that may prove highly useful for future investigators in this area.

Back to Top | Article Outline
Acknowledgments:

The authors wish to thank Dr. Robert Sweet, Troy Reihsen, and John Raymond for their contributions on behalf of the Simulation PeriOperative Resource for Training and Learning (SimPORTAL) at the University of Minnesota. The authors are indebted to faculty, residents, and the Surgical Education Office at the University of Minnesota Department of Surgery for their support of this course. Additionally, the authors wish to thank faculty and residents of the Departments of Radiology and Anesthesiology at the University of Minnesota for their contributions. Portions of the research described in this paper were conducted as part of the Association for Surgical Education's Surgical Education Research Fellowship (SERF) Program (Dr. Kim Schenarts, SERF Director).

Back to Top | Article Outline

References

1. Logan-Collins JM, Barnes SL, Huezo KL, Pritts TA. Management of common postoperative emergencies: Are July interns ready for prime time? J Surg Educ. 2009;65:453–459.

2. Lind DS, Deladisma AM, Cue JI, Thomas AM, MacFadyen BV, Nesbit RR. Survey of student education in surgery. J Am Coll Surg. 2007;204:969–974.

3. Antonoff MB, Shelstad RC, Schmitz C, Chipman J, D'Cunha J. A novel critical skills curriculum for surgical interns incorporating simulation training improves readiness for acute inpatient care. J Surg Educ. 2009;66:248–254.

4. Schwind CJ, Boehler ML, Markwell SJ, Williams RG, Brenner MJ. Use of simulated pages to prepare medical students for internship and improve patient safety. Acad Med. 2011;86:77–84.

5. Antonoff MB, Swanson JA, Acton RD, et al.. Improving surgery intern confidence through the implementation of expanded orientation sessions. Surgery. 2010;148:181–186.

6. American College of Surgeons Division of Education. Successfully Navigating the First Year of Surgical Residency: Essentials for Medical Students and PGY-1 Surgical Residents. Chicago, Ill: American College of Surgeons; 2005.

7. Boehler ML, Rogers DA, Schwind CJ, Fortune J, Ketchum J, Dunnington G. A senior elective designed to prepare medical students for surgical residency. Am J Surg. 2004;187:695–697.

8. Peyre SE, Peyre CG, Sullivan ME, Towfigh S. A surgical skills elective can improve student confidence prior to internship. J Surg Res. 2006;133:11–15.

9. Esterl RM Jr, Henzi DL, Cohn SM. Senior medical student “boot camp”: Can result in increased self-confidence before starting surgery internships. Curr Surg. 2006;63:264–268.

10. Brunt LM, Halpin VJ, Klingensmith ME, et al.. Accelerated skills preparation and assessment for senior medical students entering surgical internship. J Am Coll Surg. 2008;206:897–904.

11. Antonoff MB, Chipman JG, Schmitz CC, D'Cunha J. Critical response training for surgical interns. MedEdPORTAL; 2010. http://www.aamc.org/mededportal,ID=8056. Accessed November 12, 2011.

12. Spielberger CD, Gorsuch RL, Lushene RE. STAI: Manual for the State-Trait Anxiety Inventory. Palo Alto, Calif: Consulting Psychologists Press; 1970.

13. Spielberger CD, Gorsuch RL, Lushene RE, Vagg PR, Jacobs GA. Manual for State-Trait Anxiety Inventory STAI (Form Y). Palo Alto, Calif: Consulting Psychologists Press; 1983.

14. Promes SB, Chudgar SM, Grochowski COC, et al.. Gaps in procedural experience and competency in medical school graduates. Acad Emerg Med. 2009;16(12 suppl):S58–S62.

15. Adashi EY, Gruppuso PA. Commentary: The unsustainable cost of undergraduate medical education: An overlooked element of U.S. health care reform. Acad Med. 2010;85:763–765.

16. Levey RE. Sources of stress for residents and recommendations for programs to assist them. Acad Med. 2001;76:142–150.

17. Willcock SM, Daly MG, Tennant CC, Allard BJ. Burnout and psychiatric morbidity in new medical graduates. Med J Aust. 2004;181:357–360.

18. Stucky ER, Dresselhaus TR, Dollarhide A, et al.. Intern to attending: Assessing stress among physicians. Acad Med. 2009;84:251–257.

19. Gordon GH, Hubbell FA, Wyle FA, Charter RA. Stress during internship: A prospective study of mood states. J Gen Intern Med. 1986;1:228–231.

20. Tendulkar AP, Victorino GP, Chong TJ, Bullard MK, Liu TH, Harken AH. Quantification of surgical resident stress “on call.” J Am Coll Surg. 2005;201:560–564.

21. Coombes ID, Stowasser DA, Coombes JA, Mitchell C. Why do interns make prescribing errors? A qualitative study. Med J Aust. 2008;188:89–94.

22. Horwitz LI, Moin T, Krumholz HM, Wang L, Bradley EH. Consequences of inadequate sign-out for patient care. Arch Intern Med. 2008;168:1755–1760.

23. Jagsi R, Kitch BT, Weinstein DF, Campbell EG, Hutter M, Weissman JS. Residents report on adverse events and their causes. Arch Intern Med. 2005;165:2607–2613.

24. Sorokin R, Riggio JM, Hwang C. Attitudes about patient safety: A survey of physicians-in-training. Am J Med Qual. 2005;20:70–77.

25. Sachdeva AK, Philibert I, Leach DC, et al.. Patient safety curriculum for surgical residency programs: Results of a national consensus conference. Surgery. 2007;141:427–441.

26. Phillips DP, Barker GEC. A July spike in fatal medication errors: A possible effect of new medical residents. J Gen Intern Med. 2010;25:774–779.

27. Painter K. Your health: Is a hospital visit in July bad for your well-being? USA Today. , 2010. http://www.usatoday.com/news/health/painter/2010-06-28-yourhealth28_ST_N.htm. Accessed November 12, 2011.

28. Carpenter AJ, Yang SC, Uhlig PN, Colson YL. Envisioning simulation in the future of thoracic surgical education. J Thorac Cardiovasc Surg. 2008;135:477–484.

29. Landrigan CP, Barger LK, Cade BE, Ayas NT, Czeisler CA. Interns' compliance with Accreditation Council for Graduate Medical Education work-hour limits. JAMA. 2006;296:1063–1070.

30. Accreditation Council for Graduate Medical Education. Common Program Requirements. http://www.acgme.org/acwebsite/home/Common_Program_Requirements_07012011.pdf. Accessed November 12, 2011.

31. Langdale LA, Schaad D, Wipf J, Marshall S, Vontver L, Scott CS. Preparing graduates for the first year of residency: Are medical schools meeting the need? Acad Med. 2003;78:39–44.

32. Klingensmith ME, Brunt LM. Focused surgical skills training for senior medical students and interns. Surg Clin North Am. 2010;90:505–518.

33. Parent RJ, Plerhoples TA, Long EE, et al.. Early, intermediate, and late effects of a surgical skills “boot camp” on an objective structured assessment of technical skills: A randomized controlled study. J Am Coll Surg. 2010;210:984–989.

34. Lindor KD, Pawlina W, Porter BL, et al.. Commentary: Improving medical education during financially challenging times. Acad Med. 2010;85:1266–1268.

35. Coates WC, Crooks K, Slavin SJ, Guiton G, Wilkerson L. Medical school curricular reform: Fourth-year colleges improve access to career mentoring and overall satisfaction. Acad Med. 2008;83:754–760.

36. Stevens CD. Commentary: Taking back year 4: A call to action. Acad Med. 2010;85:1663–1664.

37. Walling A, Merando A. The fourth year of medical education: A literature review. Acad Med. 2010;85:1698–1704.

38. Lyss-Lerman P, Teherani A, Aagaard E, Loeser H, Cooke M, Harper GM. What training is needed in the fourth year of medical school? Views of residency program directors. Acad Med. 2009;84:823–829.

Back to Top | Article Outline
Funding/Support:

The authors wish to thank the James Lord Foundation for the philanthropic support of this work.

Back to Top | Article Outline
Other disclosures:

None.

Back to Top | Article Outline
Ethical approval:

Ethical approval was granted by the University of Minnesota institutional review board; study number: 0903E62382.

Cited By:

This article has been cited 2 time(s).

American Journal of Surgery
Who's covering our loved ones: surprising barriers in the sign-out process
Antonoff, MB; Berdan, EA; Kirchner, VA; Krosch, TC; Holley, CT; Maddaus, MA; D'Cunha, J
American Journal of Surgery, 205(1): 77-84.
10.1016/j.amjsurg.2012.05.009
CrossRef
Journal of the American College of Surgeons
Surgical Residency and Attrition: Defining the Individual and Programmatic Factors Predictive of Trainee Losses
Sullivan, MC; Yeo, H; Roman, SA; Ciarleglio, MM; Cong, XY; Bell, RH; Sosa, JA
Journal of the American College of Surgeons, 216(3): 461-471.
10.1016/j.jamcollsurg.2012.11.005
CrossRef
Back to Top | Article Outline

© 2012 Association of American Medical Colleges

Login

Article Tools

Images

Share