Immediately before and immediately after the four-week elective, participants were asked to rate their levels of confidence in handling pages concerning specific patient management issues. These confidence ratings were ranked on the following six-point Likert scale:
I can handle pages concerning specific patient management issues—
* 0 = Not at all
* 1 = Maybe with complete supervision walking me through it step by step
* 2 = I still need someone right at my side
* 3 = I think I could do it with someone nearby (maybe in the next room)
* 4 = I think I can do it myself
* 5 = I cannot only do it myself but I can teach others
Simulated page procedure
Students were exposed to the patient scenarios, with clinical nurse educators initiating the pages. There were rare instances where students missed pages because of absences from the elective, such as for residency interviews or illness. In addition, medical students entering residency in obstetrics–gynecology had some variations to certain scenarios, and these alternate scenarios were not included in the final analysis. For example, a few students received a scenario on management of a “boggy fundus,” a scenario that was less relevant to the other students enrolled in the elective.
The simulated medical pages were performed in the inpatient setting to mirror the on-call experience of an intern. The participants received pages throughout the course of their work days, as well as one assigned night of “home” call. These pages were distributed across a four-week period, which minimized the effects of fatigue during the battery of scenarios. The order of presentation of scenarios was randomized for each medical student, to minimize practice-related effects that might otherwise skew scores on any particular scenario. Learners made clinical judgments based on the information communicated during the exercise. Participants were instructed not to communicate with peers regarding the educational scenarios.
The simulated pages all involved a structured presentation to the medical student via phone. The medical students were not presented with specific training on responding to pages prior to the exercise, nor did they undergo didactics specific to the scenarios prior to participating in the simulated paging exercise. Pertinent assessment data were scripted for each case, and in the event that students asked for information not scripted, the expert nurse would ad-lib and then record this information to use in future calls to provide standardization. The medical students received information on a patient care finding; they then were required to ask pertinent questions, assess the problem, and provide a management plan along with verbal orders.
The following is an example of the opening presentation of a simulated page: “Mr. Jones is a 55-year-old male, status post myocardial infarction with a stent placed three days ago. He is now complaining of abdominal pain and diarrhea.” The student would then elicit additional information and proceed with evaluation, including an opportunity to “see” the patient by asking for the nurse to provide assessment data from the virtual bedside. The student was expected to elicit pertinent findings from the patient, develop a plan, and give orders. No feedback on performance was provided until the end of the scenario. The scenarios' average time for completion ranged from 2.5 minutes to 9.8 minutes, not including time for feedback. A total of approximately 20 hours of educator time was required to administer the scenarios to all of the students enrolled in the study.
The scoring was performed in real time by the clinical educators (M.L.B. and C.J.S.) during the simulated paging exercise. Medical student responses were scored with the weighted evaluation instrument. To facilitate scoring, students were instructed to complete all queries and diagnostic maneuvers (assessments) before proceeding to treatment (management). Points were added for appropriate maneuvers and subtracted for inappropriate maneuvers, and these results were tabulated separately for assessment and management. Each medical student had 11 scoring sheets completed by the end of the exercise, one sheet per paging scenario. Charts 1 and 2 are examples of scoring sheets.
After the assessment and management outcomes had been recorded, the two clinical educators would each provide a global rating. Global assessments reflected the two evaluators' overall judgment as to whether the medical student was competent to evaluate and treat the patient in a given scenario. This global assessment was intended to capture nuances of competence that might not be measured well by the quantitative assessment and management scores. For example, a medical student that stumbled onto most of the right diagnostic/treatment maneuvers—despite an obvious lack of knowledge and understanding regarding appropriate workup and management—would be judged “not competent.” The global rating was a binary score of competent/not competent and therefore did not attempt to assess varying degrees of competence. The hard copies of scoring results were maintained in folders, and the data were transferred to an electronic SPSS database spreadsheet for analysis at the conclusion of the experiment.
After completion of each simulated paging scenario, the clinical educators would inform the medical student of the diagnosis and then summarize the recommended diagnostic and therapeutic maneuvers. Notes taken by the clinical educator were used to provide specific feedback regarding areas for improvement. Feedback was provided immediately after each individual scenario so as to optimize educational benefit. The critical importance of the patient exam was emphasized during debriefing. The global assessment regarding whether the medical student was deemed competent for that particular situation was also revealed to the medical student, with subjective feedback based on notes taken during the study.
After completion of the course, medical students were asked to again rate their confidence in managing patient-related pages. Students were also asked to comment on whether they felt the exercise improved their clinical skills and whether they were now more comfortable with handling patient-related pages. Last, students were asked to identify any aspects of the scenarios that were unclear or in which the “gold standard” diagnosis/treatment might be improved.
Data analysis and statistical methods
Descriptive statistics were calculated and inspected for all measurements. Pearson correlation coefficients were used to examine relationships between assessment and management for each of the 11 scenarios. For each scenario, kappa coefficients were computed to assess interrater reliability between the two raters on global assessment. The analysis of correlation in global rating scores was performed to evaluate the interrater consistency when judging global performance. This correlation reflected how likely the two raters were to agree/disagree in their assessments regarding the competence of a given student. A paired t test was used to test for a change in medical student confidence from before to after the intervention. Comments of medical students regarding their feedback on the usefulness and value of the exercise were also reviewed and recorded. Results were considered statistically significant for P < .05.
Data were collected on medical student assessment, management, and global performance for all 11 simulated paging cases. Pre/postintervention self-report measures and comments were collected from all 16 students.
The mean scores on the assessment portions of the 11 scenarios were lowest for hypokalemia (57.9 ± 28.1) and highest for hypertension (86.3 ± 14.6). A wider range of scores was observed in the management portions of the scenarios, with the lowest scores received in wound infection management (−15.6 ± 41.6) and the highest scores received for pain management (95.7 ± 9.4). Mean assessment and management scores by scenario are detailed in Table 1. The scores recorded in this table are the range of scores that students actually received on the test scenarios, rather than the theoretical maximum and minimum scores that were possible for students. The students' ratings regarding their confidence in managing inpatient care increased from 1.87 ± 0.83 prior to the simulated paging exercise to 3.53 ± 0.52 postintervention, using ratings on the six-point scale described earlier (P < .0001).
A similar spread in medical student performance was observed for global assessments across scenarios. The number of medical students rated globally competent ranged from a low of 2 (12%; management of wound infection) to a high of 15 (93%; management of hypertension). No significant correlation was observed between medical student assessment and management scores. Interrater agreement for global assessments was high (mean kappa = 0.88). There was perfect concordance between raters on global assessments for 7 of the 11 scenarios (hypertension, hypotension, atelectasis, nausea/vomiting, oliguria, pain, and wound infection, all with kappa = 1.0). Correlations for the remaining scenarios were as follows: hypoxia 0.865, hypokalemia 0.842, chest pain 0.754, and ischemic bowel 0.194.
For the ischemic bowel scenario, there was poor agreement between the two evaluators on global assessments of medical student competence. In exploring the source of this discrepancy, it became apparent that one evaluator required a higher level of performance than the other to designate competence. The evaluator who was more of a “hawk” maintained that a high index of suspicion was required in order to make this diagnosis promptly. If the medical student did not articulate that there was high mortality risk with ischemic bowel and that urgent surgery was needed, the medical student was judged not competent. In contrast, the evaluator who was more of a “dove” felt that a conscientious evaluation, supportive measures, and making provisions for possible future surgery were adequate at the medical student level because medical students are often unsure of themselves and reluctant to take decisive action. This particular case was brought back to the multidisciplinary team for discussion, and it was agreed that the stricter interpretation/standard was appropriate for this particular scenario.
With respect to medical students' subjective comments and observations, three major themes emerged: First, students uniformly indicated that they found the simulated paging exercise to be of benefit. Several medical students specifically cited the value of experiencing a “preview” of what they might expect when on call as interns. Second, students felt that exposure to simulated pages contributed to their clinical experience and comfort; some students speculated that this experience might even decrease their risk of errors and/or improve patient safety early in their internships. Last, the students felt that the experience was effective in promoting critical thinking skills and would help them be better prepared for the internship. Some students proposed additions to the scoring templates, but there were no significant concerns expressed regarding the “gold standard” recommendations for diagnosis and treatment.
Among the most stressful situations for the new intern is handling urgent patient matters that come to their attention by way of their pager. Our data suggest that a simulated paging intervention may ease this difficult transition, as reflected in highly significant improvements in medical student confidence after the simulated paging exercise. We found that a wide variety of patient scenarios can be effectively simulated using this approach, and students found this approach of high educational value. There are several potential benefits to improving the proficiency of new interns in handling pages, ranging from decreased intern stress to better participation in educational activities to improved patient safety.
We observed a wide range of medical student performance across the 11 scenarios. This finding suggests that the scenarios chosen represented an appropriate mix of low-, intermediate-, and high-difficulty clinical situations. The spread of scores was higher for the management than for the assessment portions of most cases. Although it is possible that the assessment portion of the scenarios was easier than the management portion, a more likely explanation relates to medical student development. Another possible explanation for the smaller spread of scores in the assessment section was that there were no negative values assigned to the assessment section. The decision not to penalize superfluous assessments was a potential weakness of the study design. The fourth-year medical students participating in this study seemed to be more proficient in gathering information than in processing and synthesizing it. This finding is not surprising, because history and exam skills are emphasized early in medical training, whereas clinical judgment often tends to develop more gradually over the course of years.
Although there was excellent agreement among the raters, the global assessment was fairly subjective. The discrepancy in judging students' competence for the ischemic bowel scenario reflects the differences in perceptions of competence. One potential source of confounding in the global assessments was that only one evaluator carried out a given paging scenario for any given student. For the evaluator who did not conduct the exercise with the student, the global assessment was based on notes recorded during the paging exercise. Therefore, the rater who directly interacted with the medical student during the exercise may have unduly influenced the evaluator who was relying on the notes alone. One possible solution would be to use videos or recordings, although this would double the time required for the exercise.
An attractive feature of simulated pages is that they foster interdisciplinary teamwork between nurses and future doctors. Medical students are sometimes reluctant to learn from other members of the health care team, such as nurses, pharmacists, and other allied health professionals. Yet, one of the major goals in medical education is to avoid silos and improve interdisciplinary teamwork.29 The teamwork involved in this exercise may help students to understand the shared nature of patient care. For example, when medical students discover the vital role of their nursing colleagues—that nurses not only bring attention to many patient problems but can also be a key part of the solution—teamwork is improved. This approach improves competencies in patient care, practice-based learning, interpersonal and communication skills, and professionalism in accordance with the recommendations of the Accreditation Council for Graduate Medical Education.30
The urgency of pager-initiated communication makes simulated pages highly effective as a teaching intervention. When responding to pages in this exercise, medical students were forced to commit to a course of action regarding diagnostic or therapeutic interventions. This model allows for teachable moments in which concepts can be emphasized with excellent retention. Surveys show that residents prefer pagers to other mediums for receiving patient information.31 An alternative to a simulated paging exercise might be to have medical students take primary call pages with interns. This latter approach affords medical students experience fielding pages, but it does not allow for a systematic exposure to key patient care issues. Also, such an approach would not allow for the structured feedback provided by our design. Another approach might be to introduce a simulated paging exercise at the start of the internship, although orientation for internships is often very brief, limiting the scope and feasibility of such an endeavor.
The use of simulated pages is also appealing from a patient safety perspective. Interns exhibit a substantially greater incidence of serious errors in ICUs when working longer shifts and fatigued.32 If simulated paging exercises allow for earlier acquisition of patient management skills, such improvement may eventually translate into improvements in patient safety.33,34 Based on the short duration of this exercise, no conclusions are possible regarding the effect of simulated pages on patient safety, but this approach may prove beneficial when included as part of a larger curriculum on preparedness for the internship. We are currently undertaking a multi-institutional study that uses audio taping to allow for multiple ratings.
Some limitations of this study warrant mention. Although we measured the performance of our medical students, we did not evaluate the effect of the simulation exercise on the development of patient management skills. In addition, although our scoring instrument was based on accepted standards of care, the weightings used were subjective. Last, the study did not investigate whether simulated pages resulted in error reduction or in a decrease in adverse patient events during internship. Longitudinal studies tracking the effect of medical school interventions on the performance of new interns may allow for assessment of long-term effects.
Our findings suggest that the use of simulated patient pages fosters confidence among medical students regarding their preparedness for internship. This didactic approach is highly interactive and, once validated, may be used to identify areas of strength and weakness in individual medical students. This exercise is also likely to decrease unnecessary stress on the new intern. Our hope is that a simulated paging exercise will leave medical students more confident, better able to triage care, and more insightful as to the importance of multidisciplinary teamwork. Further studies need to be performed, including pretraining and posttraining assessments, to demonstrate the effectiveness of this intervention in improving patient care skills and optimizing patient safety once students have become interns.
The authors wish to acknowledge the members of the multidisciplinary team who helped in the development of simulated pages, and the Southern Illinois University Surgical Education and Performance Group, which provided expert input regarding the refinement of the scenarios used in this study.
Ethical approval was obtained from the Springfield Committee for Research Involving Human Subjects.
Presented at the second International Conference on Surgical Education and Training, Dublin, Ireland, May 2010.
1 Katz MH, Schroeder SA. The sounds of the hospital. Paging patterns in three teaching hospitals. N Engl J Med. 1988;319:1585–1589.
2 Wentz DK, Ford CV. A brief history of the internship. JAMA. 1984;252:3390–3394.
3 Abuhusain H, Chotirmall SH, Hamid N, O'Neill SJ. Prepared for internship? Ir Med J. 2009;102:82–84.
4 Tendulkar AP, Victorino GP, Chong TJ, Bullard MK, Liu TH, Harken AH. Quantification of surgical resident stress “on call.” J Am Coll Surg. 2005;201:560–564.
5 West CP, Tan AD, Habermann TM, Sloan JA, Shanafelt TD. Association of resident fatigue and distress with perceived medical errors. JAMA. 2009;302:1294–1300.
6 Barone JE. Problems with the fourth-year curriculum of students entering surgical residencies. Am J Surg. 1995;169:334–337.
7 Nakayama DK, Steiber A. Surgery interns' experience with surgical procedures as medical students. Am J Surg. 1990;159:341–343.
8 Sloan DA, Donnelly MB, Johnson SB, Schwartz RW, Strodel WE. Use of an objective structured clinical examination (OSCE) to measure improvement in clinical competence during the surgical internship. Surgery. 1993;114:343–350.
9 Antonoff MB, Shelstad RC, Schmitz C, Chipman J, D'Cunha J. A novel critical skills curriculum for surgical interns incorporating simulation training improves readiness for acute inpatient care. J Surg Educ. 2009;66:248–254.
10 Bansal PK, Saoji VA, Gruppen LD. From a “generalist” medical graduate to a “specialty” resident: Can an entry-level assessment facilitate the transition? Assessing the preparedness level of new surgical trainees. Ann Acad Med Singapore. 2007;36:719–724.
11 Boehler ML, Rogers DA, Schwind CJ, Fortune J, Ketchum J, Dunnington G. A senior elective designed to prepare medical students for surgical residency. Am J Surg. 2004;187:695–697.
12 Brunt LM, Halpin VJ, Klingensmith ME, et al. Accelerated skills preparation and assessment for senior medical students entering surgical internship. J Am Coll Surg. 2008;206:897–904.
13 Esterl RM Jr, Henzi DL, Cohn SM. Senior medical student “boot camp”: Can result in increased self-confidence before starting surgery internships. Curr Surg. 2006;63:264–268.
14 Fisher JW, Thompson BM, Garcia AD. Integrative clinical experience: An innovative program to prepare for internship. Teach Learn Med. 2007;19:302–307.
16 Nishisaki A, Hales R, Biagas K, et al. A multi-institutional high-fidelity simulation “boot camp” orientation and training program for first year pediatric critical care fellows. Pediatr Crit Care Med. 2009;10:157–162.
17 Peyre SE, Peyre CG, Sullivan ME, Towfigh S. A surgical skills elective can improve student confidence prior to internship. J Surg Res. 2006;133:11–15.
18 Pliego JF, Wehbe-Janek H, Rajab MH, Browning JL, Fothergill RE. Ob/gyn boot camp using high-fidelity human simulators: Enhancing residents' perceived competency, confidence in taking a leadership role, and stress hardiness. Simul Healthc. Summer 2008;3:82–89.
19 Blum NJ, Lieu TA. Interrupted care. The effects of paging on pediatric resident activities. Am J Dis Child. 1992;146:806–808.
20 Harvey R, Jarrett PG, Peltekian KM. Patterns of paging medical interns during night calls at two teaching hospitals. CMAJ. 1994;151:307–311.
21 Arora VM, Georgitis E, Siddique J, et al. Association of workload of on-call medical interns with on-call sleep duration, shift duration, and participation in educational activities. JAMA. 2008;300:1146–1153.
22 Dawson D, Reid K. Fatigue, alcohol and performance impairment. Nature. 1997;388:235.
23 Bellini LM, Baime M, Shea JA. Variation of mood and empathy during internship. JAMA. 2002;287:3143–3146.
24 Vygotsky LS. Theories of Development—Concepts and Applications. Englewood Cliffs, NJ: Prentice Hall; 1992.
25 Knowles M. The Adult Learner: A Neglected Species. Houston, Tex: Gulf Publishing; 1990.
26 Lefor A, Gomella L, Wiebke E, Fraker D. Surgery on Call. 4th ed. New York, NY: McGraw-Hill Company; 2005.
27 Fauci AS, Braunwald E, Kasper DL, Hauser SL, Longo DL, Jameson JL. Harrison's Principles of Internal Medicine. 18th ed. New York, NY: McGraw-Hill; 2008.
28 Townsend CM, Beauchamp RD, Evers BM, Mattox KL. Sabiston Textbook of Surgery: The Biological Basis of Modern Surgical Practice. 16th ed. Philadelphia, Pa: Saunders; 2007.
29 Margalit R, Thompson S, Visovsky C, et al. From professional silos to interprofessional education: Campuswide focus on quality of care. Qual Manag Health Care. 2009;18:165–173.
31 Wagner MM, Eisenstadt SA, Hogan WR, Pankaskie MC. Preferences of interns and residents for e-mail, paging, or traditional methods for the delivery of different types of clinical information. Proc AMIA Symp. 1998:140–144.
32 Landrigan CP, Rothschild JM, Cronin JW, et al. Effect of reducing interns' work hours on serious medical errors in intensive care units. N Engl J Med. 2004;351:1838–1848.
33 Marshall RL, Gorman PJ, Verne D, et al. Practical training for postgraduate year 1 surgery residents. Am J Surg. 2000;179:194–196.
© 2011 Association of American Medical Colleges
34 Qayumi AK, Cheifetz RE, Forward AD, et al. Teaching and evaluation of basic surgical techniques: The University of British Columbia experience. J Invest Surg. 1999;12:341–350.