McEvoy, Matthew D. MD; DeWaay, Deborah J. MD; Vanderbilt, Allison EdD; Alexander, Louise A. MD; Stilley, Marna C. MA; Hege, Maura C. MA; Kern, Donna H. MD
Dr. McEvoy is vice chair for educational affairs, program director, and associate professor, Department of Anesthesiology, Vanderbilt University, Nashville, Tennessee.
Dr. DeWaay is assistant professor and associate vice chair for education, Department of Internal Medicine, Division of General Medicine and Geriatrics, Medical University of South Carolina, Charleston, South Carolina.
Dr. Vanderbilt is assistant professor and director of assessment and evaluation for the Center of Health Disparities, School of Medicine, Virginia Commonwealth University, Richmond, Virginia.
Dr. Alexander is an anesthesiology resident, Vanderbilt University, Nashville, Tennessee.
Ms. Stilley is research and academic information systems analyst, College of Medicine, Medical University of South Carolina, Charleston, South Carolina.
Ms. Hege is director, Center for Clinical Evaluation and Teaching, College of Medicine, Medical University of South Carolina, Charleston, South Carolina.
Dr. Kern is associate professor, Department of Family Medicine, and associate dean for curriculum–clinical science, College of Medicine, Medical University of South Carolina, Charleston, South Carolina.
Funding/Support: Internal funding from Medical University of South Carolina College of Medicine.
Other disclosures: None reported.
Ethical approval: The institutional review board of the Medical University of South Carolina determined that this was not human research and waived the need for IRB approval.
Previous presentations: A portion of these data was presented as an oral abstract at the 2011 Annual Meeting of the Clerkship Directors in Internal Medicine (CDIM; Anaheim, California).
Correspondence should be addressed to Dr. McEvoy, 2301 Vanderbilt University Hospital, Nashville, TN 37232; telephone: (615) 936-1830; e-mail: firstname.lastname@example.org.
Each year in the United States, approximately one million people have a myocardial infarction, more than 600,000 people are hospitalized for chronic obstructive pulmonary disease exacerbations, and 11% to 42% of medical inpatients have delirium.1–3 Interns are frequently the first line of care during the initial assessment and management of unstable patients with these conditions as well as during in-hospital cardiac arrest events, but their confidence and performance in these critical situations is poor.4–6 Recent research has highlighted that undergraduate medical education is not adequately preparing students to manage unstable patients.7,8 Furthermore, the licensing exams administered to medical students, including the United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills (CS), do not test whether medical students are prepared to address the life-threatening situations that they will encounter during their intern year and beyond.9,10 The stated goal of licensing exams is “to ensure that they [students] have the competencies they require for practice.”10 Although evaluating the ability to assess and manage unstable patients is not a stated goal of the USMLE Step 2 CS exam, it is important to evaluate whether medical students are being prepared to deliver safe, quality care in situations that involve cardiac, pulmonary, and neurologic urgencies and emergencies.
The clinical performance examination (CPX) is designed to measure medical student competence in basic clinical skills and prepare students for the USMLE Step 2 CS. At the Medical University of South Carolina (MUSC), we administer the CPX as a summative assessment at the beginning of the fourth year of the medical school curriculum. The CPX’s clinical scenarios are specifically designed to (1) represent a variety of the conditions and diagnoses that students are required to learn in clinical training; (2) measure competency in clinical reasoning, physical examination, and patient communication skills; and (3) prepare students to pass the USMLE Step 2 CS exam. Prior to the 2010–2011 academic year, the MUSC CPX did not evaluate students’ initial assessment and management of an unstable patient. In this study, we sought to determine whether there is a quantifiable difference in fourth-year medical students’ performance in evaluating and managing a simulated case of ST-elevation myocardial infarction (STEMI) as compared with a variety of stable medical conditions assessed in our CPX. We hypothesized that students would perform better in the encounters with stable patients.
This study was deemed exempt by the institutional review board of MUSC because all students participated in the same examination format and content (i.e., no intervention), and all data provided to the research team were deidentified. We included all fourth-year medical students who took an eight-station CPX exam immediately after the 2010–2011 academic year.
In July, 2011, we administered the standard seven-station exam along with an added eighth station of a patient with acute chest pain having a STEMI. Because this was the first academic year that the STEMI station was included in the CPX, we did not include the students’ performance on this station in their grades or in their transcript. Students were allowed to review all video recordings of their performance during the CPX, which is the standard practice at MUSC. Students who failed any of the other seven stations remediated the exam, and all subsequently passed. No one remediated the STEMI station, regardless of their performance.
All CPX simulations took place in settings that replicate a patient exam room at the MUSC Medical Center (for nonemergent scenarios) or an emergency room bay (for the unstable patient scenario). For the unstable patient scenario, the monitors—the bedside monitor, medical gases monitor (e.g., oxygen), and so forth—imitated those in use at MUSC’s medical center, as did the available medications, intravenous (IV) fluids, IV lines, facemasks, and other supplies. Trained standardized patients (SPs) were used for the simulation sessions at the standard seven stations. For the STEMI station, the simulated patient was a SimMan 2G manikin and SimMan software (Laerdal, Inc., Stavanger, Norway), and an SP was trained to play the role of a nurse. An SP in an adjoining room role-played the voice of the simulated patient through the manikin and responded to questions according to a standardized script. The student was told to assume the role of an intern in the emergency room and act as the responding physician. To increase realism, the SP playing the nurse performed all actions that the student–doctor ordered; these included attaching vital sign monitors, checking vital signs, giving IV fluids or medication, and drawing labs. Student performance at each station was graded by the SPs according to checklists developed through a modified Delphi technique11 (as described below). The exact cases and medical diagnosis are blinded for publication to protect test integrity.
The grading checklists for the CPX’s seven nonacute stations were created by the multispecialty group of MUSC faculty clinician–educators responsible for the development, oversight, and ongoing assessment of the CPX program. We constructed the STEMI station checklist based on current American Heart Association consensus publications and with review by a group of cardiologists at MUSC according to previously reported methods.11–15 The development process for all station checklists proceeded through three stages: (1) generating suggested checklist items; (2) endorsing and removing checklist items until consensus was reached; and (3) receiving feedback about the categories into which the steps would be arranged (e.g., history, counseling, treatment) and the order in which the checklist steps should be presented for grading, with the goal of anticipating the approximate order in which students would accomplish the items. The STEMI Checklist items were grouped into six categories: history; physical exam; differential diagnosis; labs and tests; diagnosis; and management. The STEMI Checklist is presented as Appendix 1. To maintain test integrity, we have not provided the checklists for the seven nonacute stations.
Grading Checklist fo...Image Tools
Orientation for students
There are several points to note about students’ familiarity with the information and skills tested in the STEMI station and the construct of testing using mannequin-based simulation. Targeted didactics delivered during the third-year internal medicine clerkship at MUSC address the approach to the unstable patient with chest pain, with a particular focus on the differential diagnosis of acute coronary syndrome (ACS), aortic dissection, esophageal rupture, pneumothorax, tamponade, and pericarditis. Additionally, we use mannequin and SP-based simulation extensively throughout our medical school curriculum. At the beginning of their first year, students are given an extensive orientation to the SimMan mannequin and have at least four simulation exercises in which they interact with the mannequin during their first and second years.
Prior to the CPX, we gave the students a thorough orientation to the confederate role of the SP nurse and informed them that the nurse would be available to perform any tasks that they directed (e.g., placing monitors, drawing labs). We advised the students that the mannequin would have a live person’s voice supplied by a trained SP from the control room.
Orientation for the two SP roles
We trained the two SPs in three 1-hour sessions with two members of the research team (M.D.M. and M.C.H.; the latter is the person responsible for putting all SPs through a standardized training program). The actors we used had each been a part of the SP program for several years. In addition to the standard training for SPs at MUSC, we gave these team members extensive training on the use of the grading checklist. Prior to grading students, both SPs for the STEMI station practiced using the checklist during 10 live simulations when they were in the role of the nurse SP. We compared the SP’s grading results of the practice simulations with videos of those sessions in order to assess reliability. During the first five sessions, questions emerged about the checklist that we did not anticipate during the initial orientation. For instance, the SP would ask what constituted completion of a certain item on the checklist that could be performed in multiple ways, such as obtaining patient history. In the last five sessions, the SP’s in-scenario grading had perfect agreement when compared with the video review by our team that constructed the checklist. To ensure consistency, one SP played the voice of the patient and one SP played the role of the nurse for all STEMI simulations during each day of the CPX.
During the simulation sessions, the nurse SP recorded whether the student completed checklist items on the grading checklist. Concerning the checklist items involving the differential diagnosis, ordering of labs and test, statement of definitive diagnosis, and initial patient management (items 23–42 in Appendix 1), the student verbalized these to the nurse SP in the following manner. The nurse SP would ask the student a series of questions to standardize how the student was given an opportunity to give feedback and display his or her thought process. After the student appeared finished with the focused history and physical, the nurse SP would attempt to elicit a differential diagnosis, asking: “Doctor, what do you think is going on with this patient?” If the student did not answer or said “I don’t know,” the nurse SP would repeat the question after roughly one minute.
After the student provided a leading or differential diagnosis, the student was given a prompt to think about the next step of ordering labs and tests. The nurse SP would ask, “Is there anything that you would like for me to do? Can I get you anything?” If the student did not answer or said, “I don’t know,” the nurse SP would repeat the question after roughly one minute. If the student ordered labs or tests for which results would be returned during the scenario (e.g., complete blood count, chemistries, troponin, 12-lead electrocardiogram, or chest x-ray), the nurse SP would say, “I think that we have those results already” and give them to the student in paper form. Then, after the student had time to review the results, the nurse SP would ask, “Is there anything else that you would like for me to do? What can I do to help you?” We intended to give each student a prompt to think about the next steps in stabilizing the patient prior to definitive management. If the student ordered something for which a set of results was not available, then the nurse SP would say, “I’ll make sure that gets ordered.”
We calculated the percentage of correct actions completed by each student in each CPX station by dividing the number of items marked as accomplished by the total number of steps on each scenario’s checklist. We performed a repeated-measures analysis of variance (ANOVA) to compare different stations within the CPX. We checked the assumption of sphericity using the Mauchly test and used a Bonferroni correction in the pairwise comparisons following a significant overall test result. To evaluate performance on different components of the STEMI Checklist and to see whether certain areas accounted for the poor performance, we performed a one-way ANOVA on performance scores by checklist component, which were History (11 items), Physical Exam (11 items), Differential Diagnosis (1 item), Labs/Test (3 items), Diagnosis (3 items), and Management/Treatment Plan (13 items) (see Appendix 1). Data are presented as mean percentage (standard deviation [SD]). In the analysis, we defined significance as P < .05. All analyses were conducted in SPSS version 18.0 (IBM Corporation, Armonk, New York).
A total of 143 fourth-year MUSC students completed the CPX in July 2011. The mean percentage (SD) of correct actions in each station was as follows: gastrointestinal, 72.3 (12.2); neurologic, 72.3 (15.1); noncardiac chest pain, 72.0 (11.0); endocrine, 68.2 (12.6); pulmonary (adolescent), 66.7 (12.6); obstetric, 66.0 (13.1); and urologic, 62.9 (15.1) (see Figure 1).
Performance on the STEMI encounter, 47.8 (9.5), was significantly lower than that on all of the other stations (P < .001). There were no statistically significant differences in performance among the other seven stations. In the STEMI station, students performed significantly worse on the physical exam, 37.7 (14.6), and patient management/treatment, 36.4 (14.5), components of the grading checklist as compared with the other areas: history, 59.43 (18.1); differential diagnosis, 52.8 (50.1); labs/test, 75.5 (23.3); and diagnosis, 63.2 (24.5) (Figure 2).
The clinical years of training that prepare a learner for the transition from medical student to a first-year resident are crucial. There is a large chasm that educators must cross in order for students to be prepared to handle the emergencies that they will face as interns and residents. In this study, we found that fourth-year students’ assessment and initial management of a simulated unstable patient with chest pain who was having a classic presentation of a STEMI was poor during a CPX. This poor performance was in stark contrast to students’ consistently better performance with seven SP scenarios involving nonacute conditions that covered a wide range of diagnoses in various organ systems.
When we investigated the components of our students’ poor performance, we found that scores on the physical exam (including obtaining vital signs) and management/treatment plan were significantly lower than the other areas (history, differential diagnosis, labs/tests, and diagnosis). This highlights two potential areas for improvement. First, students may be unaccustomed to directing other members of the health care team to perform any action, such as placing monitors to obtain vital signs. Second, in the CPX’s nonacute stations, students made a plan, but they did not have to do anything in real time that might affect the patient’s condition.8 This is very different from the STEMI station, in which the patient complained of chest pain and asked the student for immediate help. The student was expected to begin initial therapies and perform multiple tasks, ranging from administering supplemental oxygen and ordering appropriate labs to communicating with the cardiology service and ordering initial pharmacologic therapies for an ACS.
As noted above, each year in the United States millions of patients are treated for unstable cardiac, pulmonary, and neurologic conditions.1–3 As interns are frequently the first line of care in these and other emergencies, it is essential that medical schools train students to be able to adequately respond during these crises.4–8 Although there is often backup from upper-level residents and attending physicians in these emergency situations, a recent study has reported that interns feel unprepared in dealing with the assessment and initial management of acutely ill patients.16 When employing unstable patient scenarios in training and assessment, medical schools should focus on areas where breaches to patient safety in high-stakes medical events often occur, such as assessment, diagnosis, communication, and patient management.17 The deficits in patient management that we report here and that others have previously reported4,6,8,16,18,19 are of particular concern given that national guidelines from the Clerkship Directors in Internal Medicine (CDIM) state that medical students should have the skills to discuss, diagnose, and develop a treatment plan for a patient with an acute myocardial infarction, and nearly all schools have in place curricula that address emergency and acute conditions.20 Our results are consistent with those of Nikendei and colleagues,21 who found that ward rounding skills in patient assessment and management are insufficiently developed at the end of the fourth year of medical school. The authors suggest that poor ward rounding skills are secondary to a lack of opportunities to independently round on patients and insufficient supervision of student assessment and management of acutely ill patients. There may be a myriad of reasons why these deficits exist, but it is clear that many trainees beginning their final year of medical school are not prepared to assess and begin treatment for acute myocardial infarctions.
Several studies have noted that simulation-based medical education (SBME) can improve the acute care skills of medical students and residents. Steadman and colleagues18 reported that SBME was superior to problem-based learning (PBL) in terms of student performance across a wide range of unstable patients when students were tested at the end of a weeklong SBME training course versus a PBL course. Additionally, McCoy and colleagues19 reported that SBME was superior to didactic instruction in training medical students to assess and manage simulated cases of an acute myocardial infarction and anaphylaxis. They reported results of 93% correct steps during an objective structured clinical exam for the SBME-trained students and 71% correct steps for the didactic-trained students. The scoring checklist that they used for the ACS scenario was similar to the one that we used, although they used weighted scoring while we did not. Their results showed a very significant effect of SBME, but several notes must be made. First, the follow-up testing in both studies was either one day or one week after initial training. Second, the students in their studies were fourth-year medical students who had self-selected to take an emergency medicine elective that involved this training and testing, whereas we report performance across an entire class of fourth-year medical students who had received didactic training months prior to the testing date. Thus, the findings that SBME produces excellent performance in caring for the unstable patient, including those with simulated ACS, needs to be tested over time and over larger cohorts of students.
The major strength of our study is that it included an entire medical school class, and thus represents the full spectrum of students and their skill level in their final year of medical school. This group of students represents a reliable historical control group for testing the type of intern that MUSC’s curriculum produces in both knowledge and skill. To our knowledge, no other studies have examined student performance in acute care patient treatment in this manner. When considering the implications of our findings, however, we must ask whether there is any clinical importance related to the educational problem under consideration. In response to this, numerous studies have shown that medical students and residents are deficient in their abilities to interpret 12-lead ECGs and perform a basic cardiac exam.22–25 Furthermore, Jayes and colleagues26 demonstrated that errors in interpreting ECGs often lead to increased morbidity and mortality related to the misdiagnosis of life-threatening cardiac events, including ACS. Therefore, it seems appropriate to surmise that improved skills at interpreting ECGs would lead to improved diagnosis of life-threatening cardiac events, which in turn would lead to better patient outcomes. Although this has not been definitively proven, we attempted to take this concept one step further and look at the entire spectrum of assessment and initial management of a patient suffering a STEMI as compared with nonacute patient encounters.
There are several weaknesses in our study. First, we report observational data rather than testing an educational intervention. Although we believe that results from the entire medical school class serve as a reliable historical baseline and clearly demonstrate a deficiency in our educational process, these data do not shed light on the solution to the problem. Second, in the seven nonacute medical condition stations the patient role was played by a trained SP. In the STEMI station, the patient was a mannequin, and an SP in another room gave responses through a speaker in the mannequin. Although this allowed for a conversational nature to the encounter, the absence of nonverbal clues that could have helped the student in the assessment and diagnosis phase may have presented a barrier to better student performance. It is possible that the students did especially poorly on the physical exam and management portions of the encounter because they were not directly interacting with a human. Finally, our CPX is given in the first quarter of the fourth year of medical school in order to allow time for remediation for anyone who does not pass. Many students take critical care, anesthesiology, or emergency medicine rotations during their fourth year, all of which may greatly aid in learning a structured approach to the unstable patient. We demonstrated a deficiency in students’ clinical competence after three years of training, but did not assess their competency at the time of graduation. Finally, this study was only conducted at one medical school, which limits generalizability.
Future research is needed to address a number of issues. The effects of a standardized SBME curriculum on student performance across large groups of students, possibly an entire medical school class, and over time should be investigated. The effect of a mannequin versus a human SP on student performance in the approach to the unstable patient should be evaluated. Finally, future directions concerning this research may have impact on expectations by licensing and credentialing boards. The CDIM currently has higher and more explicit expectations for what should be taught and assessed in every internal medicine clerkship concerning acute care of unstable patients than what appears to be intended by the National Board of Medical Examiners and Accreditation Council for Graduate Medical Education in their expectations and requirements for interns in internal medicine.20,27
In conclusion, we demonstrated that fourth-year medical students performed significantly worse in the assessment and management of a simulated ACS case as compared with a range of various stable patients. Given the necessity of interns to manage urgent and emergent situations, further research should address the best pedagogical method for preparing students to care for the unstable patient. These scenarios may be more appropriate for assessing whether students are ready to transition to the role of doctor. Future research will investigate a wide range of unstable conditions in order to determine whether these findings represent an isolated deficiency or are representative of a larger gap in medical education for acute care training.
1. Kumbhani DJ, Fonarow GC, Cannon CP, et al. Predictors of adherence to performance measures in patients with acute myocardial infarction. Am J Med. 2013;126:1–9
2. Cole MG, Ciampi A, Belzile E, Zhong L. Persistent delirium in older hospital patients: A systematic review of frequency and prognosis. Age Ageing. 2009;38:19–26
3. Stein BD, Charbeneau JT, Lee TA, et al. Hospitalizations for acute exacerbations of chronic obstructive pulmonary disease: How you count matters. COPD. 2010;7:164–171
4. Young JS, Dubose JE, Hedrick TL, Conaway MR, Nolley B. The use of “war games” to evaluate performance of students and residents in basic clinical scenarios: A disturbing analysis. J Trauma. 2007;63:556–564
5. Mayo PH, Hackney JE, Mueck JT, Ribaudo V, Schneider RF. Achieving house staff competence in emergency airway management: Results of a teaching program using a computerized patient simulator. Crit Care Med. 2004;32:2422–2427
6. Kelly C, Noonan CL, Monagle JP. Preparedness for internship: A survey of new interns in a large Victorian health service. Aust Health Rev. 2011;35:146–151
7. Fessler HE. Undergraduate medical education in critical care. Crit Care Med. 2012;40:3065–3069
8. Tallentire VR, Smith SE, Skinner J, Cameron HS. The preparedness of UK graduates in acute care: A systematic literature review. Postgrad Med J. 2012;88:365–371
9. Dillon GF, Boulet JR, Hawkins RE, Swanson DB. Simulations in the United States Medical Licensing Examination (USMLE). Qual Saf Health Care. 2004;13(suppl 1):i41–i45
10. Papadakis MA. The Step 2 clinical-skills examination. N Engl J Med. 2004;350:1703–1705
11. Morgan PJ, Lam-McCulloch J, Herold-McIlroy J, Tarshis J. Simulation performance checklist generation using the Delphi technique. Can J Anaesth. 2007;54:992–997
12. Antman EM, Anbe DT, Armstrong PW, et al. ACC/AHA guidelines for the management of patients with ST-elevation myocardial infarction; a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee to Revise the 1999 Guidelines for the Management of patients with acute myocardial infarction). J Am Coll Cardiol. 2004;44:E1–E211
13. Kushner FG, Hand M, Smith SC Jr, et al. 2009 focused updates: ACC/AHA guidelines for the management of patients with ST-elevation myocardial infarction (updating the 2004 guideline and 2007 focused update) and ACC/AHA/SCAI guidelines on percutaneous coronary intervention (updating the 2005 guideline and 2007 focused update) a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2009;54:2205–2241
14. O’Connor RE, Brady W, Brooks SC, et al. Part 10: Acute coronary syndromes: 2010 American Heart Association Guidelines for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care. Circulation. 2010;122(18 suppl 3):S787–S817
15. McEvoy MD, Smalley JC, Nietert PJ, et al. Validation of a detailed scoring checklist for use during advanced cardiac life support certification. Simul Healthc. 2012;7:222–235
16. Illing JC, Morrow GM, Rothwell Nee Kergon CR, et al. Perceptions of UK medical graduates’ preparedness for practice: A multi-centre qualitative study reflecting the importance of learning on the job. BMC Med Educ. 2013;13:34
17. Kohn LT, Corrigan J, Donaldson MS To Err Is Human: Building a Safer Health System. 2000 Washington, DC National Academy Press
18. Steadman RH, Coates WC, Huang YM, et al. Simulation-based training is superior to problem-based learning for the acquisition of critical assessment and management skills. Crit Care Med. 2006;34:151–157
19. McCoy CE, Menchine M, Anderson C, Kollen R, Langdorf MI, Lotfipour S. Prospective randomized crossover study of simulation vs. didactics for teaching medical students the assessment and management of critically ill patients. J Emerg Med. 2011;40:448–455
20. De Fer TFSaUTF. Core Medicine Clerkship Curriculum Guide: A Resource for Teachers and Learners. CDIM/SGIM Core Medicine Clerkship Curriculum Guide. Version 3.0. Belfast, United Kingdom; 2006
21. Nikendei C, Kraus B, Schrauth M, Briem S, Junger J. Ward rounds: How prepared are future doctors? Med Teach. 2008;30:88–91
22. Berger JS, Eisen L, Nozad V, et al. Competency in electrocardiogram interpretation among internal medicine and emergency medicine residents. Am J Med. 2005;118:873–880
23. Eslava D, Dhillon S, Berger J, Homel P, Bergmann S. Interpretation of electrocardiograms by first-year residents: The need for change. J Electrocardiol. 2009;42:693–697
24. Little B, Mainie I, Ho KJ, Scott L. Electrocardiogram and rhythm strip interpretation by final year medical students. Ulster Med J. 2001;70:108–110
25. Trzeciak S, Erickson T, Bunney EB, Sloan EP. Variation in patient management based on ECG interpretation by emergency medicine and internal medicine residents. Am J Emerg Med. 2002;20:188–195
26. Jayes RL Jr, Larsen GC, Beshansky JR, D’Agostino RB, Selker HP. Physician electrocardiogram reading in the emergency department—accuracy and effect on triage decisions: Findings from a multicenter study. J Gen Intern Med. 1992;7:387–392
27. Iobst W, Aagaard E, Bazari H, et al. Internal medicine milestones. J Grad Med Educ. 2013;5:14–23