In the past decade, educators in the graduate medical education community have used the Accreditation Council for Graduate Medical Education general competencies to develop a competency-based education and training system—the Next Accreditation System. This system represents the shift to criterion-based (competency) assessment, from normative assessment and time-based training, as the driver for resident promotion.1–3 Naturally, this shift in graduate medical education has led in turn to an examination of current models of undergraduate medical education and increasingly to calls for curriculum reform. Questions about the value and the potential for a redesign of the undergraduate medical education experience have focused on the fourth-year curriculum. Recently, some medical schools have designed tracks that eliminate or truncate the fourth year, calling into question the importance of the clinical experiences during this time.4,5
The Association of American Medical Colleges (AAMC) recently published core entrustable professional activities for entering residency (CEPAER), which focused the discussion on the value of the third- and fourth-year curriculum as a means to prepare medical students to enter supervised practice.6 Increasingly, this discussion is moving from theory into practice,7 and the CEPAER document has provided a framework for curricular reform with the goal of optimally preparing students for internship and beyond.
At the same time that the CEPAER recommendations were being developed, the Alliance for Academic Internal Medicine’s Committee on Transitions to Internship (CACTI) Group, comprising members from the Clerkship Directors in Internal Medicine and the Association of Program Directors in Internal Medicine, was charged with developing recommendations to help medical students who are pursuing careers in internal medicine plan their fourth-year curriculum. As a first step, the committee members reviewed the literature and found that much of the discussion on the value and need for a redesign of the fourth-year curriculum took the form of expert opinion. A few studies systematically gathered input from program directors about the perceived deficits and training needs of new interns.8–10 However, limited input has been gathered from residents about the competencies and skills they needed upon entering residency and about the relative value of their medical school clinical experiences in preparing them for internship. For example, one small, interdisciplinary, qualitative study of internal medicine, psychiatry, and surgery postgraduate year (PGY) 2 residents identified the subinternship as most valuable for preparing them for internship and other electives as less valuable. These participants identified deficits in evidence-based medicine, critical care, specialty-specific procedures, and electrocardiogram interpretation, and suggested that medical students be given greater responsibility during the fourth year, including handling pages, prioritizing nurse calls, and making management decisions, to better prepare them for internship.11
The purpose of this study was to obtain input about internship preparation from internal medicine residents, a key stakeholder group underrepresented in the literature on this topic.
A member of the Alliance for Academic Internal Medicine CACTI Group obtained permission from the American College of Physicians, who administers the Internal Medicine In-Training Examination (IM-ITE), to include two detailed questions related to internship preparation on the 2013–2014 iteration of the exam. At the time, the IM-ITE was offered each fall to over 20,000 internal medicine trainees. Although intended for PGY2 residents to identify knowledge strengths and weaknesses, most programs offer the exam to interns and PGY3 residents as well. On finishing the exam, residents are asked to complete a survey to evaluate the exam and to share their perspectives on current issues in training and career selection.
Members of the CACTI Group chose to focus their two survey questions on the skills and fourth-year courses that are important for internship. They selected 10 predefined skills based on a review of the literature, including surveys of program directors, a national subinternship curriculum, and expert consensus. They focused on skills that are important for an intern but that may not receive formal coverage in the clinical clerkships.12 Specifically, residents were asked, “What skills do you feel would have been most important to learn prior to internship?” They were given the list of 10 predefined skills and asked to rate each as very important, somewhat important, not important, or unsure. The second question addressed the degree to which specific courses were helpful in internship preparation. Residents were asked, “What fourth-year courses were most helpful in preparing you for internship (select 3)?” They could select up to 3 courses from a dropdown menu listing 11 of the most common courses offered during the fourth year of medical school.13 (See Appendix 1 for the complete questions and response options.)
We excluded from our analysis responses from residents in years other than PGY1, PGY2, and PGY3. Chi-square statistics were used to test whether there were differences in how residents rated the importance of specific skills and courses based on their training level. Because of the many comparisons we tested and the large sample size, we considered P values < .01 to be statistically significant. Analyses were performed using SPSS version 23 (IBM; Armonk, New York).
The Mayo Clinic institutional review board deemed this study exempt from ethical review.
Of the 24,820 internal medicine residents who completed the 2013–2014 IM-ITE, 22,860 (92%) completed the postexam survey. Of these survey respondents, 20,484 (90%) had complete identification numbers and consented to have their responses used for research. The response rate was similarly high across each of the years of postgraduate training.
Overall, more than 75% of respondents listed 6 of the 10 skills as very important, and all of the skills were rated as very important by more than half of respondents (see Table 1). Furthermore, with the exception of obtaining informed consent (1,582; 8%), less than 4% of respondents listed any of the skills as not important (data not shown). The 3 skills most frequently rated as very important were identifying when to seek additional help and expertise, prioritizing clinical tasks and managing time efficiently, and communicating with other providers around care transitions (admission, discharge, and intensive care unit transfer). As the data in Table 1 illustrate, we found several statistically significant differences between respondents at different training levels. However, the absolute differences were quite small and of questionable practical significance.
Table 2 shows the three most commonly selected courses for helping to prepare medical students for internship in descending order of frequency. Overall, respondents identified the subinternship/acting internship, ward rotations, and subspecialty elective in internal medicine as the most valuable. Less than 20% listed an ambulatory medicine rotation as one of the three most helpful courses in preparing them for internship, and the courses selected by less than 10% of respondents included international electives, subspecialty electives outside internal medicine, electives in performing procedures, capstones/internship 101 courses, and nonclinical electives. This trend was almost the same for every training level, and differences between levels were small.
To our knowledge, ours represents the first national survey of residents’ perspectives on the importance of various skills and common fourth-year experiences in preparing for internship in internal medicine. The results of a recent survey of internal medicine program directors largely aligned with our data—the program directors most frequently ranked the skills of knowing when to seek assistance, communicating with nurses, and time management as high priority.13
Furthermore, the 10 skills we asked residents to rate overlapped with the AAMC’s CEPAER. Specifically, 4 skills correspond directly to a core EPA—practicing evidence-based medicine at the point of care (EPA 7: Form clinical questions and retrieve evidence to advance patient care), providing a prioritized/organized signout (EPA 8: Give or receive a patient handover to transition care responsibility), communicating with nonphysician members of the health care team (EPA 9: Collaborate as a member of an interprofessional team), and obtaining informed consent (EPA 11: Obtain informed consent for tests and/or procedures). About 70% or more of residents who completed our survey described these skills as being very important, with the exception of obtaining informed consent, which 60% identified as being very important. The large percentage of residents indicating that these core EPAs are very important skills for internship also demonstrates consistency between their perspectives and those offered by the AAMC. Thus, internal medicine residency directors, internal medicine residents, and the AAMC are unified in identifying these skills as important ones to have on entering residency.
Medical schools can use this information to design a fourth-year curriculum that optimally prepares students for entering residency. Although the list of courses residents selected as most helpful is not surprising, as the courses appear to be those that afford the most active learning and participation, and the skills identified as important are likely already considered important by those who plan fourth-year curricula, these data from the first national survey of residents on the topic can help prioritize these courses and skills as medical schools continue to develop curricula to bridge the gap between undergraduate and graduate medical education.
In the last few years, new trends regarding the fourth-year curriculum have emerged. It is now common for schools to require one or more subinternships/acting internships. Our data demonstrating that residents find educational value in these courses support this development. However, acting internships tend to be mostly experiential and deemphasize more formal teaching sessions. This characteristic of subinternships in part has led to the growth in courses often labeled as a capstone or internship 101 that are designed specifically to cover topics for internship preparation that may not be formally covered during acting internships or elsewhere in the medical school curriculum. Although these courses vary widely in content from school to school, our data reinforce that they should emphasize core skills.13 What is less clear is why residents rarely rated such capstone courses as one of the most helpful in preparing for internship. Perhaps our data underscore the perceived value of learning from patients and realistic clinical scenarios rather than learning from courses designed by educators. However, these capstone courses are not offered at all medical schools, and they do differ across those schools that offer them, which could account for their low rating. Given a recent recommendation to require capstones during the fourth year,14 our data suggest that more research into the ideal content of such courses and into their effectiveness is needed before mandating them.
Another notable finding was that less than 20% of residents indicated that an ambulatory medicine rotation was one of the three most valuable courses in preparing them for internship, despite the increase in the number of ambulatory care experiences mandated by the Internal Medicine Review Committee of the Accreditation Council for Graduate Medical Education. Residents may not have found such courses helpful because internal medicine residencies continue to be inpatient based rather than ambulatory. Furthermore, little is known about medical students’ experiences in the ambulatory setting, which may be more passive than their experiences during a subinternship or other inpatient ward time. These medical school ambulatory experiences may not address the skills that residents use most in their ambulatory rotations. With medical schools incorporating more ambulatory training for students, it is important to note that residents continue to value those educational experiences less than they do those in the inpatient setting. How best to prepare residents for their ambulatory rotations is another area ripe for further investigation.
Our study has a number of limitations. First, our findings represent only the views of those residents in internal medicine training programs and may not be generalizable to other specialties. However, residents in internal medicine represent a substantial percentage of graduate medical education trainees (the 6,957 PGY1 respondents to our survey represent 26% of the 26,678 first-year positions offered through the Match15). In addition, several other specialties require a PGY1 year in internal medicine, thus expanding the scope of these findings beyond a single discipline. Still, trainees in some disciplines likely would respond differently when asked the same questions. For example, the ability to obtain informed consent may be considered more important for a surgical resident than a medical one.
Another limitation, but also a strength of our dataset, is that it includes the full diversity of internal medicine residents—U.S. MD graduates, U.S. DO graduates, international medical school graduates, and U.S. citizens who graduated from international medical schools. Including this full spectrum of trainees adds strength to our findings regarding those skills that residents find valuable. However, recognizing that only 49% of those matching into internal medicine positions in 2015 were U.S. MD graduates makes our findings regarding the helpfulness of medical school courses more difficult to analyze.16 Because curricula and approaches to medical training vary by country, these findings should be interpreted with caution.
Next, residents’ assessments of newly implemented curricula such as capstone or gateway courses may be difficult to interpret. Our findings may be limited by the lack of a consistent and widely accepted nomenclature and/or curriculum for these emerging courses. However, a recent survey of clerkship directors in internal medicine showed that they recommend capstone courses to their students only slightly less strongly than they do subinternships as experiences to prepare for residency.17
The wording of the questions on the IM-ITE presents additional limitations. Because of space and logistical issues, we did not allow residents to identify skills as important if they were not on our predefined list; other skills may have been identified if we had used a different survey format. By focusing on those skills that may be underrepresented in the curriculum to inform curricular change, we did not include important foundational skills like history taking, physical diagnosis, and oral presentation skills. Similarly, we did not assess the importance of medical knowledge and the development of procedural expertise; thus, we cannot address the medical content or procedural skills that need to remain or be incorporated into the fourth-year curriculum. By asking residents to identify the three most valuable courses in preparing them for internship, however, we have some indirect knowledge of the medical content areas they value. Finally, because residents were asked to select three courses rather than rank all those listed, some important courses may have been overlooked, as residents may have perceived them as valuable but not ranked them in the top three.
Our findings are the first available that demonstrate residents’ perspectives on the skills and experiences they need to prepare for internship. However, as the work of interns evolves in the hospital setting, with changes in technology, limits on resident work hours, and a shift toward an interprofessional team approach to patient care, the skills that residents need also will change. Although our study relied on broad categories of skills, rather than discrete items, revisiting this type of inquiry periodically in the future would be valuable.
These data provide specific feedback from residents engaged in supervised practice during internal medicine training regarding those medical school experiences that best prepared them to function effectively in their current roles. Although the expertise and experience of educational leaders and the “boots on the ground experience” of residency program directors can guide discussions of the optimal fourth-year curriculum to prepare students for residency, we believe that the resident perspective also provides valuable insights to inform such discussions. Ideally, our findings will complement input from educators and undergraduate and graduate medical education leaders alike to help them plan a fourth-year curriculum that will best prepare medical students for supervised practice in graduate medical training.
Acknowledgments: The authors would like to acknowledge the support of the Alliance for Academic Internal Medicine staff in support of the Committee on Transitions to Internship Group.
1. Speer AJ, Solomon DJ, Ainsworth MA. An innovative evaluation method in an internal medicine clerkship. Acad Med. 1996;71(1 suppl):S76S78.
2. Schwind CJ, Williams RG, Boehler ML, Dunnington GL. Do individual attendings’ post-rotation performance ratings detect residents’ clinical performance deficiencies? Acad Med. 2004;79:453457.
3. Aagaard E, Kane GC, Conforti L, et al. Early feedback on the use of the internal medicine reporting milestones in assessment of resident performance. J Grad Med Educ. 2013;5:433438.
4. Abramson SB, Jacob D, Rosenfeld M, et al. A 3-year M.D.—accelerating careers, diminishing debt. N Engl J Med. 2013;369:10851087.
5. Boodman SG. Some medical schools shaving off a year of training. Kaiser Health News. January 14, 2014. http://khn.org/news/medical-schools-shaving-off-a-year-of-training/
. Accessed January 7, 2016.
6. Englander R, Aschenbrener CA, Call SA, et al. Core entrustable professional activities for entering residency. 2013. MedEdPORTAL Publications. https://www.mededportal.org/icollaborative/resource/887
. Accessed January 7, 2016.
7. Englander R, Carraccio C. From theory to practice: Making entrustable professional activities come to life in the context of milestones. Acad Med. 2014;89:13211323.
8. Lyss-Lerman P, Teherani A, Aagaard E, Loeser H, Cooke M, Harper GM. What training is needed in the fourth year of medical school? Views of residency program directors. Acad Med. 2009;84:823829.
9. Langdale LA, Schaad D, Wipf J, Marshall S, Vontver L, Scott CS. Preparing graduates for the first year of residency: Are medical schools meeting the need? Acad Med. 2003;78:3944.
10. Angus S, Vu TR, Halvorsen AJ, et al. What skills should new internal medicine interns have in july? A national survey of internal medicine residency program directors. Acad Med. 2014;89:432435.
11. O’Brien BC, Niehaus B, Teherani A, Young JQ. Residents’ perspectives on the final year of medical school. Int J Med Educ. 2012;3:151158.
12. Harrell H, Aiyer M, Appel J, Gliatto P, Sweet M. Primer to the internal medicine subinternship. 2012. MedEdPORTAL Publications. https://www.mededportal.org/publication/9277
. Accessed January 7, 2016.
13. Elnicki DM, Gallagher S, Willett L, et al. Course offerings in the fourth year of medical school: How U.S. medical schools are preparing students for internship. Acad Med. 2015;90:13241330.
14. Reddy ST, Chao J, Carter JL, et al. Alliance for clinical education perspective paper: Recommendations for redesigning the “final year” of medical school. Teach Learn Med. 2014;26:420427.
15. National Resident Matching Program. Results and Data: 2014 Main Residency Match. 2014. Washington, DC: National Resident Matching Program; http://www.nrmp.org/wp-content/uploads/2014/04/Main-Match-Results-and-Data-2014.pdf
. Accessed January 7, 2016.
16. National Residency Matching Program. Results and Data: 2015 Main Residency Match. 2015. Washington, DC: National Residency Matching Program; http://www.nrmp.org/wp-content/uploads/2015/05/Main-Match-Results-and-Data-2015_final.pdf
. Accessed January 7, 2016.
17. Chretien KC, Elnicki DM, Levine D, Aiyer M, Steinmann A, Willett LR. What are we telling our students? A national survey of clerkship directors’ advice for students applying to internal medicine residency. J Grad Med Educ. 2015;7:382387.
Appendix 1 Questions and Response Options Added to the 2013–2014 Internal Medicine In-Training Examination Related to Preparing for Internship
What skills do you feel would have been most important to learn prior to internship?
Please rate as: very important (4), somewhat important (3), not important (2), or unsure (1)
- Providing a prioritized/organized signout
- Communicating with other providers around care transitions (admission, discharge, intensive care unit transfer)
- Identifying when to seek additional help and expertise
- Communicating with nonphysician members of the health care team
- Communicating with consultants
- Obtaining informed consent
- Recognizing burnout/depression in self and others
- Practicing evidence-based medicine at the point of care
- Reflecting on patient care performance and identifying steps for improvement
- Prioritizing clinical tasks and managing time efficiently
What fourth-year courses were most helpful in preparing you for internship (select 3)?
- Subinternship/acting internship
- Ward rotation
- Subspecialty elective in internal medicine
- Critical care rotation
- Subspecialty elective outside of internal medicine
- Elective in performing procedures
- Capstone/internship 101 course
- Emergency medicine rotation
- Ambulatory medicine rotation
- Nonclinical elective (e.g., medical humanities)
- International elective