Secondary Logo

Journal Logo

Flipped Classrooms in Graduate Medical Education: A National Survey of Residency Program Directors

Wittich, Christopher M. MD, PharmD; Agrawal, Anoop MD; Wang, Amy T. MD; Halvorsen, Andrew J. MS; Mandrekar, Jayawant N. PhD; Chaudhry, Saima MD, MSHS; Dupras, Denise M. MD, PhD; Oxentenko, Amy S. MD; Beckman, Thomas J. MD

doi: 10.1097/ACM.0000000000001776
Research Reports
Free
SDC

Purpose To begin to quantify and understand the use of the flipped classroom (FC)—a progressive, effective, curricular model—in internal medicine (IM) education in relation to residency program and program director (PD) characteristics.

Method The authors conducted a survey that included the Flipped Classroom Perception Instrument (FCPI) in 2015 regarding programs’ use and PDs’ perceptions of the FC model.

Results Among the 368 IM residency programs, PDs at 227 (61.7%) responded to the survey and 206 (56.0%) completed the FCPI. Regarding how often programs used the FC model, 34 of the 206 PDs (16.5%) reported “never”; 44 (21.4%) reported “very rarely”; another 44 (21.4%) reported “somewhat rarely”; 59 (28.6%) reported “sometimes”; 16 (7.8%) reported “somewhat often”; and 9 (4.4%) reported “very often.” The mean FCPI score (standard deviation [SD]) for the in-class application factor (4.11 [0.68]) was higher (i.e., more favorable) than for the preclass activity factor (3.94 [0.65]) (P < .001). FC perceptions (mean [SD]) were higher among younger PDs (≤ 50 years, 4.12 [0.62]; > 50 years, 3.94 [0.61]; P = .04) and women compared with men (4.28 [0.56] vs. 3.91 [0.62]; P < .001). PDs with better perceptions of FCs had higher odds of using FCs (odds ratio, 4.768; P < .001).

Conclusions Most IM programs use the FC model at least to some extent, and PDs prefer the interactive in-class components over the independent preclass activities. PDs who are women and younger perceived the model more favorably.

C.M. Wittich is consultant, Division of General Internal Medicine, Mayo Clinic, and associate professor of medicine, Mayo Clinic College of Medicine and Science, Rochester, Minnesota.

A. Agrawal is attending physician, Department of Medicine and Department of Pediatrics, Baylor College of Medicine, and assistant professor, Baylor College of Medicine, Houston, Texas.

A.T. Wang is attending physician, Department of Medicine, Harbor–University of California, Los Angeles, and assistant professor of medicine, David Geffen School of Medicine, University of California, Los Angeles, Torrance, California.

A.J. Halvorsen is statistician, Division of General Internal Medicine, Mayo Clinic, and assistant professor of medicine, Mayo Clinic College of Medicine and Science, Rochester, Minnesota.

J.N. Mandrekar is consultant, Division of Biomedical Statistics and Informatics, Mayo Clinic, and professor of biostatistics and neurology, Mayo Clinic College of Medicine and Science, Rochester, Minnesota.

S. Chaudhry is vice president of academic affairs, Department of Medicine, Memorial Healthcare System, Fort Lauderdale, Florida.

D.M. Dupras is consultant, Division of Primary Care Internal Medicine, Mayo Clinic, and associate professor of medicine, Mayo Clinic College of Medicine and Science, Rochester, Minnesota.

A.S. Oxentenko is consultant, Division of Gastroenterology and Hepatology, Mayo Clinic, and associate professor of medicine, Mayo Clinic College of Medicine and Science, Rochester, Minnesota.

T.J. Beckman is consultant, Division of General Internal Medicine, Mayo Clinic, and professor of medical education and medicine, Mayo Clinic College of Medicine and Science, Rochester, Minnesota.

Funding/Support: This study was supported in part by the Mayo Clinic Internal Medicine Residency Office of Educational Innovations as part of the Accreditation Council for Graduate Medical Education Educational Innovations Project.

Other disclosures: None reported.

Ethical approval: The Mayo Clinic Institutional Review Board reviewed this study and deemed it exempt.

Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A456.

Correspondence should be addressed to Christopher M. Wittich, Division of General Internal Medicine, Mayo Clinic, 200 First St. SW, Rochester, MN 55905; telephone: (507) 284-2511; e-mail: wittich.christopher@mayo.edu.

A current movement to transform medical education emphasizes the learning process over the transmission of information.1–5 This movement stresses learning as enhanced by reflection, group discussion, and application of curricula to problems.2 Traditional methods, in contrast, often require learners to process information independently, an approach that fails to incorporate the expertise of teachers and the social context of learning.2

The flipped classroom (FC) is a progres sive, effective curricular model2,3 that allows learners to process information before class and then apply it during class through the use of facilitated, small-group discussions and activities.3,6,7 An example of traditional content delivery in graduate medical education (GME) is a didactic noon conference delivered by an expert in a lecture style from a podium. Alternatively, an example of an FC model is a journal club for which participants read and review the article independently and then meet to discuss its application to clinical practice. FCs have gained popularity with the advent of massive open online courses and academies, which provide the advantage of continuous, unlimited, and often free access to educational material.3 In this era of electronic learning and duty hours restrictions, FCs might effectively engage residents who must balance the demands of education and patient care.1,8 FCs can provide more flexible options for studying that allow residents to learn off-site and that do not interrupt clinical calendars with traditional, scheduled, on-site lectures. The extent of FC use in GME, however, is unknown.

Although the concept of FCs originated in high schools and undergraduate (i.e., baccalaureate) institutions,3 they have been studied in many settings. For example, the FC model in an undergraduate physics course was associated with higher class attendance and higher test scores,9 and in a medical school radiology clerkship, the use of an FC model improved knowledge compared with a lecture alone.10 In public health,11 nursing,12–14 pharmacy,6,15,16 dentistry,17 and undergraduate medical education,1,10,18–22 the use of FCs has been associated with improved learner outcomes. Specifically among pharmacy students, FCs resulted in better class attendance and knowledge retention; further, the students reported a positive perception of the model.6,15,16 Additionally, flipping a core medical school biochemistry course improved class attendance.1 Studies of the FC model in GME are limited.23–27 One study showed that internal medicine (IM) residents valued in-class application sessions more than the online component of the FC approach.26 Other research has shown that although FCs can improve resident learning, engagement, and satisfaction, they may be underused in GME.26,27

We decided, therefore, to conduct a national survey in collaboration with the Association of Program Directors in Internal Medicine (APDIM) to determine the application of FCs among IM residency programs in the United States. Our aims were (1) to quantify the use of FCs in IM residency education, (2) to validate a method to measure program directors’ (PDs’) perceptions of the FC model, and (3) to examine associations between the implementation of FCs and the characteristics of PDs and residency programs.

Back to Top | Article Outline

Method

The Mayo Clinic Institutional Review Board reviewed this study and deemed it exempt.

Back to Top | Article Outline

Study setting and participants

The APDIM Survey Committee has developed and delivered an annual questionnaire to determine characteristics of allopathic IM residency programs in the United States and to learn about important, emerging topics in GME as perceived by PDs. The APDIM sent the 2015 survey to 368 member PDs (representing 92.9% of the 396 U.S. IM residency programs) through an e-mail link in August 2015. Survey administrators e-mailed reminders weekly and made reminder phone calls in October. The survey closed in November 2015. The Mayo Clinic Survey Research Center administered the survey using Qualtrics software (Qualtrics LLC; Provo, Utah). All survey responses were confidential.

The APDIM survey collected demographic data about each responding PD and the program he/she leads. The FC portion of the survey (Supplemental Digital Appendix 1, http://links.lww.com/ACADMED/A456) included both an assessment of the frequency of FC use in each IM program (“never,” “very rarely,” “somewhat rarely,” “sometimes,” “somewhat often,” and “very often”) and the seven-item Flipped Classroom Perception Instrument (FCPI).6,16,26 To provide a common understanding among PD respondents, the survey instructions defined FC as “learning core content before a session, with class time devoted to applying the core content in a facilitated, group setting.”

We linked survey responses with publicly available data, including program type, region, American Board of Internal Medicine (ABIM) three-year rolling pass rate for 2012–2014, number of Accreditation Council for Graduate Medical Education (ACGME)-approved training positions, PD tenure status, number of international medical graduates [IMGs], number of hospital beds, and mean FCPI score.28–31

Back to Top | Article Outline

FCPI development

We developed the content for the FCPI portion of the survey from existing instruments6,16,26 and from our own experiences and expertise. Specifically, we derived items from an instrument to measure pharmacy students’ perceptions of FCs6,16 and from a previous version of the FCPI developed to study quality improvement (QI) education for resident physicians.26 We modified these items from the original instruments minimally for the current study, so that they could be answered by someone experiencing or directing an FC course. For example, we replaced words such as textbooks, articles, and readings with online modules. We also adapted previous items6,16,26 to assess the following components of an FC: off-loaded content, in-class work, active learning techniques, and group work. After iteratively revising and incorporating input from residency PDs on the APDIM Survey Committee, we included seven items: three focused on preclass activities and four focused on in-class applications. Respondents answered all seven using a five-point Likert scale (where 1 = strongly disagree; 2 = somewhat disagree; 3 = neutral; 4 = somewhat agree; and 5 = strongly agree). A higher score indicated a more favorable perception.

Back to Top | Article Outline

Data analysis

We presented categorical variables as number and percentage of respondents, and continuous variables as mean followed by standard deviation (SD). To assess the representativeness of the programs sampled, we used the Fisher exact test or Welch t test to compare publicly available variables for the survey respondents and nonrespondents.

We completed a factor analysis on the FCPI items. We extracted factors with the minimal proportion criteria and confirmed the results by inspecting the scree plot. We retained all items with factor loadings greater than 0.6. We calculated internal consistency reliability using Cronbach α, and we considered α > 0.7 to be acceptable.32

We assessed associations between PD characteristics and mean FCPI score through a multiple analysis of variance (ANOVA) model. Using simple linear regression models, we assessed the numeric variables of age and tenure separately to test for a possible linear relationship with the mean FCPI score. Then, we dichotomized age and tenure using their medians for inclusion in the ANOVA model. We used a multiple logistic regression model to produce odds ratios and test associations between program characteristics and the use of the FC model. We checked the continuous predictors (ACGME-approved positions, ABIM three-year rolling pass rate for 2012–2014, percentage of positions filled by IMGs, number of hospital beds, and mean FCPI score) and the adequacy of bivariate models, assuming linearity of the log odds, using Hosmer–Lemeshow goodness-of-fit tests. We set statistical significance at P < .05. We conducted all statistical analyses using SAS version 9.4 (SAS Institute Inc.; Cary, North Carolina).

Back to Top | Article Outline

Results

Respondents and FC use by programs

Of the PDs at the 368 IM residency programs, 227 (61.7%) responded to the APDIM survey and 206 (56.0%) completed the FCPI. The programs whose PDs completed the FCPI did not significantly differ from those whose PDs did not complete the FC component, according to publicly available characteristics (Table 1). Regarding how often programs used the FC model, 34 of the 206 PDs (16.5%) reported “never”; 44 (21.4%) reported “very rarely”; another 44 (21.4%) reported “somewhat rarely”; 59 (28.6%) reported “sometimes”; 16 (7.8%) reported “somewhat often”; and 9 (4.4%) reported “very often” (Figure 1). The mean (SD) FCPI score was lower for programs that “never” used an FC model (3.65 [0.62]) than for those that used it “very often” (4.46 [0.42]).

Table 1

Table 1

Figure 1

Figure 1

Back to Top | Article Outline

PDs’ perceptions of FCs

Factor analysis of all completed FCPIs revealed a two-dimensional model for FCPI scores (Table 2). The two identified factors were (1) perception that preclass activity enhances learning (three items), and (2) perception that in-class application enhances learning (four items). The internal consistency reliabilities (Cronbach α) were as follows: 0.831 for the preclass activity factor, 0.924 for the in-class application factor, and 0.921 for all seven items overall. Mean (SD) scores for individual items ranged from 3.67 (0.76) to 4.19 (0.76) on a five-point scale; the overall mean FCPI score was 4.04 (0.62). The overall mean for the perceptions of in-class application factor (4.11 [0.68]) was higher than the overall mean for the perceptions of preclass activity factor (3.94 [0.65]); the difference—0.17 (0.49)—was significant (P < .001).

Table 2

Table 2

The mean (SD) age of the PDs was 51.1 (9.1) years, and the mean duration of tenure was 6.8 (6.4) years. Of the 202 PDs who reported gender, 121 were men (59.9%) and 81 were women (40.1%). Table 3 provides additional PD characteristics. The mean FCPI score was higher for PDs who were 50 years or younger (4.12 [0.62]) compared with those older than 50 years (3.94 [0.61]; P = .04) and for women (4.28 [0.56]) compared with men (3.91 [0.62]; P < .001). We noted no statistically significant associations between FCPI score and PD tenure, academic rank, or specialty when we used multiple ANOVA to adjust for all PD characteristics simultaneously (all P ≥ .51).

Table 3

Table 3

Back to Top | Article Outline

Residency program characteristics and FC use

Most of the 206 residency programs were university based (77; 37.4%) or community based with a university affiliation (106; 51.4%). The mean (SD) program size was 68.1 (40.1) ACGME-approved positions. Table 4 lists additional program characteristics. Programs with PDs with higher FCPI scores had higher odds of using FC sessions (odds ratio, 4.768; P < .001). This means that a one-point increase in mean FCPI score (e.g., from 3 to 4) is associated with a nearly five-fold increase in the odds of the program using FC sessions. We noted no associations between program type, region, size, ABIM pass rate, IMGs, or hospital size, and the odds of using FC sessions (all P ≥ .30).

Table 4

Table 4

Back to Top | Article Outline

Discussion

FCs in U.S. IM residency programs

The results from our 2015 survey study show that most U.S. IM residency programs have used FCs at least to some extent, that PDs who are women and younger have a more favorable perception of FCs (compared with PDs who are men or who are older), that PDs view the in-class activity more favorably than the preclass component, and that PDs’ perceptions of the FC model are positively associated with the implementation of FCs by residency programs. These findings have implications for the application of FCs in GME.

Strengths of the FC model include opportunities to engage learners in small-group discussions and to apply knowledge to clinical scenarios.1–3,7,26 Despite these advantages and calls to incorporate FCs in medical education,1–3,5 our results indicate that some IM residency programs in the United States (16.5%) never use FCs. Still, most IM residency programs are heeding the call to incorporate FCs—at least to some extent. About 40% of the programs incorporate FCs sometimes, somewhat often, or very often.

One could argue that FCs have long been used in medical education, even before the advent of technological advancements.3,7 For example, anatomy students traditionally review core content before class and then apply this knowledge when dissecting their cadavers—an illustration of how the FC model does not require electronic technology. Although electronic technologies, such as videos (e.g., Khan Academy, iTunes, and TED-Ed), learning management systems (e.g., Moodle and Blackboard), and Web-based repositories (e.g., DropBox) have stimulated an interest in applying FCs to medical education,7 the use of technology does not, in itself, create a different definition of FC than previous models whereby students would read assignments on paper and then discuss those assignments in the classroom. Emerging technologies can improve the portability, accessibility, interactivity, and aesthetics of curricular content, which, in turn, may increase the likelihood that PDs will adopt an FC approach for discussion-based teaching.

As noted, our results show that PDs who are younger perceived FCs more favorably than their comparatively older (> 50) counterparts—even after accounting for tenure and sex. Studies have shown that generational differences exist among medical students regarding attitudes, preferences, and motivation to learn.33–35 Prior results have also indicated that younger physicians perceived social media use in medical education more positively.36 The findings in the present study may similarly indicate that younger PDs are more willing to adopt newer approaches and the new technologies that are commonly used for the preclass component of FCs. Additionally, since exposure to FC interventions has been shown to improve residents’ attitudes toward FCs,26 the younger PDs who responded to the current survey had possibly experienced FCs during their own residency training.

Like younger PDs, the female PDs in our study reported more favorable perceptions of the FC model, which is consistent with results of previous studies. Prior research has shown that female physicians have better perceptions of using technology, such as mobile applications, in medical education.37 Furthermore, in an undergraduate (Baccalaureate) biochemistry course, female students benefited more than male students from an FC model.38 The authors proposed that the reason for this greater benefit may have been that FCs exposed learners to a wider variety of learning tools that might align with their individual, sex-influenced learning styles,38 especially since previous research indicates that medical and physiology students have gender-related learning style preferences.39,40

The residency PDs in our study favored the in-class component of FCs over the preclass activity. This finding aligns with prior research. For example, residents who participated in an FC QI curriculum expressed that in-class sessions improved their acquisition of QI knowledge more than the online content did.26 Likewise, researchers in an undergraduate (baccalaureate) setting compared a traditional lecture versus the activity of solving real physics problems, which trained (but otherwise inexperienced) teaching assistants facilitated; the researchers found that the active learning experience led by inexperienced instructors outperformed the lecture delivered by experts with respect to student attendance, course engagement, and knowledge acquisition.9 The authors of this study proposed that benefits of the FC model may be due in large part to the use of active learning, which has been shown to improve educational outcomes.9 Furthermore, the only study in any discipline that compared the FC versus an active learning environment found no differences, and students in both the FC group and the non-FC active learning group ranked in-class time with the instructor as more influential than out-of-class learning (i.e., homework).41 In this era of duty hours restrictions and high clinical and educational demands, off-loading curricular material to electronically mediated self-study may be tempting. We feel, therefore, reassured that the PDs in our study indicated a preference for in-class discussion and application of knowledge, which has been shown to be a superior method for learning.9,26

Back to Top | Article Outline

Validating the FCPI

Even though a necessary component of education research quality is validity evidence for survey instrument scores, a systematic review revealed that most authors do not provide such validity evidence.42 Content, response process, internal structure, relations to other variables (i.e., criterion variables), and consequences validity evidence all reinforce construct validity,43 and in medical education research, the most commonly reported categories of validity evidence are content and internal structure.44 We selected the FCPI for this study to conduct a replication analysis of its previous use in QI education, where it was shown to have validity evidence.26 Furthermore, from a practical standpoint, the FCPI allowed us to more deeply assess PDs’ attitudes on other detailed and recognized aspects of FCs—namely, preclass activities and in-class application. Strong validity evidence supports the FCPI. Specifically, item content is based on prior instruments6,16,26 regarding perceptions of FCs, along with iterative revisions of items by the study team. Internal structure evidence is demonstrated by a two-factor model of the FC. The two factors—preclass activity and in-class application—replicate the same FCPI model previously demonstrated as effective in GME.26Internal structure evidence is also supported by excellent internal consistency reliability. Finally, we established criterion validity evidence through the positive association between FCPI scores and the odds of programs having an FC curriculum.

Back to Top | Article Outline

Future studies, limitations, strengths

We did not design this study to ascertain the quality or rigor of the FC experiences in IM residency programs in the United States. Future intervention studies should determine the effectiveness of FCs in GME. Furthermore, the development of resources to enable the sharing of FC curricula among medical educators would increase the overall accessibility of FCs. We propose that a national online sharing Web site for the pooling of resources would minimize silos and redundant content. We also encourage residency PDs to be more intentional about using FCs.

We acknowledge our study has limitations. First, our measure of FC use is based on PDs’ accounting, as opposed to direct measurement; however, we believe that PDs generally know whether FCs are used within their programs and the extent to which they are used. Second, the main outcomes of this study were attitudes regarding FCs and the use of FCs, which are, according to Kirkpatrick, level 1 outcomes45; nonetheless, these outcomes provide baseline information. Further, our results do not differ from those of many others according to a systematic review of education research studies, which noted that the majority of published education research involves lower-level outcomes.42 Third, of the 368 PDs, 227 (61.7%) responded to the demographic portion of the APDIM survey, and 206 (56.0%) responded to the FC section. We do not know why 21 PDs started the survey but did not complete it. Notably, the respondents were not significantly different from the 21 nonrespondents for publicly available characteristics. Fourth, this study included only IM training programs; however, FCs emphasize active, participatory learning more than the transmission of information, which is not unique to IM education.

Noteworthy strengths of the present study are the national sample; the validation of FCPI scores among U.S. residency PDs; the replication of FCPI validity among resident physicians26; and the new, useful information regarding the emerging topic of FCs in GME.

Back to Top | Article Outline

Conclusions

Previous research shows that the FC is a student-centered approach that uses active learning and the application of knowledge to clinical scenarios.45–47 This national survey of IM residency PDs extends the FC literature by examining its use in GME. Our findings show that IM residency programs in the United States incorporate FCs at least to some extent; that PDs who are women and younger perceive FCs more favorably than their male and older colleagues; that PDs favor the in-class component of FCs over the preclass activity; and that positive perceptions of FCs are associated with a greater use of the FC model. This study’s findings might provide a useful baseline with respect to the current frequency of FC use in U.S. IM residency programs (approximately 40% reporting at least some use)—and possibly other GME programs. Further, our study may prove informative for PDs who want to adopt or increase their use of the FC approach. Our findings indicate a need for further study not only to investigate why PDs who are women and younger view FCs more favorably but also to examine the effect of FCs on medical knowledge acquisition, engagement in team-based learning,26 actual learner behavior, and the delivery of patient care.

Acknowledgments: The authors are grateful for the support of the Association of Program Directors of Internal Medicine (APDIM), the members of the APDIM Survey Committee, and the residency program directors who completed this survey. The Mayo Clinic Survey Research Center provided assistance with the survey design and data collection.

Back to Top | Article Outline

References

1. Prober CG, Heath C. Lecture halls without lectures—A proposal for medical education. N Engl J Med. 2012;366:1657–1659.
2. van der Vleuten CP, Driessen EW. What would happen to education if we take education evidence seriously? Perspect Med Educ. 2014;3:222–232.
3. Prober CG, Khan S. Medical education reimagined: A call to action. Acad Med. 2013;88:1407–1410.
4. Mehta NB, Hull AL, Young JB, Stoller JK. Just imagine: New paradigms for medical education. Acad Med. 2013;88:1418–1423.
5. Alberts B. Failure of skin-deep learning. Science. 2012;338:1263.
6. McLaughlin JE, Roth MT, Glatt DM, et al. The flipped classroom: A course redesign to foster learning and engagement in a health professions school. Acad Med. 2014;89:236–243.
7. Moffett J. Twelve tips for “flipping” the classroom. Med Teach. 2015;37:331–336.
8. Martin SK, Farnan JM, Arora VM. Future: New strategies for hospitalists to overcome challenges in teaching on today’s wards. J Hosp Med. 2013;8:409–413.
9. Deslauriers L, Schelew E, Wieman C. Improved learning in a large-enrollment physics class. Science. 2011;332:862–864.
10. Belfi LM, Bartolotta RJ, Giambrone AE, Davi C, Min RJ. “Flipping” the introductory clerkship in radiology: Impact on medical student performance and perceptions. Acad Radiol. 2015;22:794–801.
11. Galway LP, Corbett KK, Takaro TK, Tairyan K, Frank E. A novel integration of online and flipped classroom instructional models in public health higher education. BMC Med Educ. 2014;14:181.
12. Critz CM, Knight D. Using the flipped classroom in graduate nursing education. Nurse Educ. 2013;38:210–213.
13. Missildine K, Fountain R, Summers L, Gosselin K. Flipping the classroom to improve student performance and satisfaction. J Nurs Educ. 2013;52:597–599.
14. Schwartz TA. Flipping the statistics classroom in nursing education. J Nurs Educ. 2014;53:199–206.
15. Khanova J, Roth MT, Rodgers JE, McLaughlin JE. Student experiences across multiple flipped courses in a single curriculum. Med Educ. 2015;49:1038–1048.
16. Pierce R, Fox J. Vodcasts and active-learning exercises in a “flipped classroom” model of a renal pharmacotherapy module. Am J Pharm Educ. 2012;76:196.
17. Park SE, Howell TH. Implementation of a flipped classroom educational model in a predoctoral dental course. J Dent Educ. 2015;79:563–570.
18. Morgan H, Marzano D, Lanham M, Stein T, Curran D, Hammoud M. Preparing medical students for obstetrics and gynecology milestone level one: A description of a pilot curriculum. Med Educ Online. 2014;19:25746.
19. Morgan H, McLean K, Chapman C, Fitzgerald J, Yousuf A, Hammoud M. The flipped classroom for medical students. Clin Teach. 2015;12:155–160.
20. Heitz C, Prusakowski M, Willis G, Franck C. Does the concept of the “flipped classroom” extend to the emergency medicine clinical clerkship? West J Emerg Med. 2015;16:851–855.
21. Nematollahi S, St John PA, Adamas-Rappaport WJ. Lessons learned with a flipped classroom. Med Educ. 2015;49:1143.
22. Liebert CA, Lin DT, Mazer LM, Bereknyei S, Lau JN. Effectiveness of the Surgery Core Clerkship Flipped Classroom: A prospective cohort trial. Am J Surg. 2016;211:451–457.e1.
23. Young TP, Bailey CJ, Guptill M, Thorp AW, Thomas TL. The flipped classroom: A modality for mixed asynchronous and synchronous learning in a residency program. West J Emerg Med. 2014;15:938–944.
24. Khandelwal A, Nugus P, Elkoushy MA, et al. How we made professionalism relevant to twenty-first century residents. Med Teach. 2015;37:538–542.
25. Ramar K, Hale CW, Dankbar EC. Innovative model of delivering quality improvement education for trainees—A pilot project. Med Educ Online. 2015;20:28764.
26. Bonnes SL, Ratelle JT, Halvorsen AJ, et al. Flipping the quality improvement classroom in residency education. Acad Med. 2017;92:101–107.
27. Balwan S, Fornari A, DiMarzio P, et al. Use of team-based learning pedagogy for internal medicine ambulatory resident teaching. J Grad Med Educ. 2015;7:643–648.
28. American Medical Association. Search the AMA Residency & Fellowship Database. http://www.ama-assn.org/ama/pub/education-careers/graduate-medical-education/freida-online.page. Published 2017. Accessed April 6, 2017.
29. U.S. Department of Commerce Economics and Statistics Administration. U.S. Census Bureau. Census regions and divisions of the United States. https://www2.census.gov/geo/pdfs/maps-data/maps/reference/us_regdiv.pdf. Accessed April 6, 2017.
30. American Board of Internal Medicine. Exam pass rates. http://www.abim.org/about/statistics-data/exam-pass-rates.aspx. Published 2017. Accessed April 6, 2017.
31. Accreditation Council for Graduate Medical Education. Accreditation Council for Graduate Medical Education (ACGME)—Public. https://apps.acgme.org/ads/public/. Published 2017. Accessed April 6, 2017.
32. DeVellis RF. Scale Development: Theory and Applications. 1991.Newbury Park, CA: Sage.
33. Borges NJ, Manuel RS, Elam CL, Jones BJ. Comparing millennial and Generation X medical students at one medical school. Acad Med. 2006;81:571–576.
34. Borges NJ, Manuel RS, Elam CL, Jones BJ. Differences in motives between millennial and Generation X medical students. Med Educ. 2010;44:570–576.
35. Twenge JM. Generational changes and their impact in the classroom: Teaching Generation Me. Med Educ. 2009;43:398–405.
36. Wang AT, Sandhu NP, Wittich CM, Mandrekar JN, Beckman TJ. Using social media to improve continuing medical education: A survey of course participants. Mayo Clin Proc. 2012;87:1162–1170.
37. Wittich CM, Wang AT, Fiala JA, et al. Measuring participants’ attitudes toward mobile device conference applications in continuing medical education: Validation of an instrument. J Contin Educ Health Prof. 2016;36:69–73.
38. Gross D, Pietri ES, Anderson G, Moyano-Camihort K, Graham MJ. Increased preclass preparation underlies student outcome improvement in the flipped classroom. CBE Life Sci Educ. 2015;14:ar36.
39. Slater JA, Lujan HL, DiCarlo SE. Does gender influence learning style preferences of first-year medical students? Adv Physiol Educ. 2007;31:336–342.
40. Wehrwein EA, Lujan HL, DiCarlo SE. Gender differences in learning style preferences among undergraduate physiology students. Adv Physiol Educ. 2007;31:153–157.
41. Jensen JL, Kummer TA, d M Godoy PD. Improvements from a flipped classroom may simply be the fruits of active learning. CBE Life Sci Educ. 2015;14:ar5.
42. Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298:1002–1009.
43. Messick S. Linn RL. Reliability. In: Educational Measurement. 1989:3rd ed. New York, NY: American Council on Education/MacMillan; 610.
44. Beckman TJ, Cook DA, Mandrekar JN. What is the validity evidence for assessments of clinical teaching? J Gen Intern Med. 2005;20:1159–1164.
45. Beckman TJ, Cook DA. Developing scholarly projects in education: A primer for medical teachers. Med Teach. 2007;29:210–218.
46. Bloom BS, Engelhart MD, Furst EJ, Hill WH, Krathwohl DR. Taxonomy of Educational Objectives: Cognitive and Affective Domains. 1956.Ann Arbor, MI: David McKay Co..
47. Beckman TJ, Lee MC. Proposal for a collaborative approach to clinical teaching. Mayo Clin Proc. 2009;84:339–344.

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2017 by the Association of American Medical Colleges