PDs’ perceptions of FCs
Factor analysis of all completed FCPIs revealed a two-dimensional model for FCPI scores (Table 2). The two identified factors were (1) perception that preclass activity enhances learning (three items), and (2) perception that in-class application enhances learning (four items). The internal consistency reliabilities (Cronbach α) were as follows: 0.831 for the preclass activity factor, 0.924 for the in-class application factor, and 0.921 for all seven items overall. Mean (SD) scores for individual items ranged from 3.67 (0.76) to 4.19 (0.76) on a five-point scale; the overall mean FCPI score was 4.04 (0.62). The overall mean for the perceptions of in-class application factor (4.11 [0.68]) was higher than the overall mean for the perceptions of preclass activity factor (3.94 [0.65]); the difference—0.17 (0.49)—was significant (P < .001).
The mean (SD) age of the PDs was 51.1 (9.1) years, and the mean duration of tenure was 6.8 (6.4) years. Of the 202 PDs who reported gender, 121 were men (59.9%) and 81 were women (40.1%). Table 3 provides additional PD characteristics. The mean FCPI score was higher for PDs who were 50 years or younger (4.12 [0.62]) compared with those older than 50 years (3.94 [0.61]; P = .04) and for women (4.28 [0.56]) compared with men (3.91 [0.62]; P < .001). We noted no statistically significant associations between FCPI score and PD tenure, academic rank, or specialty when we used multiple ANOVA to adjust for all PD characteristics simultaneously (all P ≥ .51).
Residency program characteristics and FC use
Most of the 206 residency programs were university based (77; 37.4%) or community based with a university affiliation (106; 51.4%). The mean (SD) program size was 68.1 (40.1) ACGME-approved positions. Table 4 lists additional program characteristics. Programs with PDs with higher FCPI scores had higher odds of using FC sessions (odds ratio, 4.768; P < .001). This means that a one-point increase in mean FCPI score (e.g., from 3 to 4) is associated with a nearly five-fold increase in the odds of the program using FC sessions. We noted no associations between program type, region, size, ABIM pass rate, IMGs, or hospital size, and the odds of using FC sessions (all P ≥ .30).
FCs in U.S. IM residency programs
The results from our 2015 survey study show that most U.S. IM residency programs have used FCs at least to some extent, that PDs who are women and younger have a more favorable perception of FCs (compared with PDs who are men or who are older), that PDs view the in-class activity more favorably than the preclass component, and that PDs’ perceptions of the FC model are positively associated with the implementation of FCs by residency programs. These findings have implications for the application of FCs in GME.
Strengths of the FC model include opportunities to engage learners in small-group discussions and to apply knowledge to clinical scenarios.1–3,7,26 Despite these advantages and calls to incorporate FCs in medical education,1–3,5 our results indicate that some IM residency programs in the United States (16.5%) never use FCs. Still, most IM residency programs are heeding the call to incorporate FCs—at least to some extent. About 40% of the programs incorporate FCs sometimes, somewhat often, or very often.
One could argue that FCs have long been used in medical education, even before the advent of technological advancements.3,7 For example, anatomy students traditionally review core content before class and then apply this knowledge when dissecting their cadavers—an illustration of how the FC model does not require electronic technology. Although electronic technologies, such as videos (e.g., Khan Academy, iTunes, and TED-Ed), learning management systems (e.g., Moodle and Blackboard), and Web-based repositories (e.g., DropBox) have stimulated an interest in applying FCs to medical education,7 the use of technology does not, in itself, create a different definition of FC than previous models whereby students would read assignments on paper and then discuss those assignments in the classroom. Emerging technologies can improve the portability, accessibility, interactivity, and aesthetics of curricular content, which, in turn, may increase the likelihood that PDs will adopt an FC approach for discussion-based teaching.
As noted, our results show that PDs who are younger perceived FCs more favorably than their comparatively older (> 50) counterparts—even after accounting for tenure and sex. Studies have shown that generational differences exist among medical students regarding attitudes, preferences, and motivation to learn.33–35 Prior results have also indicated that younger physicians perceived social media use in medical education more positively.36 The findings in the present study may similarly indicate that younger PDs are more willing to adopt newer approaches and the new technologies that are commonly used for the preclass component of FCs. Additionally, since exposure to FC interventions has been shown to improve residents’ attitudes toward FCs,26 the younger PDs who responded to the current survey had possibly experienced FCs during their own residency training.
Like younger PDs, the female PDs in our study reported more favorable perceptions of the FC model, which is consistent with results of previous studies. Prior research has shown that female physicians have better perceptions of using technology, such as mobile applications, in medical education.37 Furthermore, in an undergraduate (Baccalaureate) biochemistry course, female students benefited more than male students from an FC model.38 The authors proposed that the reason for this greater benefit may have been that FCs exposed learners to a wider variety of learning tools that might align with their individual, sex-influenced learning styles,38 especially since previous research indicates that medical and physiology students have gender-related learning style preferences.39,40
The residency PDs in our study favored the in-class component of FCs over the preclass activity. This finding aligns with prior research. For example, residents who participated in an FC QI curriculum expressed that in-class sessions improved their acquisition of QI knowledge more than the online content did.26 Likewise, researchers in an undergraduate (baccalaureate) setting compared a traditional lecture versus the activity of solving real physics problems, which trained (but otherwise inexperienced) teaching assistants facilitated; the researchers found that the active learning experience led by inexperienced instructors outperformed the lecture delivered by experts with respect to student attendance, course engagement, and knowledge acquisition.9 The authors of this study proposed that benefits of the FC model may be due in large part to the use of active learning, which has been shown to improve educational outcomes.9 Furthermore, the only study in any discipline that compared the FC versus an active learning environment found no differences, and students in both the FC group and the non-FC active learning group ranked in-class time with the instructor as more influential than out-of-class learning (i.e., homework).41 In this era of duty hours restrictions and high clinical and educational demands, off-loading curricular material to electronically mediated self-study may be tempting. We feel, therefore, reassured that the PDs in our study indicated a preference for in-class discussion and application of knowledge, which has been shown to be a superior method for learning.9,26
Validating the FCPI
Even though a necessary component of education research quality is validity evidence for survey instrument scores, a systematic review revealed that most authors do not provide such validity evidence.42 Content, response process, internal structure, relations to other variables (i.e., criterion variables), and consequences validity evidence all reinforce construct validity,43 and in medical education research, the most commonly reported categories of validity evidence are content and internal structure.44 We selected the FCPI for this study to conduct a replication analysis of its previous use in QI education, where it was shown to have validity evidence.26 Furthermore, from a practical standpoint, the FCPI allowed us to more deeply assess PDs’ attitudes on other detailed and recognized aspects of FCs—namely, preclass activities and in-class application. Strong validity evidence supports the FCPI. Specifically, item content is based on prior instruments6,16,26 regarding perceptions of FCs, along with iterative revisions of items by the study team. Internal structure evidence is demonstrated by a two-factor model of the FC. The two factors—preclass activity and in-class application—replicate the same FCPI model previously demonstrated as effective in GME.26Internal structure evidence is also supported by excellent internal consistency reliability. Finally, we established criterion validity evidence through the positive association between FCPI scores and the odds of programs having an FC curriculum.
Future studies, limitations, strengths
We did not design this study to ascertain the quality or rigor of the FC experiences in IM residency programs in the United States. Future intervention studies should determine the effectiveness of FCs in GME. Furthermore, the development of resources to enable the sharing of FC curricula among medical educators would increase the overall accessibility of FCs. We propose that a national online sharing Web site for the pooling of resources would minimize silos and redundant content. We also encourage residency PDs to be more intentional about using FCs.
We acknowledge our study has limitations. First, our measure of FC use is based on PDs’ accounting, as opposed to direct measurement; however, we believe that PDs generally know whether FCs are used within their programs and the extent to which they are used. Second, the main outcomes of this study were attitudes regarding FCs and the use of FCs, which are, according to Kirkpatrick, level 1 outcomes45; nonetheless, these outcomes provide baseline information. Further, our results do not differ from those of many others according to a systematic review of education research studies, which noted that the majority of published education research involves lower-level outcomes.42 Third, of the 368 PDs, 227 (61.7%) responded to the demographic portion of the APDIM survey, and 206 (56.0%) responded to the FC section. We do not know why 21 PDs started the survey but did not complete it. Notably, the respondents were not significantly different from the 21 nonrespondents for publicly available characteristics. Fourth, this study included only IM training programs; however, FCs emphasize active, participatory learning more than the transmission of information, which is not unique to IM education.
Noteworthy strengths of the present study are the national sample; the validation of FCPI scores among U.S. residency PDs; the replication of FCPI validity among resident physicians26; and the new, useful information regarding the emerging topic of FCs in GME.
Previous research shows that the FC is a student-centered approach that uses active learning and the application of knowledge to clinical scenarios.45–47 This national survey of IM residency PDs extends the FC literature by examining its use in GME. Our findings show that IM residency programs in the United States incorporate FCs at least to some extent; that PDs who are women and younger perceive FCs more favorably than their male and older colleagues; that PDs favor the in-class component of FCs over the preclass activity; and that positive perceptions of FCs are associated with a greater use of the FC model. This study’s findings might provide a useful baseline with respect to the current frequency of FC use in U.S. IM residency programs (approximately 40% reporting at least some use)—and possibly other GME programs. Further, our study may prove informative for PDs who want to adopt or increase their use of the FC approach. Our findings indicate a need for further study not only to investigate why PDs who are women and younger view FCs more favorably but also to examine the effect of FCs on medical knowledge acquisition, engagement in team-based learning,26 actual learner behavior, and the delivery of patient care.
Acknowledgments: The authors are grateful for the support of the Association of Program Directors of Internal Medicine (APDIM), the members of the APDIM Survey Committee, and the residency program directors who completed this survey. The Mayo Clinic Survey Research Center provided assistance with the survey design and data collection.
1. Prober CG, Heath C. Lecture halls without lectures—A proposal for medical education. N Engl J Med. 2012;366:1657–1659.
2. van der Vleuten CP, Driessen EW. What would happen to education if we take education evidence seriously? Perspect Med Educ. 2014;3:222–232.
3. Prober CG, Khan S. Medical education reimagined: A call to action. Acad Med. 2013;88:1407–1410.
4. Mehta NB, Hull AL, Young JB, Stoller JK. Just imagine: New paradigms for medical education. Acad Med. 2013;88:1418–1423.
5. Alberts B. Failure of skin-deep learning. Science. 2012;338:1263.
6. McLaughlin JE, Roth MT, Glatt DM, et al. The flipped classroom: A course redesign to foster learning and engagement in a health professions school. Acad Med. 2014;89:236–243.
7. Moffett J. Twelve tips for “flipping” the classroom. Med Teach. 2015;37:331–336.
8. Martin SK, Farnan JM, Arora VM. Future: New strategies for hospitalists to overcome challenges in teaching on today’s wards. J Hosp Med. 2013;8:409–413.
9. Deslauriers L, Schelew E, Wieman C. Improved learning in a large-enrollment physics class. Science. 2011;332:862–864.
10. Belfi LM, Bartolotta RJ, Giambrone AE, Davi C, Min RJ. “Flipping” the introductory clerkship in radiology: Impact on medical student performance and perceptions. Acad Radiol. 2015;22:794–801.
11. Galway LP, Corbett KK, Takaro TK, Tairyan K, Frank E. A novel integration of online and flipped classroom instructional models in public health higher education. BMC Med Educ. 2014;14:181.
12. Critz CM, Knight D. Using the flipped classroom in graduate nursing education. Nurse Educ. 2013;38:210–213.
13. Missildine K, Fountain R, Summers L, Gosselin K. Flipping the classroom to improve student performance and satisfaction. J Nurs Educ. 2013;52:597–599.
14. Schwartz TA. Flipping the statistics classroom in nursing education. J Nurs Educ. 2014;53:199–206.
15. Khanova J, Roth MT, Rodgers JE, McLaughlin JE. Student experiences across multiple flipped courses in a single curriculum. Med Educ. 2015;49:1038–1048.
16. Pierce R, Fox J. Vodcasts and active-learning exercises in a “flipped classroom” model of a renal pharmacotherapy module. Am J Pharm Educ. 2012;76:196.
17. Park SE, Howell TH. Implementation of a flipped classroom educational model in a predoctoral dental course. J Dent Educ. 2015;79:563–570.
18. Morgan H, Marzano D, Lanham M, Stein T, Curran D, Hammoud M. Preparing medical students for obstetrics and gynecology milestone level one: A description of a pilot curriculum. Med Educ Online. 2014;19:25746.
19. Morgan H, McLean K, Chapman C, Fitzgerald J, Yousuf A, Hammoud M. The flipped classroom for medical students. Clin Teach. 2015;12:155–160.
20. Heitz C, Prusakowski M, Willis G, Franck C. Does the concept of the “flipped classroom” extend to the emergency medicine clinical clerkship? West J Emerg Med. 2015;16:851–855.
21. Nematollahi S, St John PA, Adamas-Rappaport WJ. Lessons learned with a flipped classroom. Med Educ. 2015;49:1143.
22. Liebert CA, Lin DT, Mazer LM, Bereknyei S, Lau JN. Effectiveness of the Surgery Core Clerkship Flipped Classroom: A prospective cohort trial. Am J Surg. 2016;211:451–457.e1.
23. Young TP, Bailey CJ, Guptill M, Thorp AW, Thomas TL. The flipped classroom: A modality for mixed asynchronous and synchronous learning in a residency program. West J Emerg Med. 2014;15:938–944.
24. Khandelwal A, Nugus P, Elkoushy MA, et al. How we made professionalism relevant to twenty-first century residents. Med Teach. 2015;37:538–542.
25. Ramar K, Hale CW, Dankbar EC. Innovative model of delivering quality improvement education for trainees—A pilot project. Med Educ Online. 2015;20:28764.
26. Bonnes SL, Ratelle JT, Halvorsen AJ, et al. Flipping the quality improvement classroom in residency education. Acad Med. 2017;92:101–107.
27. Balwan S, Fornari A, DiMarzio P, et al. Use of team-based learning pedagogy for internal medicine ambulatory resident teaching. J Grad Med Educ. 2015;7:643–648.
31. Accreditation Council for Graduate Medical Education. Accreditation Council for Graduate Medical Education (ACGME)—Public. https://apps.acgme.org/ads/public/
. Published 2017. Accessed April 6, 2017.
32. DeVellis RF. Scale Development: Theory and Applications. 1991.Newbury Park, CA: Sage.
33. Borges NJ, Manuel RS, Elam CL, Jones BJ. Comparing millennial and Generation X medical students at one medical school. Acad Med. 2006;81:571–576.
34. Borges NJ, Manuel RS, Elam CL, Jones BJ. Differences in motives between millennial and Generation X medical students. Med Educ. 2010;44:570–576.
35. Twenge JM. Generational changes and their impact in the classroom: Teaching Generation Me. Med Educ. 2009;43:398–405.
36. Wang AT, Sandhu NP, Wittich CM, Mandrekar JN, Beckman TJ. Using social media to improve continuing medical education: A survey of course participants. Mayo Clin Proc. 2012;87:1162–1170.
37. Wittich CM, Wang AT, Fiala JA, et al. Measuring participants’ attitudes toward mobile device conference applications in continuing medical education: Validation of an instrument. J Contin Educ Health Prof. 2016;36:69–73.
38. Gross D, Pietri ES, Anderson G, Moyano-Camihort K, Graham MJ. Increased preclass preparation underlies student outcome improvement in the flipped classroom. CBE Life Sci Educ. 2015;14:ar36.
39. Slater JA, Lujan HL, DiCarlo SE. Does gender influence learning style preferences of first-year medical students? Adv Physiol Educ. 2007;31:336–342.
40. Wehrwein EA, Lujan HL, DiCarlo SE. Gender differences in learning style preferences among undergraduate physiology students. Adv Physiol Educ. 2007;31:153–157.
41. Jensen JL, Kummer TA, d M Godoy PD. Improvements from a flipped classroom may simply be the fruits of active learning. CBE Life Sci Educ. 2015;14:ar5.
42. Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298:1002–1009.
43. Messick S. Linn RL. Reliability. In: Educational Measurement. 1989:3rd ed. New York, NY: American Council on Education/MacMillan; 610.
44. Beckman TJ, Cook DA, Mandrekar JN. What is the validity evidence for assessments of clinical teaching? J Gen Intern Med. 2005;20:1159–1164.
45. Beckman TJ, Cook DA. Developing scholarly projects in education: A primer for medical teachers. Med Teach. 2007;29:210–218.
46. Bloom BS, Engelhart MD, Furst EJ, Hill WH, Krathwohl DR. Taxonomy of Educational Objectives: Cognitive and Affective Domains. 1956.Ann Arbor, MI: David McKay Co..
47. Beckman TJ, Lee MC. Proposal for a collaborative approach to clinical teaching. Mayo Clin Proc. 2009;84:339–344.
Supplemental Digital Content
Copyright © 2017 by the Association of American Medical Colleges