As higher education undergoes intense scrutiny, a formidable challenge for faculty is to adequately prepare students, especially those entering a dynamic and complex health care arena. Newly graduated health care professionals must be equipped with the skill of clinical reasoning to assess and understand client problems, implement a plan, and evaluate outcomes. The characteristics of today's millennials, abundance of information in the knowledge domains, and necessity for advanced reasoning skills endorse the incorporation of active learning strategies in the classroom. Two Institute of Medicine reports and the Carnegie Foundation recognized the requisites for unique instructional design.1-3 These recommendations include teaching for a sense of salience by using contextual and integrated clinical reasoning. In search of strategies to meet these demands, educators give considerable attention to the flipped classroom approach.4-6
The flipped classroom considers students as adult learners by emphasizing and valuing student accountability and self-determination. This model of design is described as an independent and asynchronous study of class content by students and active learning during scheduled class times.7 Active learning and unique instructional design describe the main concepts of a flipped classroom. As the student-student interactions increase the individualization of education, actual instruction parallels more adeptly with the skills of our digital natives.6 Currently, rigorous research is limited on the effectiveness of a flipped classroom in the context of health professions education.
The purpose of this review was to comprehensively and systematically examine relevant research concerned with the use of the flipped classroom approach in health professions education. This review was guided by the question: Among health professions students, what are the quantitative academic achievement outcomes for those taught in a flipped classroom compared with those in a traditional lecture format?
The review methods were derived from the World Health Organization systematic review template protocol at http://www.who.int/hrh/education/planning/en/.8 Specifically, as illustrated in section 1 of the template protocol (p3), a working definition of flipped learning was adopted, and this guided development of the purpose (objective) for the review (template section 2). The review question was developed using the PICO (population, intervention, comparison, and outcomes) format outlined in the template (section 3), and 4 sources of data were used to gather evidence (section 4) including databases, hand searching, gray literature, and reference lists from articles already collected.
To guide the search, flipped learning was defined as a set of pedagogical approaches that include relocating information transmission to outside the classroom, implementing active learning activities into the classroom, and requiring some element of preclass and postclass assessments to appreciate the achievements of in-class activities.9 English-language peer-reviewed original research published in peer-reviewed journals from 2005 through September 2017 was included. The search dates were based on the exponential growth of literature about the flipped classroom during this time frame.10 Keywords for the electronic search included “flipped classroom” and “flipped learning.” Other search terms were “health professions students” and “quantitative academic achievement/performance.” A university health professions librarian was consulted for search term refinement and assistance in identifying appropriate databases.
Eleven electronic databases were searched through EBSCOhost for peer-reviewed literature including Cumulative Index for Nursing and Allied Health Literature, PubMed/MEDLINE, Cochrane Library, Academic Search Premier, Education Research Complete, Education Resources Information Center, Education Abstracts and Retrospective, ProQuest, Journal Storage, Science Direct, and Psychological Information. An additional electronic search for gray literature was conducted using New York Academy of Grey Literature, FADE Library, BioMed Central, Harvard DSpace Archive, NIH Public Access Project, OAlster, Web of Science, and Google Scholar. A hand search was conducted of reference lists in selected articles.
Inclusion criteria for the studies were as follows: (a) health professions students comprised the sample, and (b) quantitative outcomes (quizzes, examinations, course grades) were reported. The process of article selection is illustrated in Figure 1, Supplemental Digital Content (http://links.lww.com/NE/A487). Studies using only subjective measures such as student engagement, satisfaction, and perception of academic achievement were excluded from the review. Studies were also excluded if the faculty used methodologies inconsistent with the flipped classroom method, such as when in-class time was replaced by online activities rather than used to supplement learning. Twenty-four studies were eliminated based on exclusion factors. One study was excluded because a more recent publication used the same data.
Electronic searches of titles and keywords yielded 1948 articles. A large number of duplicates were eliminated. The reference lists of systematic reviews of the flipped classroom and an integrative review of 13 nursing studies were also assessed to find relevant literature.4,11,12 Forty-nine studies were identified as relevant to the topic. Full-text articles were obtained. To avoid selection bias, 2 team members independently reviewed each relevant article to initially assess its relevancy, and then all reviewers made the final decision whether the inclusion and exclusion criteria were met. An iterative process was used to reach consensus about which articles to include.
Assignment of Quality Ratings
To ascertain the quality of the selected studies, a rating scale was developed and assigned to each study by the review team (Table 1, Supplemental Digital Content, http://links.lww.com/NE/A488). The items of the scale were modeled after a published scoring instrument.13 The reviewers further clarified the initial criteria in this scale. Revised items included scoring for sample sizes, clarity of the methods used, and specificity in outcomes measured.
Of the 49 articles identified, 24 met the inclusion criteria. Table 2, Supplemental Digital Content, http://links.lww.com/NE/A489, describes each selected study by the design and quality score, course, duration, and description of intervention. Table 3, Supplemental Digital Content, http://links.lww.com/NE/A490, describes the sample size, student academic outcome measured, and study statistics.
As detailed in Table 2, Supplemental Digital Content, http://links.lww.com/NE/A489, the researchers used various methods. Methods included 7 mixed, 12 quasi-experimental, 4 experimental, and 1 descriptive designs. Only 2 of the experimental studies were randomized controlled trials.14,15 Three studies lacked a methods section.15-17 The study quality varied from a score of 3 to 6 on the rating scale, with a potential range of 0 to 7. Reviewers assigned scores of 0 to 3, considered low quality, to 4 studies. Scores of 4 to 5, considered to be of moderate quality, were assigned to 18 studies. Scores of 6 to 7, considered to be of high quality, were assigned to 2 studies. While the majority of reviewer scores were in agreement, an occasional discrepancy occurred in ratings of outcome definitions and methods. Discrepancies were resolved by discussion within the entire team.
As noted in Table 2, Supplemental Digital Content, http://links.lww.com/NE/A489, the majority of studies (n = 18) involved graduate students. One study involved both undergraduate- and graduate-level students.18 Six studies involved only undergraduate students.14,19-23 Courses from a variety of disciplines were involved in the study including 2 chiropractic health, 11 medical, 5 nursing, 5 pharmacology, and 1 public health. The majority of studies (n = 13) were of 1 semester duration, ranging from 1 hour to 1 semester of content repeated over 2 years. Ten studies conducted over consecutive semesters evaluated different cohorts of students on the same subject content.
Because of the various terms used for flipped, such as active or blended, it was often difficult for reviewers to determine if the methods used were similar. There was great variability in the way faculty described the flipped and traditional methods. Table 2, Supplemental Digital Content, http://links.lww.com/NE/A489, describes how the active learning was implemented and, when reported, the content taught.
Table 3, Supplemental Digital Content, http://links.lww.com/NE/A490, provides findings extracted from the studies. Sample size in individual studies ranged from 17 to 449 students. Course enrollment included students from chiropractic health (n = 122), medicine (n = 1646), nursing (n = 944), pharmacy (n = 992), and public health (n = 33). Table 3 Supplemental Digital Content, http://links.lww.com/NE/A490, lists the results of statistical analysis for student academic outcomes. Types of academic outcomes varied by study. Academic improvements were significantly higher for the flipped classrooms compared with the traditional groups in 17 studies (n = 2589).15-22,24-32 Pierce and Fox18 and Tune et al32 found significant differences in favor of the flipped classroom between years 2011 and 2012, with increases of 4% and 10%, respectively. In 2 other studies, effect sizes were small to medium (0.03 to 0.57) in favor of the flipped classroom.19,28
In 8 studies, no significant differences were found in academic performance between the traditional method and the flipped classroom (n = 966).14,23,30,33-37 Harrington et al14 reported a small effect size of 0.04. None of the studies contained evidence that the students in the flipped classroom had adverse academic outcomes compared with those in the traditional method. Also, no studies provided longitudinal data to allow assessment of change over time.
In any educational endeavor, the goal is to improve the scores from pretest to posttest. Significant differences between the flipped classroom and traditional approaches were noted in 4 studies by comparing pretest-posttest knowledge scores.18,24,25,31 However, without comparisons to equivalent pretest and posttest scores for traditional classroom groups, it is difficult to assess whether the change could be attributed to teaching methodology.
In 7 studies, examination scores were significantly greater for students in the flipped classrooms.15,19-22,26,27 In 2 studies, significant differences were reported in favor of the flipped classroom for some outcomes but not for others.28,32 In 4 studies, no significant differences were reported in examination scores of students experiencing flipped versus traditional classrooms.14,33,34,37
Final Examinations and Final Grades
In comparing students' final examination scores in a traditional lecture group to those in a flipped classroom, Pierce and Fox18 (n = 71) observed that the latter scored significantly higher. Everly19 (n = 139) and Geist et al20 (n = 86) found no significant differences in final examination scores between the 2 methods. In a 2013 pilot study of 35 students, there were no significant differences between traditional and flipped classrooms in final examination scores; however, with the addition of 71 more participants, significant differences were obtained.17
Kiviniemi28 found significant differences between the traditional lecture and flipped classroom students on the final course grades. Ferreri and O'Connor16 found significant changes in the grade distribution, with more A and fewer B, C, and F grades given in the flipped classroom than a traditional lecture group. Harrington et al14 and Whillier and Lystad23 found no significant differences between traditional and flipped groups in the final course grade.
This systematic review examined research investigating use of the flipped classroom approach in health care education. Students and faculty from a variety of health disciplines were represented in the sampled investigations, and studies were quite diverse in terms of courses, content, and size. The review team was struck by the proportion of studies that met the selection criteria, yet revealed mixed results or no significant differences in student academic achievement between the flipped classrooms and the comparison groups. Possible explanations for these findings include overall study quality and student and faculty factors.
The quality of the included research is an important consideration. A rating scale was developed as an objective method of reviewing studies (see Methods). Using the rating scale, the reviewers noted that overall quality ratings were low to moderate. No studies received the highest score of 7 points. Only 1 study rated a score of 6, and most rated a score of 3 to 4. Most study designs were pre-experimental or quasi-experimental rather than true experimental.
Low ratings for studies often reflected a lack of clarity in describing the flipped classroom method. The lack of clarity interfered with reviewers' abilities to discern whether other issues such as the difficulty of the topics or faculty or student factors were more influential in the outcomes than the flipped classroom approach. Faculty factors include experience or inexperience with the pedagogy; actions, such as the overt or covert display of enthusiasm or indifference to the use of the flipped classroom; and consistency in approach during team teaching. Most of the research reports failed to describe the preparation of the faculty for the flipped classroom approach. Only 1 study detailed the processes used to train faculty for consistent content delivery. Student factors included potential baseline differences between cohorts and use of retrospective cohorts for the comparison groups.
The length of time of flipped classroom use varied greatly by study. In some, the approach was used only for the content of 1 unit in a semester, whereas in others, it was studied for a full semester, or longer. Significant differences favoring the flipped classroom groups at the beginning of the semester, which were not sustained through the semester, may have resulted from the traditional lecture students lagging behind at first but catching up developmentally by the end of the course.
The varying lengths of implementation of the flipped classroom approach affected the inferences that could be made from this review. Some academic outcomes were measured after a single topical unit, and others used the content from entire semesters. The variability in these measures circumvented the possibility of conducting a meta-analysis. Clearly, this review would be strengthened by the statistical evidence a meta-analysis would yield.
This review shares a limitation of other research syntheses in that publication bias confined the literature available to the team. The amount of gray literature available to the search was limited. Also, the scale used to rate the quality of the reviewed studies was created by the research team and requires further development as a measure of study merit.
Implications and Recommendations
Inconsistency of terminology is an issue in the literature. Blended learning and other terms have been used to describe flipped classroom approaches, and this complicated the search process. As well as being used to describe the flipped classroom, blended learning refers to online teaching with sessions held in the traditional classroom. A commonly accepted definition for the terms is needed. In the absence of this, faculty should clarify the conceptual definitions they have used for research audiences.
Although the team confined the review to health professions students, some differences in baseline characteristics of students may still have been present among study samples. For example, the academic backgrounds of nursing students differ from many other health professions students because they are undergraduate, may have had a greater number of science courses before their nursing courses, and may have higher grade point averages compared with students in health programs with less competitive admissions. Insufficient information about these background factors was reported in studies to ascertain their contributions to study outcomes.
Another important issue impacting the research on effects of the flipped classroom approach is one that arises from the ways in which “flipping” occurs. Overall, evidence supports active learning strategies as effective approaches for maximizing learning.38 In this review, only Belfi et al24 attempted to isolate the effects of active learning from those of the flipped classroom. Much work is yet to be done to differentiate the effects of the flipped classroom approach from other active learning modalities.
This systematic review of the academic outcomes of the flipped classroom approach for the education of health professions students did not reveal compelling evidence of the effectiveness of the method above that of traditional classroom approaches. However, many studies had methodological weaknesses that were possible reasons that few studies yielded significant differences between study groups. From this systematic review, it can be concluded that the flipped classroom has either a positive or equal effect on academic outcomes as the traditional lecture method. As much as possible, future research should emphasize strong experimental designs and control of variables through study procedures as well as statistical techniques to improve the credibility and merit of the evidence underlying the flipped classroom approach.
1. Benner P, Sutphen M, Leonard V, Day L. Educating Nurses: A Call for Radical Transformation
. San Francisco, CA: Jossey-Bass; 2010.
2. Institute of Medicine. Health Professions Education: A Bridge To Quality
. Washington, DC: The National Academies Press; 2003.
3. Institute of Medicine. The Future of Nursing: Leading Change, Advancing Health
. Washington, DC: The National Academies Press; 2011.
4. Betihavas V, Bridgman H, Kornhaber R, Cross M. The evidence for ‘flipping out’: a systematic review
of the flipped classroom
in nursing education. Nurse Educ Today
5. Critz CM, Knight D. Using the flipped classroom
in graduate nursing education. Nurse Educ
6. Ratta CB. Flipping the classroom with team-based learning in undergraduate nursing education. Nurse Educ
7. Long T, Cummins J, Waugh M. Use of the flipped classroom
instructional model in higher education: instructors' perspectives. J Comput High Educ
9. Abeysekera L, Dawson P. Motivation and cognitive load in the flipped classroom
: definition, rationale and a call for research. High Educ Res Dev
10. Green RD, Schlairet MC. Moving toward heutagogical learning: illuminating undergraduate nursing students' experiences in a flipped classroom
. Nurse Educ Today
11. Chen F, Lui A, Martinelli S. A systematic review
of the effectiveness of flipped classrooms in medical education. Med Educ
12. Presti C. The flipped learning
approach in nursing education: a literature review. J Nurs Educ
13. Han H, Lee J, Kim J, Hedin H, Song H, Kim M. A meta-analysis of interventions to promote mammography among ethnic minority women. Nurs Res
14. Harrington S, Vanden Bosch M, Schoofs N, Beel-Bates C, Anderson K. Quantitative outcomes of nursing students in a flipped classroom
. Nurs Educ Perspect
15. Rui Z, Lian-rui X, Rong-zheng Y, Jing Z, Xue-hong W, Chuan Z. Friend or foe? Flipped classroom
for undergraduate electrocardiogram learning: a randomized controlled study. BMC Med Educ
16. Ferreri SP, O'Connor SK. Redesign of a large lecture course into a small-group learning course. Am J Pharm Educ
. 2013;77(1):article 13.
17. McLaughlin J, Roth M, Glatt D, et al. The flipped classroom
: a course redesign to foster learning and engagement in a health professions school. Acad Med
18. Pierce R, Fox J. Vodcasts and active-learning exercises in a “flipped classroom
” model of a renal pharmacotherapy module. Am J Pharm Educ
19. Everly M. Are students' impressions of improved learning through active learning methods reflected by improved test scores? Nurse Educ Today
20. Geist M, Larimore D, Rawiszer H, Al Sager A. Flipped versus traditional instruction and achievement in a baccalaureate nursing pharmacology course. Nurs Educ Perspect
21. Mattis K. Flipped classroom
versus traditional textbook instruction: assessing accuracy and mental effort at different levels of mathematical complexity. Tech Know Learn
22. Missildine K, Fountain R, Summers L, Gosselin K. Flipping the classroom to improve student performance and satisfaction. J Nurs Educ
23. Whillier S, Lystad RP. No differences in grades or level of satisfaction in a flipped classroom
for neuroanatomy. J Chiropr Educ
24. Belfi LM, Bartolotta RJ, Giambrone AE, Davi C, Min RJ. “Flipping” the introductory clerkship in radiology: impact on medical student performance and perceptions. Acad Radiol
25. Bosner S, Pickert J, Stibane T. Teaching differential diagnosis in primary care using an inverted classroom approach: student satisfaction and gain in skills and knowledge. BMC Med Educ
26. Boysen-Osborn M, Anderson C, Navarro R, et al. Flipping the advanced cardiac life support classroom with team-based learning: comparison of cognitive testing performance for medical students at the University of California, Irvine, United States. J Educ Eval Health Prof
27. Gillispie V. Using the flipped classroom
to bridge the gap to Generation Y. Ochsner J
28. Kiviniemi MT. Effects of a blended learning approach on student outcomes in a graduate-level public health course. BMC Med Educ
29. Koo CL, Demps EL, Farris C, Bowman JD, Panahi L, Boyle P. Impact of flipped classroom
design on student performance and perceptions in a pharmacotherapy course. Am J Pharm Educ
. 2016;80(2):article 33.
30. Morton D, Colbert-Getz J. Measuring the impact of the flipped anatomy classroom: the importance of categorizing an assessment by Bloom's taxonomy. Anat Sci Educ
31. O’Connor E, Fried J, McNulty N, et al. Flipping radiology education right side up. Acad Radio
32. Tune JD, Sturek M, Basile DP. Flipped classroom
model improves student performance in cardiovascular, respiratory, and renal physiology. Adv Physiol Educ
33. Evans K, Thompson A, O'Brien C, et al. An innovative blended preclinical curriculum in clinical epidemiology and biostatistics: impact on student satisfaction and performance. Acad Med
34. Galway L, Corbett K, Takaro T, Tairyan K, Frank E. A novel integration of online and flipped classroom
instructional models in public health higher education. BMC Med Educ
35. Heitz C, Prusakowshi M, Willis G, Franck C. Does the concept of the “flipped classroom
” extend to the emergency medicine clinical clerkship? West J Emerg Med
36. Liebert C, Lin D, Mazur L, Bereknyei S, Lau J. Effectiveness of the surgery core clerkship flipped classroom
: a prospective cohort trial. Am J Surg
37. Marshall L, Nykamp D, Momary K. Impact of abbreviated lecture with interactive mini-cases vs traditional. Am J Pharm Educ
38. Maxwell K. Evaluating the Effectiveness of Two Teaching Strategies to Improve Nursing Students' Knowledge, Skills, and Attitudes in Quality Improvement and Patient Safety
[doctoral dissertation]. Tuscaloosa, AL: University of Alabama Department of Educational Leadership, Policy, & Technology Studies; 2014.