In 2010 the global independent Commission on Education of Healthcare Professionals for the 21st Century called for educational reform to generate transformative learning, which focuses on the higher levels of cognition—analyzing, evaluating, and creating.1 The commission also recommended a shift to competency-driven instruction that correlates with the “shows how” and “does” levels in Miller’s2 pyramid of clinical assessment. In support of transformative learning, thought leaders in nursing, medicine, and pharmacy have advocated a shift from teacher-centered methods of instruction to student-centered learning.3–9 Student-centered methods of instruction shift the focus of instructional activities from the teacher to the students and foster student engagement, motivation, and empowerment. The flipped classroom has emerged in undergraduate settings and health professions schools as one means to achieve this end.10
Flipped learning has been defined as
a pedagogical approach in which direct instruction moves from the group learning space to the individual learning space, and the resulting group space is transformed into a dynamic, interactive learning environment where the educator guides students as they apply concepts and engage creatively in the subject matter.11
Whereas the traditional lecture-based, teacher-centered model may include brief in-class active learning exercises, in the flipped learning model the majority of class time is spent engaged in active learning. Flipped classrooms typically also include preclass learning activities to promote learners’ background understanding of the material.12
In health professions education, student satisfaction after implementing the flipped learning model has been inconsistent, with some studies demonstrating increased learner satisfaction13–24 and others finding diminished satisfaction.25–27 Shifting from a teacher-centered to a learner-centered approach by flipping the classroom resulted in improved performance, as measured by exam scores and final grades, in some studies.14,17,22–29 In other studies, however, flipping courses in nursing and pharmacy programs failed to improve performance on exams.20,30
Whereas written exams allow evaluation of performance at the lower levels of Miller’s2 pyramid of clinical assessment (“knows” and “knows how”), objective structured clinical examinations (OSCEs) provide a measure of skill-based performance (“shows how”) and shift assessment to higher cognitive levels. Given the limited evidence that flipped learning approaches improve both knowledge and skill attainment, we undertook this study to compare student performance on the same skill- and knowledge-based assessments following traditional and flipped methods of instruction.
First-professional-year doctor of pharmacy (PharmD) students enrolled in a required pharmacology and therapeutics course at the University of Maryland School of Pharmacy in spring 2015 and spring 2016 were included in this study. The University of Maryland School of Pharmacy consists of a main campus in Baltimore and a distance campus at the Universities at Shady Grove in Rockville, Maryland. Approximately 160 students are admitted to the four-year PharmD program each year, with 120 students based on the Baltimore campus and 40 students at the distance campus. Didactic content for the PharmD program has largely been delivered face-to-face in the Baltimore classrooms and video recorded for later online distribution via Mediasite (Sonic Foundry, Madison, Wisconsin). Students on both campuses may choose to attend lectures on the Baltimore campus, to review the video recordings through their personal computers, or both.
Students participate in the pain management module of a pharmacology and therapeutics course in the second semester of their first professional year. Under the traditional model of instruction, the pain management module consisted of 12 hours of therapeutics-related content and 5 hours of pharmacology, toxicology, and medicinal chemistry. In spring 2015, this module was delivered using traditional lecture-based methods to students in the class of 2018 (traditional cohort [TC]), who served as the control group in this study. In spring 2016, a flipped classroom redesign of the pain therapeutics content was implemented for students in the class of 2019 (flipped cohort [FC]). Competencies and objectives remained the same for both cohorts. Moreover, the module was taught by the same instructor in both years, and the required texts remained the same.
Traditional module design
The traditional module design (Figure 1) was consistent with the customary model of instruction and content delivery in the University of Maryland PharmD curriculum. Module learning objectives were written to support achievement of 10 competencies in pain management. Lectures were delivered live in a classroom on the Baltimore campus; recordings were posted in Mediasite. TC students had access to the PowerPoint slides for each lecture. They were given one ungraded problem set for the module and were expected to read relevant chapters in the required textbooks to supplement in-class learning. No incentives (e.g., points awarded toward course grade) were given for attendance or completing in-class or supplemental learning activities. No graded quizzes were administered. A key to the problem set was posted in the online learning management system. Recommended postclass learning activities included reading the textbook and reviewing notes, slides, and the problem set.
Flipped module design
To inform the design of the flipped module (Figure 1), we analyzed preceptor feedback and student performance data. We identified comprehensive pain assessment, opioid conversion, and knowledge of drug-specific patient- and agent-related variables as areas of weakness. Therefore, these content areas were given priority.
We did not employ a single method for flipping the pain therapeutics content. Rather, we considered the learning objectives and the nature of the content for each competency with a goal of shifting from a teacher-centered to a learner-centered model. In-class instructional methods selected included cases, scenario-based e-learning modules, and quiz games. Infographics (job aids) were created to supplement learning materials.
In an effort to engage students in the redesign, we surveyed TC students in April 2015, at the midpoint of the pain management module, and considered their learning preferences when selecting and designing preclass learning activities for the flipped module. Survey respondents (n = 145/156; 93%) ranked preclass learning activities in the following order of preference: prerecorded lectures, YouTube-style videos, online interactive modules, case-based guided learning questions, reading textbooks, reading guidelines, reading review articles, and reading clinical trials. On the basis of these findings, we minimized readings as preclass learning activities in the flipped module.
In the flipped module, classroom-based sessions were held as live, synchronous sessions on both campuses. Videoconferencing technology allowed students at the distance campus to see, hear, and verbally interact with the instructor and students on the main campus. Audio recordings of most sessions were posted to the course learning management system (Blackboard; Blackboard, Washington, DC). We provided a module guide to the FC students, outlining the preclass, in-class, and postclass learning opportunities, resources, and suggested completion dates. No incentives (e.g., points awarded toward course grade) were given for attendance or for completing preclass or in-class learning activities. No graded quizzes were administered. Recommended postclass learning activities included reviewing preclass and in-class learning activities.
We considered an instructional method to be learner-centered if the instructor’s role was largely facilitative; examples included scenario-based e-learning modules, problem sets, cases, audience response questions, readings, student-led review sessions, and modeling. We considered teacher-centered methods to be instructional strategies where students passively received information from the instructor, such as live or prerecorded lectures.10
As intended, the class time spent on therapeutics using learner-centered methods increased substantially, from 1 hour in the traditional module to 11 hours in the flipped module. There was a corresponding decrease in the class time using instructor-centered methods, from 11 hours in the traditional module to 2 hours in the flipped module. Total classroom-based instructional time increased from 12 hours in the traditional module to 13 hours in the flipped module. FC students were provided with a variety of preclass learning activities (available through the course learning management system), including mini-lectures, guided learning questions, e-learning modules, and problem sets; as noted above, TC students were provided with one problem set, its key, and reading assignments. Because of the significant decrease in hours using teacher-centered methods, the total number of PowerPoint slides displayed and shared with students during the module decreased from 600 in the traditional module to 259 in the flipped module (Figure 1).
The primary outcome of this study was performance (mean score) on a postmodule one-station OSCE. Secondary outcomes included OSCE grade distribution, mean score on a postmodule multiple-choice exam, multiple-choice exam grade distribution, and performance on individual multiple-choice exam and OSCE rubric items.
The OSCE case and analytical checklist were peer reviewed by the university’s OSCE coordinator. Quality control measures were implemented following the customary procedures used for all OSCEs at our institution, including training of simulated patients (SPs), pilot testing the OSCE station with senior pharmacy students, and video review of all encounters by a second observer following the examination.
The same multiple-choice exam and OSCE were deployed to the TC (n = 156 students) and FC (n = 162 students) at the end of the module. The multiple-choice exam consisted of 37 questions, of which 20 assessed pain therapeutics content and 17 assessed pharmacology, medicinal chemistry, and toxicology content. All exams were delivered in proctored environments, and students did not have access to previous exam materials. The OSCE was administered several days before the multiple-choice exam.
During the OSCE, students were required to interview an SP who was experiencing chronic pain, assess the pain complaint, and develop a pain management plan. SPs evaluated students’ interpersonal skills, including empathy, using a global rating scale.31 SPs also scored the students’ information-gathering skills using a standardized checklist and provided formative feedback to each student. Nine of the 10 SPs who participated in the case in spring 2015 returned in spring 2016.
At the conclusion of the SP encounter, students wrote a SOAP (subjective, objective, assessment, plan) note to document their findings, pain assessment, and management plan. Each SOAP note was scored by a faculty member (M.L.M., n = 70), an instructional design fellow (K.L., n = 174), or other teaching assistants (n = 74) using a rubric, which was not shared with students before the assessment. Because of the potential for inconsistencies in SOAP note scoring and the lack of a validated rubric, a quality assurance procedure was employed at the end of the study. For each cohort, 4 SOAP notes were randomly selected from each quartile of student performance, deidentified (cohort and student identity), and rescored in a blinded manner by an instructor specializing in pain (M.L.M.) and an instructor who is not a pain specialist (S.T.H.). Thus, 16 notes per cohort (32 total) were rescored by 2 instructors blinded to the cohort and student.
At the midpoint of the module, students in each cohort were asked to participate in a voluntary survey. The survey instrument included items from the Motivated Strategies for Learning Questionnaire (MSLQ),32 specifically the 25 items related to 5 domains: intrinsic goal orientation, extrinsic goal orientation, task value, elaboration, and critical thinking. (The full MSLQ consists of 81 questions in 15 domains.) Responses use a seven-point Likert-type scale ranging from 1 = “not at all true of me” to 7 = “very true of me,” with higher scores representing a strength in the given domain. Additional survey items queried students about their learning preferences. Because this survey was reflective in nature and a form of metacognitive development, a bonus point incentive equivalent to 1% of the course grade was given to students who completed it. These bonus points were not included in the analysis.
The University of Maryland, Baltimore Institutional Review Board determined this study protocol to be exempt.
Data were analyzed using GraphPad Prism 7 (GraphPad Software, La Jolla, California). Cohort demographics were compared using independent t tests or Mann–Whitney tests as appropriate. Cohort mean scores on the multiple-choice exam, OSCE, and MSLQ were compared using the independent t test. The multiple-choice exam included questions on therapeutics content, which was flipped in 2016 for the FC, as well as the pharmacology, toxicology, and medicinal chemistry content, which was not flipped for either cohort. Raw scores on these two sections of the exam were converted to 100 points, and mean percentage points for the TC and FC for each section were compared using the independent t test. Effect sizes were determined with Cohen d. The multiple-choice exam scores of two students (one student in each cohort) were excluded because these students missed the original exam and were unable to take the makeup exam in a timely manner. Differences in grade distributions between the TC and FC were analyzed using the Mann–Whitney test. Cohort performance on individual multiple-choice items and on OSCE rubric items was compared using the Fisher exact test. Global rating scores were compared using the Mann–Whitney test. The selection of SOAP notes for blinded rescoring was randomized using Microsoft Excel (Microsoft, Redmond, Washington); the interrater reliability comparing agreement between scores awarded by the two blinded graders was analyzed using the Bland–Altman correlation. P values < .05 were considered to be significant, and all analyses were two sided.
Demographic characteristics and MSLQ scores
The baseline characteristics of the students in the TC (n = 156) and FC (n = 162) were similar at the time of admission (Table 1). Survey respondents in the TC (n = 144; 92%) and FC (n = 146; 90%) did not differ in terms of their scores on the MSLQ’s intrinsic goal orientation, extrinsic goal orientation, task value, elaboration, or critical thinking domains (Table 2).
The mean OSCE score increased by 12.33 points (95% CI 10.28–14.38), from 67.01 (SD 9.6) of 100 possible points in the TC to 79.34 (SD 9.0) in the FC (P < .0001), with an effect size of 1.33. There was also a significant shift in the OSCE grade distribution, with more FC students earning an A, B, or C and fewer earning a D or F compared with the TC students (P < .001) (Figure 2). The FC students had a higher mean score than the TC students on the information-gathering skills checklist (84.0% [SD 12.4] vs. 73.6% [SD 16.1], P < .0001). Significantly more FC students than TC students asked the SP about previous treatments for pain, precipitating factors, functional impact, and tolerable pain level (P < .05). Students in the TC and FC were equally likely to ask questions about the location, severity, and quality of the pain as well as the temporal pattern and palliating factors (see Supplemental Digital Appendix 1A at http://links.lww.com/ACADMED/A453).
FC students also achieved a higher mean score than TC students on the OSCE SOAP note (64.0% [SD 19.7] vs. 30.8% [SD 12.9], P < .001), with improvements in documentation, assessment, and management strategies (see Supplemental Digital Appendix 1B–D at http://links.lww.com/ACADMED/A453). A modest 9.5-percentage-point bias was detected between the two graders on the 32 SOAP notes that both rescored in a blinded manner. However, the mean scores for these 32 SOAP notes were significantly and consistently higher in the FC cohort, regardless of grader:
Grader 1 (nonspecialist): FC = 69.3% (SD 21.9) vs. TC = 39.8% (SD 10.8), P < .001
Grader 2 (pain specialist): FC = 57.9% (SD 21.2) vs. TC = 32.3% (SD 10.1), P = .001
There was no significant difference between the TC and FC in the distribution of overall global rating scale scores (P = .29). However, on the empathy subscale of the global rating scale, more FC students than TC students were rated as “responds consistently in a perceptive and genuine manner to the patient’s needs and cues” (41.0% vs. 34.6%, P = .047).
Multiple-choice exam score
Student performance on the pain therapeutics content of the end-of-module multiple-choice examination improved by 5.07 percentage points (95% CI 2.56–7.59) from a mean of 77.23% (SD 12.43) in the TC to 82.30% (SD 10.25) in the FC, P < .0001, with an effect size of 0.45. There were no significant differences in performance on the exam’s pharmacology, medicinal chemistry, and toxicology content (TC mean = 76.74% [SD 12.81] vs. FC mean = 78.66% [SD 11.14], P = .154). Grade distribution on the multiple-choice exam shifted significantly: FC students were more likely to earn an A or B and less likely to earn a C, D, or F than TC students (P = .0005) (Figure 3). In addition, FC student performance improved on items assessing higher cognitive levels, and more FC students correctly answered items related to designing an equianalgesic opioid dosing regimen (P < .05; see Supplemental Digital Appendix 2A–B at http://links.lww.com/ACADMED/A454).
The FC’s improvement in overall performance on the pain therapeutics section of the exam was attributed to improved performance on seven individual questions (P < .05; see Supplemental Digital Appendix 2A at http://links.lww.com/ACADMED/A454).
This study’s results add to the growing body of evidence that student-centered, active learning approaches often result in improved skill development and similar or better knowledge acquisition. Flipping the pain therapeutics content of the pain management module in the University of Maryland School of Pharmacy’s first-professional-year PharmD curriculum improved performance on both the end-of-module OSCE and multiple-choice exam.
OSCEs are used in nursing, medicine, and pharmacy curricula to evaluate clinical skills and readiness. Although OSCEs are not yet a component of licensure for nurses or pharmacists, Step 2 Clinical Skills of the United States Medical Licensing Examination uses OSCEs to assess clinical competency of medical students and medical school graduates.33 In this study, we demonstrated that flipping a didactic classroom resulted in improved clinical skills, as assessed by an OSCE. The magnitude of improvement in the mean OSCE score observed in this study is comparable to that seen in a recent study evaluating a flipped clerkship in obstetrics–gynecology, which also compared flipped and traditional cohorts of medical students.34 In addition to an improvement in mean OSCE score, OSCE grade distribution shifted in our study, with significantly fewer Fs and Ds and more As and Bs earned in the FC than in the TC.
We did not determine whether certain student demographic groups performed better in the FC in this study. However, given recent findings that flipped approaches may benefit women and students with lower GPAs disproportionately, this is an important area to explore in future studies.35
In a previous study evaluating the OSCE as an assessment strategy for problem-based learning, pharmacy student performance on OSCEs and multiple-choice exams was not correlated.36 In another study evaluating the flipped classroom across 10 courses at one school of pharmacy, students noted a discrepancy between the focus on application in the flipped classroom and assessment using multiple-choice exams.37 In our study, we saw improved performance on the pain therapeutics section of the multiple-choice examination after flipping the module. These performance gains were attributed to seven questions that students in the FC performed significantly better on, five of which assessed students at higher cognitive levels (e.g., application, analysis, creation). Skill in writing multiple-choice questions that assess higher cognitive levels is variable among professors and may account for differences in exam performance seen across studies.38–41 However, as Miller’s2 pyramid suggests, a multiple-choice exam may not be the optimal assessment tool to evaluate the impact of student-centered learning.
Our results indicate that transforming didactic content from a teacher-centered model to a learner-centered model of teaching and learning may better prepare health professions students for clinical practice. However, flipping our pain management module likely increased the time students spent with the pain therapeutics content if they followed the recommended preclass learning opportunities and participated in all in-class activities. While other effective flipped interventions have similarly increased content contact time, some educators have decreased in-class time to balance out the potential increased preclass learning time.21,42 For example, in one study, there were no significant differences in multiple-choice exam performance between students in flipped and traditional oncology modules when in-class time was decreased by four hours and preclass assignment time was increased by eight hours.42
Educators should not let concerns about student nonadherence to learning activities be a deterrent to shifting content to a learner-centered model. We did not measure engagement in preclass, in-class, or postclass learning activities in either the TC or FC, and therefore we do not know the actual time students spent engaging with the material. No incentives were provided for completing preclass learning activities, nor were graded quizzes administered in either the TC or FC. This was a purposeful choice to promote intrinsic motivation and to diminish the number of graded assignments that might increase student stress. In other studies, students noted competing demands on their time that made it challenging to complete preclass learning activities.37,43 In the fast-paced PharmD curriculum at the University of Maryland, which includes many recorded lectures, we have observed that students are prone to falling into a pattern of studying immediately prior to the next graded assignment, engaging in so-called binge-and-purge learning. Although we cannot conclude that students in the FC were more or less engaged than those in the TC, improvements in performance among the FC as a whole were significant when we shifted the module from a teacher-centered to a learner-centered design. It would be helpful in future studies to determine if engagement in preclass activities or increased time with material explain the observed improvements in performance.
We acknowledge there are several potential limitations to this study. The study was performed at a single institution and included a single OSCE station. In addition, while we report effect sizes, this was a multifaceted intervention that involved a comprehensive course redesign with an array of new learning resources provided to students as well as increased opportunities for group feedback. Thus, we are unable to determine which elements of the intervention were most impactful.
Course redesign is resource intensive, and the cost versus benefit is debated.44–46 We did not measure faculty time or resource use during this study. However, the instructional activities were developed and implemented with existing resources and instructional personnel. Therefore, we believe the instructional methods described in this report can be feasibly implemented at other health professions schools.
The improvements in both knowledge and skill observed in this and other studies14,17,22–29,34 suggest that the widespread use of learner-centered models for classroom-based instruction in health professions curricula has the potential to substantially increase learner readiness for clinical practice.
The authors acknowledge the following individuals: Sandeep Devabhakthuni, Thomas C. Wyatt, and Kathy Schiavone for assistance in developing, peer reviewing, and implementing the OSCE; Lisa Lebovitz, JD, for compiling demographic data for the classes of 2018 and 2019; and Jill Morgan, PharmD, BCPS, department chair, Pharmacy Practice and Science, for supporting this project.
1. Frenk J, Chen L, Bhutta ZA, et al. Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376:19231958.
2. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 suppl):S63S67.
3. Miller BM, Moore DE Jr, Stead WW, Balser JR. Beyond Flexner: A new model for continuous learning in the health professions. Acad Med. 2010;85:266272.
4. Speedie MK, Baldwin JN, Carter RA, Raehl CL, Yanchick VA, Maine LL. Cultivating “habits of mind” in the scholarly pharmacy clinician: Report of the 2011–12 Argus Commission. Am J Pharm Educ. 2012;76:S3.
5. Irby DM, Cooke M, O’Brien BC. Calls for reform of medical education by the Carnegie Foundation for the Advancement of Teaching: 1910 and 2010. Acad Med. 2010;85:220227.
6. Mehta NB, Hull AL, Young JB, Stoller JK. Just imagine: New paradigms for medical education. Acad Med. 2013;88:14181423.
7. Gleason BL, Peeters MJ, Resman-Targoff BH, et al. An active-learning strategies primer for achieving ability-based educational outcomes. Am J Pharm Educ. 2011;75:186.
8. Blouin RA, Joyner PU, Pollack GM. Preparing for a Renaissance in pharmacy education: The need, opportunity, and capacity for change. Am J Pharm Educ. 2008;72:42.
9. Institute of Medicine. A Summary of the February 2010 Forum on the Future of Nursing: Education. 2010.Washington, DC: National Academies Press.
10. Weimer M. Learner-centered teaching: Roots and origins. In: Learner-Centered Teaching: Five Key Changes to Practice. 2013:2nd ed. San Francisco, CA: Jossey-Bass; 327.
13. Critz CM, Knight D. Using the flipped classroom in graduate nursing education. Nurse Educ. 2013;38:210213.
14. O’Connor EE, Fried J, McNulty N, et al. Flipping radiology education right side up. Acad Radiol. 2016;23:810822.
15. Schwartz TA. Flipping the statistics classroom in nursing education. J Nurs Educ. 2014;53:199206.
16. Simpson V, Richards E. Flipping the classroom to teach population health: Increasing the relevance. Nurse Educ Pract. 2015;15:162167.
17. Liebert CA, Lin DT, Mazer LM, Bereknyei S, Lau JN. Effectiveness of the Surgery Core Clerkship Flipped Classroom: A prospective cohort trial. Am J Surg. 2016;211:451457.e1.
18. Tan E, Brainard A, Larkin GL. Acceptability of the flipped classroom approach for in-house teaching in emergency medicine. Emerg Med Australas. 2015;27:453459.
19. Young TP, Bailey CJ, Guptill M, Thorp AW, Thomas TL. The flipped classroom: A modality for mixed asynchronous and synchronous learning in a residency program. West J Emerg Med. 2014;15:938944.
20. McLaughlin JE, Griffin LM, Esserman DA, et al. Pharmacy student engagement, performance, and perception in a flipped satellite classroom. Am J Pharm Educ. 2013;77:196.
21. McLaughlin JE, Roth MT, Glatt DM, et al. The flipped classroom: A course redesign to foster learning and engagement in a health professions school. Acad Med. 2014;89:236243.
22. Wong TH, Ip EJ, Lopes I, Rajagopalan V. Pharmacy students’ performance and perceptions in a flipped teaching pilot on cardiac arrhythmias. Am J Pharm Educ. 2014;78:185.
23. Presti CR. The flipped learning approach in nursing education: A literature review. J Nurs Educ. 2016;55:252257.
24. Betihavas V, Bridgman H, Kornhaber R, Cross M. The evidence for “flipping out”: A systematic review of the flipped classroom in nursing education. Nurse Educ Today. 2016;38:1521.
25. Della Ratta CB. Flipping the classroom with team-based learning in undergraduate nursing education. Nurse Educ. 2015;40:7174.
26. Missildine K, Fountain R, Summers L, Gosselin K. Flipping the classroom to improve student performance and satisfaction. J Nurs Educ. 2013;52:597599.
27. Persky AM, Dupuis RE. An eight-year retrospective study in “flipped” pharmacokinetics courses. Am J Pharm Educ. 2014;78:190.
28. Patterson BJ, Geist MJ, Larimore D, Rawiszer H, Al Sager AW. Flipped versus traditional instruction and achievement in a baccalaureate nursing pharmacology course. Nurs Educ Perspect. 2015;36(2):114115.
29. Ferreri SP, O’Connor SK. Redesign of a large lecture course into a small-group learning course. Am J Pharm Educ. 2013;77:13.
30. Harrington SA, Bosch MV, Schoofs N, Beel-Bates C, Anderson K. Quantitative outcomes for nursing students in a flipped classroom. Nurs Educ Perspect. 2015;36(3):179181.
31. Hodges B, McIlroy JH. Analytic global OSCE ratings are sensitive to level of training. Med Educ. 2003;37:10121016.
32. Pintrich PRR, Smith D, Garcia T, McKeachie W. A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ). 1991. Ann Arbor, MI: National Center for Research to Improve Postsecondary Teaching and Learning; Report no. NCRIPTAL-91-B-004. https://eric.ed.gov/?id=ED338122
. Accessed March 22, 2017.
34. Gillispie V. Using the flipped classroom to bridge the gap to Generation Y. Ochsner J. 2016;16:3236.
35. Gross D, Pietri ES, Anderson G, Moyano-Camihort K, Graham MJ. Increased preclass preparation underlies student outcome improvement in the flipped classroom. CBE Life Sci Educ. 2015;14:ar36.
36. Salinitri FD, O’Connell MB, Garwood CL, Lehr VT, Abdallah K. An objective structured clinical examination to assess problem-based learning. Am J Pharm Educ. 2012;76:44.
37. Khanova J, Roth MT, Rodgers JE, McLaughlin JE. Student experiences across multiple flipped courses in a single curriculum. Med Educ. 2015;49:10381048.
38. Downing SM. The effects of violating standard item writing principles on tests and students: The consequences of using flawed test items on achievement examinations in medical education. Adv Health Sci Educ Theory Pract. 2005;10:133143.
39. Kibble JD, Johnson T. Are faculty predictions or item taxonomies useful for estimating the outcome of multiple-choice examinations? Adv Physiol Educ. 2011;35:396401.
40. Zheng AY, Lawhorn JK, Lumley T, Freeman S. Application of Bloom’s taxonomy debunks the “MCAT myth.” Science. 2008;319(5862):414.
41. Cramer N, Asmar A, Gorman L, et al. Application of a utility analysis to evaluate a novel assessment tool for clinically oriented physiology and pharmacology. Adv Physiol Educ. 2016;40:304312.
42. Bossaer JB, Panus P, Stewart DW, Hagemeier NE, George J. Student performance in a pharmacotherapy oncology module before and after flipping the classroom. Am J Pharm Educ. 2016;80:31.
43. Khanova J, McLaughlin JE, Rhoney DH, Roth MT, Harris S. Student perceptions of a flipped pharmacotherapy course. Am J Pharm Educ. 2015;79:140.
44. Persky AM. Cost–benefit of instructional strategies. Am J Pharm Educ. 2014;78:145.
45. Spangler J. Costs related to a flipped classroom. Acad Med. 2014;89:1429.
46. McLaughlin JE, Roth MT, Mumper RJ. In reply to Spangler. Acad Med. 2014;89:14291430.
Supplemental Digital Content
Copyright © 2017 by the Association of American Medical Colleges