Journal Logo

Research Article: Quality Improvement Study

Pediatric Online Evidence-Based Medicine Assignment Is a Novel Effective Enjoyable Undergraduate Medical Teaching Tool

A SQUIRE Compliant Study

Kotb, Magd A. MSc, MD; Elmahdy, Hesham Nabeh MSc, PhD; Khalifa, Nour El Deen Mahmoud MSc, PhD; El-Deen, Mohamed Hamed Nasr MSc, PhD; Lotfi, Mohamed Amr N. MSc, MD

Section Editor(s): Raouf, Hajji.

Author Information
doi: 10.1097/MD.0000000000001178

Abstract

Erratum

In the article “Pediatric Online Evidence-Based Medicine Assignment Is a Novel Effective Enjoyable Undergraduate Medical Teaching Tool: A SQUIRE Compliant Study, which appeared in Volume 94, Issue 29 of , Dr. Kotb's and Dr. Lotfi's MD degrees were originally omitted. The article has been corrected online.

Medicine. 94(32):e7495, August 2015.

INTRODUCTION

Evidence-based medicine (EBM) is defined as the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.1,2 The lag between the development of medical sciences and practice spans several years.3 It was found essential to incorporate EBM in curricula of medical schools, with stressing upon decision analysis.4,5 EBM is delivered through a didactic, blended learning, and mixed models. Students are supposed to construct an answerable question in PICO (patient, intervention, comparison, and outcome) framework, acquire evidence through search of literature, appraise evidence, apply it to the clinical case scenario, and assess the evidence in relation to clinical context.6

Faculty of Medicine, Cairo University is challenged by the massive number of students enrolled in its undergraduate years, with a total of 9664 to 8687 undergraduate students across its 6 undergraduate years in the academic year 2004 to 2005 and 2010 to 2011, respectively.7

Pediatrics Evidence-Based Medicine Assignment (PEBMA) was delivered didactically by competent pediatrics staff members trained in EBM teaching in Department of Pediatrics, Faculty of Medicine, Cairo University until 2011 to 2012 academic year. Students worked in 10 to 15 student groups, yet group work has its limitations in handling uncooperative students, their impact upon progress of their groups, students who fail to contribute, students who domineer, students who have personal conflict, and inconsistent individual acquisition of required skills.8

We developed a web-based online system to allow undergraduate medical student individual acquisition of PEBMA skills, allow work in group and overcome limitations of group work.9 This study aimed at assessing the effectiveness of the newly developed online system as an educational tool of EBM skills among 5th year students of Pediatrics Course, Cairo University.

MATERIAL AND METHODS

Design, Setting, and Participants

PEBMA is a part of curriculum of Pediatrics Course delivered in 5th undergraduate year. Shift from PEBMA didactic model to website was approved by Department of Pediatrics Staff Council in July 2012. We followed this educational intervention prospectively in Pediatrics Department, Cairo University during the academic years 2012 to 2013 and 2013 to 2014 to assess effectiveness of PEBMA website as a tool for EBM education for 5th year Pediatrics Course students.

Participants of this study were undergraduate medical students of 5th year Pediatrics Course.

In a given academic year, students were divided into 2 independent groups to join November or March semesters. Participants of this study were 2897 students. PEBMA spanned 7 weeks.

Interventions

Structural

We developed an interactive E-medical assignment portal that is a web-based information system, which uses Internet web technologies to deliver information, services, and statistics to the system stakeholders (students, demonstrators/facilitators, and system administrator). The power of the system resides in the lack of need to install software to navigate the system. It functions by a web browser and an Internet connection which exist already in all contemporary computers and smart devices.9

The portal consisted of 3 different layers; the first layer is the database layer, which contains all related information about the system that includes the medical case scenarios, questions, and information about stakeholders and their transactions using Open Source Database Technology (My Structured Query Language). The second layer is the interface design using web technology (Hypertext Markup Language and Cascading Style Sheets), this technology was used to build the portal interface screens for system stakeholders. The third layer was the system workflow transactions using Open Source Development Language (Personal Home Page Tools), this layer contains the brain of the system to guide and interact with system stakeholder to accomplish their tasks throughout the system.

Functional

Students received a 2 hour orientation lecture of EBM of website and its functions. Optional hands-on training involving system navigation and expected tasks were offered.

Students login by their identification card number as a username and password to be changed by student as wished. Difficulty to login was responded by an automatic message to admin requesting username and password to be sent to an email address.

PEBMA comprises acquisition of knowledge and skills that undergo criterion-based assessment. To assure individual acquisition of skills, students accessed through a login screen, filled personal profile page that was followed by multiple screens to access tutorials. Tutorials were accessible without login as well.

Students found an external link to PICO question tutorial and model answers of single and group assignments. Screens were designed to provide scholar knowledge of EBM, that is, describe concept of EBM, how to formulate a search question (PICO question), search literature, and define the results of search in terms of level of evidence. Each student was expected to submit a single assignment comprising a structured referenced summary of maximum 400 words that contains her/his argument substantiated and decision.

After deadline for submission of the single assignment is met, students were made familiar with their group members, that is, the others who answered the other questions in a same case scenario. Students were made familiar with their achievement in the single assignment often with feedback from demonstrators/facilitators. Groups were assigned randomly by e-system. Group members had to nominate a leader and work together to prepare a structured summary. Only the leader was allowed to submit group assignment on behalf of the group members, and all had to submit their contribution in a discussion platform that was visualized by demonstrator. Students were expected to work in groups and submit written referenced structured summary of maximum 800 words, design and describe a plan for future research relevant to their case scenario subject. The EBM should foster moral-based choices, and students were encouraged to check their work for plagiarism; make individualized choices to suit the patient care in the specific scenario within a given time limit; makes weighted decisions by comparing evidence and benefits; and build self-confidence by supporting decision by argument. Students put their contribution in the discussion panel to allow delineate those who shared and annulled the effect of uncooperative students. The total assignment duration was divided into a single assignment and a group assignment. Case scenario questions had 3 types of answers, positive with evidence to support, negative with evidence against, or evidence was not enough.

The system accumulated 180 case scenarios with a total of 1804 questions. Case scenarios were added twice yearly. For every case scenario there are a group of questions related to clinical picture, management, and prognosis. The system allows the student to login, read instructions, and read a specific case scenario. Each student has to answer a single question which is computer generated. The student finds links to databases10–13 and can submit the PICO question that identifies patient population/problem (P), intervention (I), comparison parameter (C), and measurable outcome (O), referenced summary, make changes, and save them. A student can get help through the get help box, where the student can compose the questions and receive answers. Students were encouraged to comment or make suggestions to improve the system and/or the assignment.

Students were allowed to engage in conversations in the group assignment through “discussion platform” online function. In group assignment, students have to describe their contribution in discussion platform in group assignment to get the mark in the group assignment.

The system allowed demonstrators/facilitators to login. For each of the 4 pediatrics courses 24 demonstrators were assigned to PEBMA. Each demonstrator had a group of computer-assigned students. Demonstrators coached and facilitated through the get help box, met students upon request from students, and corrected the submitted single and group assignments through a criterion-based system.

The Professor of Pediatrics in charge of coordinating the PEBMA during the Pediatrics Round is the admin of the e-system. Admin screens provide reports and link to all transactions on web including number of visits of all, demonstrators’ time to correct single and group assignments, comments report, final mark list, and allow search for individual specific student results, change password and/or username upon student request, and view in box messages and responses. Students were invited to seek help from admin through admin e-mail address.

Main Outcomes and Measures

Main Outcomes

  1. Students acquisition of the following EBM skills:
    1. Understanding concept of EBM.
    2. Formulating the relevant question in the PICO framework.
    3. Searching literature for evidence systematically and efficiently.
    4. Appraising the evidence.
    5. Applying the evidence to the specific clinical scenario.
    6. Decision making and providing argument to support decision.
  2. Students acquisition of the following presentation skills:
    1. Structuring answer and argument in abstract form abiding by word count (400 word abstract for single assignment and 800 words for group assignment), putting a title to work, formulating an introduction, describing their search methodology, describing results, and making an evidence-based recommendation and conclusion.
    2. Citing references.
    3. Abiding by instructions.
  3. Building an e-system that:
    1. Suits average student and delivers the intended learning outcomes.
    2. Delivers 24 hours/7day synchronized consecutive functions.
    3. Effective in mass education.
    4. Allows criterion-based assessment of students’ performance.
    5. Delivers assessment feedback to students.
    6. Delivers timely reports and statistical data.
    7. Delivers students’ comments and anonymous questionnaire.

Measures

We assessed the system through the students’ grades, number of engagements online, and students’ end of assignment feedback questionnaire.

Students’ performances in single and group assignments were subjected to analysis.

For single assignment (total 2.5 marks) 0.5 marks were assigned for each of the following:

  1. Student formulated a PICO question and defined the PICO elements correctly.
  2. Student answered his/her relevant question.
  3. Student followed instructions in preparing the 400 words abstract.
  4. References content is congruent with results and cited in Vancouver style.
  5. Conclusion and student recommendation tailored to case scenario are evidence based.

For group assignment (total 2.5 marks) 0.5 marks were assigned for each of the following:

  1. Students answered their relevant question.
  2. Students followed instructions in preparing the 800 words abstract.
  3. References content is congruent with results and cited in Vancouver style.
  4. Conclusion and recommendation tailored to case scenario are evidence based.
  5. Students forwarded a future research plan with defined patient population, intervention, comparison, and measurable outcome.

Students received a total of 5 marks divided as 2.5 for single and 2.5 for group assignment. Students were warned against plagiarism and instructed that they get a 0 in case of plagiarism, irrespectively. Our assessment meets reported validation properties,14 that is, content validity, internal reliability, item difficulty, item discrimination, and construct validity. Interrater reliability was not studied in PEBMA and is a part of future work.

Students were encouraged to answer an assignment anonymous feedback questionnaire by end of pediatrics course regarding website navigation, login, single, and group assignment, and the subjective impressions left upon the students. The screen of feedback anonymous questionnaire appeared only after students’ completely submitted their work and it was assessed to assure lack of bias or intimidation of students.

Statistical Analysis

All the statistical analyses in this study were conducted using Statistical Package for Social Sciences version 16 (SPSS, Chicago, IL). Descriptive analysis was used for data summarization. The parametric quantitative data were expressed as mean ± standard deviation (SD).

The simple frequency and cross-tabulation were employed for the qualitative data. Suitable tests of significance (t-test for parametric data and Chi-square [χ2] tests for nonparametric numbers N5) were used as indicated. Comparisons were carried upon number of students in single and group assignment.

RESULTS

A total of 2879 were enrolled in 4 pediatrics courses during 2012 to 2013 and 2013 to 2014 academic years. Of them 2779 (96.5%) logged in and only 2554 (88.7%) submitted their work. They were randomly assigned by system to 292 groups. Forty nine demonstrators participated in PEBMA across both academic years.

Students’ Performance in Single and Group Assignments

Total number of login of students was 55,886 (mean ± SD = 38.8 ± 15 per student), total number of demonstrators login was 374 (mean per demonstrator = 15 ± 6.9), and admin was 622. Students and demonstrators engaged in 1880 online conversations, of them 1023 (54.4%) were composed messages by students, and 857 (45.58%) replies by demonstrators. Students requested login information (username/password) and emailed 4349 email messages to admin on the admin announced email address. Admin responded to all, but only 3783 (86.98%) were communicated while admin failed to communicate response to 566 (13%) messages requesting username and password to login, as student correspondence email address was not valid. Table 1 shows various numbers of e-system transactions.

TABLE 1
TABLE 1:
System Recorded Transactions During the Academic Years 2012 to 2013 and 2013 to 2014

Assignment marks comprised a total of 5 marks divided as 2.5 for single and 2.5 for group assignment. Only 20 (0.78%) students of those who submitted their work failed to achieve a total of 2.5/5 (50%), while 2277 (89.15%) achieved 80% or more of total mark (4/5), of them 717 (28.1%) achieved a full mark. Table 2 presents assessed single and group assignment parameters. Mean ± SD of single assignment was 2.16 ± 0.49 and of group assignment was 2.38 ± 0.23 (P < 0.001). The individual activity of the students reflected by number of student login, correlated positively with all the 5 single assignment parameters (P < 0.001), their total grade (P < 0.001). On the contrary, frequency of login did not correlate with group assignment parameters except grade for answering relevant questions (P = 0.049). A total of 225 students did not submit their work and got a 0.

TABLE 2
TABLE 2:
Students in Single and Group Assignments Assessed Parameters

Students’ Responses to Anonymous Questionnaire

Students who responded to invitation to answer questionnaire were 1102 (43.1%). A total of 898 (81.48%) the e-educational experience satisfactory, 175 (15.88%) disagreed, and 29 (2.6%) could not decide. Figure 1 describes the various responses of student's. It is interesting that 794 (72.3%) found the overall experience enjoyable and 707 (72.2%) found the website links to databases enough.

FIGURE 1
FIGURE 1:
Student's responses to end of assignment feedback questionnaire.

DISCUSSION

The novel interactive electronic system is invaluable in teaching EBM in 5th year Pediatrics Course, Faculty of Medicine, Cairo University.

It assured that each student understood, practiced, and applied the knowledge and skills of EBM, presentation skills, and worked in group. The PEBMA capitalized on “the need to know,” fostered information acquisition, decision making, developed presentation skills, and supported arguments abiding by limits of instructions. It allowed students to construct new knowledge, assured individual active participation and acquisition of skills, allowed participation in group work, enforced a code of moral conduct, and enabled confident decision-making. PEBMA targeted the average student and was efficient in mass education without compromising quality of education. It allowed continuous surveillance and monitoring affording prompt intervention. It accommodated a large bank of questions that did not allow duplication of scenarios, giving each student a different task, stores student responses, provided future research plans bank, and allowed students to participate with their comments and feedback. The sequence of tasks allowed students group-work to follow mastering skills by individual student.

The newly devised e-system exploited the advantages afforded by the accurate, accessible, and continually updated sources of evidence such as the Cochrane Library and Clinical Evidence,15 PubMed database, and others that contain all types of reports, even those containing contradictory evidence that augment students’ power to appraise and of EBM e-learning sites.15–19

It is interesting to note that frequency of students login correlated with single assignment parameters, which highlights the engagement of students in self-acquisition of concept, search, appraisal, application, assessment, and presentation of EBM assignment. PEBMA challenged a minority of students (14.7%) who were inclined to resistance to precision and tended to ambiguity in tailoring their conclusions and recommendation for their specific case scenario.

Students’ higher marks in group assignment might be attributed to group effect education, learning from mistakes in single assignment and gaining more insight by practice.

The newly designed e-system is technically demanding, requires space, effort, maintenance, feeding of question bank twice annually and needs continuous surveillance for server malfunction, bugs, password losses, and student personal information inaccuracy. It demands too much work of the PEBMA staff team when they are overloaded by other teaching and training duties for both under and postgraduates. It pressurized students to shift from didactic into interactive and mostly self-education. The PEBMA team had to face students’ uprising frustrations and resistance at initial implementation that faded by time.

It is rewarding that 794 (72.3%) students found the novel e-system enjoyable.

Sustainability of PEBMA remains a great challenge, and the impact of PEBMA upon personal attitudes of students in their postgraduate practice remains to be known and cannot be foreseen. Reinforcement by subsequent practice remains the barrier against loss of acquired skills, that is, workshops, seminars, journal clubs (or any combination), standalone, and integrated methods to improve knowledge, skills, attitudes, behavior,20 and on-the-job EBM training.21,22

In conclusion, we found that PEBMA website is effective in teaching undergraduate medical students. It assures individual student acquisition of concepts and skills of pediatric EBM especially in courses enrolling large number of students.

Acknowledgments

The authors thank the advice of Professor Mona El Lawindi, Professor of Public Health and Biostatistics, Faculty of Medicine, Cairo University, and late Professor Ahmed Kotb, Professor of Paediatrics, Faculty of Medicine, Cairo University; thank efforts of cofounder of Pediatrics EBM discipline Professor Ghada Anwar, Professor of Pediatrics, Pediatric Diabetes and Endocrinology, Cairo University, she is awarded Diploma in Health Profession Education, and is an International certified trainer, an Associate Editor in Egyptian Pediatric Association Gazette, and Member of European Society for Pediatric Endocrinology (ESPE) American Heart Association Instructor for Pediatric Advanced Life Support (PALS). The authors also thank Pediatric Evidence-Based Medicine Assignment Team of demonstrators Abd El Monem Mostafa, Ahmed El Beheiry, Ahmed Sobhy, Alaa Gamal, Aly EL Kazaz, Amr Mohamed Salem Reda, Amir Fawzy, Antoine Fakhry, Carolin Morcos, Dalia Salah, Dina Moheb, Doaa Al Saeed, Doaa Meshref, Doaa Salah, Esraa Ahmed, Fatma Mostafa, Hala Hussein, Hala Mohsen Abd El Salam, Hanaa Rady, Hanan Naguib, Heba Raslan, Hend Mohamed, Haytham Ghita, Hossam Ibrahim, Iman Abo Bakr, Khaled Amin, Khaled Shams, Mai Abd El Salam, Mariam Saad, Miriam Magdy, Mohamed Gamal, Mona Ahmed Kamel, Noha Arafa, Nermeen Galal, Niveen Baroody, Noha Saber, Nour El Khatib, Radwa Iraky, Rasha Helmy, Rasha Rady, Salwa Al Azhary, Samah Ahmad, Samira Abd El Monem, Samia Bekhit, Sara Ali Saleh, Shaymaa Salah, Sherif Baroody, Yasmine Saeed Ali, and Yomna Shalaan.

REFERENCES

1. Evidence-Based Medicine Working GroupEvidence-based medicine. A new approach to teaching the practice of medicine. JAMA 1992; 268:2420–2425.
2. Komatsu RS. Evidence based medicine is the conscientious, explicit, and judicious use of current evidence in making decisions about the care of individual patients. Sao Paulo Med J 1996; 114:1190–1191.
3. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10:523–530.
4. Dowie J. Evidence based medicine. Needs to be within framework of decision making based on decision analysis. BMJ 1996; 313:170.
5. Muckart DJ. Evidence-based medicine – are we boiling the frog? S Afr Med J 2013; 103:447–448.
6. Ilic D, Bin Nordin R, Glasziou P, et al. Implementation of a blended learning approach to teaching evidence based practice: a protocol for a mixed methods study. BMC Med Educ 2013; 13:170.
7. El Feky N International Students at Faculty of Medicine-Cairo University 2006–2011. http://www.medicine.cu.edu.eg/beta/images/stories/docs/international_studentsfinal.pdf. [Accessed September 7, 2014].
8. Sharan Y. Cooperative learning for academic and social gains: valued pedagogy, problematic practice. Eur J Educ 2010; 45:300–313.
9. Faculty of Medicine Pediatric Department. Pediatric evidence based medicine assignment project. http://medical.i-sciences.com/ [Accessed September 9, 2014].
10. Cochrane Library Summaries. http://summaries.cochrane.org/ [Accessed September 20, 2014].
11. BMJ best practice. http://bestpractice.bmj.com/best-practice/welcome.html [Accessed September 20, 2014].
12. Trip database. http://www.tripdatabase.com/ [Accessed September 20, 2014].
13. Pub Med. US National Library of Medicine national Institutes of Health. http://www.ncbi.nlm.nih.gov/pubmed. [Accessed September 20, 2014].
14. Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence based medicine. BMJ 2003; 326:319–321.
15. The KT Clearinghouse, Centre for evidence based medicine, Toronto, Canada: http://ktclearinghouse.ca/ [Accessed September 20, 2014].
16. Centre for evidence based medicine, Oxford, UK: http://www.cebm.net/ [Accessed September 20, 2014].
17. EBM teaching material at Dartmouth Biomedical libraries: http://www.dartmouth.edu/∼library/biomed/guides/research/ebm-teach.html/ [Accessed September 20, 2014].
18. McMaster evidence-based practice workshop, course materials: http://ebm.mcmaster.ca/materials_intro.htm/ [Accessed on September 20, 2014].
19. Duke EBM workshop materials: http://sites.duke.edu/ebmworkshop/materials/ [Accessed on September 20, 2014].
20. Coomarasamy A, Khan KS. What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ 2004; 329:1017.
21. Leung EY, Malick SM, Khan KS. EBM-CONNECT CollaborationOn-the-Job Evidence-Based Medicine Training for Clinician-Scientists of the Next Generation. Clin Biochem Rev 2013; 34:93–103.
22. Straus SE, Green M, Bell D, et al. Evaluation of evidence based health care educational interventions: conceptual framework. BMJ 2004; 329:1029–1032.
Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.