Evidence-based medicine (EBM) involves integrating individual clinical expertise with the best available evidence from scientific literature.[1,2] Patient values and preferences are also incorporated into EBM to promote shared decision-making between patients and clinicians.[1,3] Implementing an evidence-based approach has strong advantages, such as increased cost-effectiveness, by minimizing clinical practices that have limited proven benefit.[4–7]
Failure to use evidence from research to make informed decisions is evident among healthcare providers, managers, and policy-makers, across all disciplines of care around the world.[6,8–10] The practice of EBM is further constrained in developing countries due to its inherent complexity, misperceptions, absence in medical curricula, and limited awareness among clinicians. In resource-poor countries, there is limited access to databases/computers/Internet, limited literature relevant to local realities, and inadequate library facilities.[7,11–18] In an effort to improve healthcare quality and physicians’ decision-making to promote optimal outcomes for patients, the use of EBM to inform healthcare practice has emerged as both a national and international priority.[19–22]
South Asian countries, in particular, have demonstrated limited uptake of EBM.[4,23] Given the often large patient load among Indian physicians, clinicians have limited time and incentives to stay up-to-date with the latest breakthroughs and innovations; consequently, they may not always provide optimal care.
One major barrier to implementing EBM in orthopaedic surgery specifically is the lack of summarized evidence that is available in a useful and acceptable format. The objectives of this study were to: investigate the accessibility, use, and impact of an online EBM knowledge dissemination portal for orthopaedic surgeons in India, explore whether receiving daily targeted evidence summaries results in increased usage of an EBM tool when compared to receiving general weekly reports, and identify and explain the barriers and benefits of an EBM resource in the Indian context.
At the Sancheti Institute of Orthopaedics and Rehabilitation (SIOR), a private orthopaedic hospital in Pune, India, approximately 54,000 patients are seen as outpatients, while 5,500 are seen as inpatients annually.
To be eligible for the study, participants had to be orthopaedic surgeons in training or in practice, full-time employees of the SIOR, and able to read and write in English. Ethics approval from McMaster University's Hamilton Integrated Research Ethics Board Student Research Committee and the SIOR's Ethics Committee was obtained.
Orthopaedic surgeons (n = 44) at the SIOR were provided free access to OE (www.myorthoevidence.com), a for-profit and online EBM knowledge dissemination portal. OE gathers high-impact peer-reviewed journal articles, summarizes them, and then sends out Advanced Clinical Evidence (ACE) reports to subscribers via email. ACE reports are approximately 1 to 2 pages and provide a study overview by answering the following questions: Why is the research needed now? What is the principal research question? What are the important findings? What should I remember most? How will this affect the care of patients?
For the benefit of the surgeons participating in this study, their access to OE continued for a total period of 2 years.
3.1 Explanatory sequential mixed methods design
This study employed an explanatory sequential mixed-methods design, incorporating 2 questionnaires, OE usage data, and semi-structured interviews in order to gain insight into the surgeons’ usage and perceptions, as well as the impact of OE. The sequential explanatory mixed methods study design occurred in 2 distinct phases. Quantitative data was first collected and analyzed using an initial survey, exit survey, and participants’ OE usage data. The quantitative data and its analysis provided a general understanding of the research question. The quantitative data then guided purposeful sampling for the qualitative phase. Qualitative data collected through semi-structured interviews was analyzed in order to better understand the quantitative findings. The qualitative data specifically aided in refining and explaining the statistical results by exploring participants’ perceptions about OE in depth.
3.2 The OrthoEvidence intervention
Stratified random sampling was used to distribute the study participants. Participants were stratified into their respective professional occupations—surgical resident trainees, attending physicians (consultants), and visiting fellows. Participants were then randomized into a group receiving daily targeted ACE reports directed toward SIOR's orthopaedic practice, or a group receiving general weekly ACE reports. Stratification by level of training ensured that an equal number of surgical resident trainees, consultants, and visiting fellows were in each arm of the study for comparison purposes. See Figure 1 for a flow diagram of study participants.
Participants in both groups received free, full online access to the OE website and could use the site as they pleased. Both groups also received newsletters presenting newly uploaded evidence or recent interviews that were sent out by the OE company to their subscribers at least once a week; these newsletters were not specific to the SIOR. See Table 1 for demographic information of the participants.
The researcher attended daily morning rounds to determine which ACE reports would be suitable for surgeons based on the cases presented. Cases that were more controversial were noted along with the reason for admission, diagnosis, and/or treatment plan. Relevant ACE reports specific to these cases were sent to surgeons receiving daily targeted mailers. For example, there was a great deal of discussion around surgical and non-surgical management of clavicle fractures. Thus, every couple of days, at least 1 clavicle fracture ACE report was included in the daily targeted mailers.
ACE reports were sent to surgeons receiving daily mailers via a newsletter. Each newsletter included 9 ACE reports with a minimum of 1 report for each sub-specialization department at the SIOR including pediatrics, trauma, knee, spine, hand and shoulder, hip, ortho-oncology, rheumatology, and general orthopaedics. ACE reports with low risk of bias scores (above 6/10 scores) based on OE's methodological quality assessment of recent articles (published within the last five years), high reported quality scores (above 17/20), author-verified ACE reports, high user scores (above 7/10), sample sizes with greater than 50 participants, randomized controlled trials and meta-analysis, were preferable to include in the daily ACE report newsletters sent to surgeons.
4 Outcome measurements
4.1 OrthoEvidence usage statistics
Individual participants’ usage data (open and click rates) were tracked through the OE database and MailChimp to determine whether receiving daily targeted evidence summaries led to more frequent use of the OE service compared to only receiving weekly general newsletters. The number of newsletters sent to each participant, the percentage of newsletters that were opened through the participant's email (open rates), the percentage of reports that were clicked within the newsletters for each participant (click rates), and the specific reports that were accessed were tracked. During the trial period, surgeons receiving daily targeted mailers received 21 daily mailers in addition to the 14 general mailers for a total of 35 mailers. Surgeons receiving general weekly mailers received 14 general mailers which were sent by OE.
The main analysis was descriptive in nature and the secondary analysis using Mann–Whitney U tests was exploratory. The data was not normally distributed. Thus, results were reported using median values and interquartile ranges. Mann–Whitney U tests were conducted to determine if there were differences in the open and click rates between the group receiving daily targeted mailers and the group receiving weekly general mailers. A threshold P value of ≤0.05 was used to determine the significance of the results.
4.2 Exit survey
All participants were surveyed about their experience with OE after a minimum of 3 weeks of having access to the product. OE use between the 2 groups was compared through self-reported data on the number of ACE reports reviewed, average time spent on OE, method of accessing OE, and ease of access to OE within the SIOR. Participants provided their perception of ACE report comprehensiveness, practicality, usefulness, efficiency, and applicability in practice on a 5-point Likert scale from strongly agree to strongly disagree. Surgeons also rated whether they felt better informed to make clinical decisions and whether they believed their patient care had improved as a result of knowledge gained from ACE reports. They were asked whether they took action to put any of the new knowledge they gained from ACE reports into practice by actively discussing reports with a colleague or changing patient care decisions. Descriptive analyses were used to report these results.
5.1 Baseline survey results
An initial baseline survey was used to obtain information about surgeons’ educational background, current use of medical literature, use of mobile technology, and the SIOR's Internet access.
Approximately 64% (n = 28) of survey respondents had access to computers connected to the Internet within the SIOR. Every surgeon that had access to these computers reported having difficulty browsing the Internet on them. All respondents used some form of mobile device (laptop, tablet, or cellular mobile phone). Furthermore, all respondents owned and used a smartphone within the SIOR. However, for 59% (n = 26) of survey respondents, connecting their smartphone to the Internet and browsing websites while in the SIOR was only sometimes or never possible depending on the location and connection. Connecting to mobile apps within the SIOR was also only sometimes or never possible depending on the location and connection for 61% (n = 27) of respondents. See Table 2 for a comparison of the 2 groups’ access to Internet.
Three quarters of respondents (75%; n = 33) said they accessed medical literature online while 73% (n = 32) also said they used textbooks. Only 39% (n = 17) said they read journals in paper form. Medical literature was read weekly by 52% (n = 23) of respondents while 66% (n = 29) read 1 to 3 scholarly journal articles in the week. Notably, 64% (n = 28) of individuals stated that their current method of keeping up to date with new research was inefficient. Interestingly, 3 quarters of the group (75%, n = 33) stated that their current method of staying up to date with research still improved their patient care. See Table 3 for a comparison of the 2 groups’ access to medical literature.
5.2 OE usage data results
Each individual's average open rate and click rate was captured for the 30-day study period. There was no difference between the 2 groups in terms of “open rates” (the percentage of newsletters that were opened through the participant's email) and “click rates” (the percentage of reports that were clicked within the newsletters for each participant). See Table 4 for comparison of open rates and click rates between groups.
Based on usage data over a 1-month time span, it was found that providing daily targeted evidence summaries to orthopaedic surgeons did not enable more frequent use of OE compared to providing weekly orthopaedic evidence summaries.
5.3 Exit survey results
An exit survey was used to compare the experiences and perceptions of surgeons receiving daily targeted ACE reports and those receiving general weekly reports. No difference was found between the 2 groups’ time spent reviewing literature in online journals, method, and ease of accessing OE. Most surgeons said they were reading 1 to 3 ACE reports weekly for 30 minutes or less and were accessing OE through their smartphones. However, the majority of respondents were only sometimes able to load the OE website without problems depending on their connection and/or location within the SIOR. See Table 5 for a comparison of the 2 groups’ usage of OE.
Overall, the survey found that OE ACE reports were perceived to be comprehensive, practical, useful, and applicable by most surgeons at the SIOR. Moreover, three-quarters of the group agreed that ACE reports had improved their efficiency and ability to keep up with new research. For more than half of the group, their confidence in medical decision-making and their perception of improved patient care also improved as a result of ACE reports. Surgeons were actively engaging with OE by reading and discussing ACE reports. See Table 6 for a comparison of the 2 groups’ perception of OE.
5.4 Semi-structured interview results
Nineteen semi-structured interviews were conducted with 8 surgical residents, 9 attending surgeons, and 2 visiting fellows. Thirteen of the interviewees were surgeons receiving daily targeted mailers while 6 were receiving weekly general mailers. Data were analyzed by the principal investigator using an edit organizing style in order to identify meaningful units and to organize these into themes.
The semi-structured interviews revealed some barriers to keeping up with evidence as well as challenges faced using OE. The problems mentioned in accessing relevant literature were a result of limited Internet connection, lack of time to access information, and minimal access to medical journals.
“The most important problem today is browsing. You don’t know what to look for and you end up wasting a lot of time reading unrelated things…because focused information is something which we lack in medicine…” (Attending Surgeon).
Limited incentives to keeping up with the literature were indicated to be due to limited decision-making powers for trainees for patient-related care, a textbook-based residency curriculum (instead of one that also incorporates findings from peer-reviewed primary and secondary literature), a lack of research methods’ knowledge, and limited context-specific research.
“Our exam patterns fit the textbooks. People read to pass their exams. If they start reading journals, they’d flunk their exams. The entire system needs to be turned around…” (Attending Surgeon).
Facilitators to using OE included the instruction and coaching provided by the researcher, as well as the overall convenience of the pre-appraised ACE report summary format, which was easily accessible through a mobile device. The majority of surgeons agreed that ACE reports were relevant because they provided support for surgeons’ decision-making. Some residents found OE useful when trying to understand certain surgical techniques and practices at the SIOR especially when they were in doubt. ACE reports also allowed the residents to better understand the surgery and its outcomes, leading to an increased knowledge base for future decision-making in the treatment of their patients.
Surgeons reported actively discussing new literature as it emerged and actively incorporating this evidence into their practice. For example, as a result of relevant ACE reports, the trauma team at the SIOR came together more frequently to discuss recent research emerging in their field from other groups. Subsequently, management of clavicle fractures shifted from nonoperative treatment to operative treatment. Surgeons also began prescribing combinations of Vitamin D, Vitamin K, and calcium to patients suffering from osteoporosis because an ACE report showed that calcium supplements would be better absorbed this way. Many attending surgeons explained that OE would be useful for doctors all over India who lack access to journal articles because receiving ACE reports through OE could help them stay up-to-date.
Given that there was no difference found between the 2 groups in terms of OE usage, it was concluded that sending daily targeted mailers did not enable more frequent usage of the OE service compared to providing weekly orthopaedic summaries over a 1-month time span. The study elicited barriers to the incorporation of EBM into surgical practice in India.
The barriers faced by surgeons in accessing medical literature (limited access to relevant literature and limited incentive to keep up with literature) are in line with studies from around the world.[6–8,11,12,15,22,23,29–36] The pre-appraised ACE report format facilitated surgeons’ use of OE and helped them keep up to date with literature. This finding is in line with research that demonstrates that increased convenience of access to reliable and applicable high quality research makes information-seeking more likely to occur and ultimately be successful.
Trauma surgeons noted that prior to having access to OE, the team was not actively discussing recent literature. But as a result of relevant ACE reports that everybody could access, the emergency department team was coming together to share their perspectives and discuss new research on a more consistent basis. Overall, OE helped the SIOR's surgeons to rethink, reassess, and redefine procedures related to fracture care in the trauma department.
The majority of surgeons agreed that OE ACE reports were comprehensive, practical, useful, and applicable for clinical practice. They also improved efficiency and ability to keep up with new research while helping to assist surgeons in making medical decisions. Surgeons described changing trauma practices as a result of information provided in OE ACE reports and active team discussions. These findings point toward the potential for knowledge dissemination portals that are easy to access, to help surgeons implement EBM in practice. The practice of EBM and the use of web-based EBM tools in India may be promoted by: improving Internet access, and integrating EBM into training programs and surgical culture.
6.1 Improving Internet access
Many of the barriers in accessing OE and generally keeping up with the literature stemmed from an overall inability to connect to the Internet at the SIOR. This means that the use of any online tool will likely be limited as a result of the inability to access the Internet within the hospital. If surgeons at the SIOR were having difficulty connecting to online tools, then many healthcare facilities in India with fewer resources will likely have more difficulty and thus, barriers to access. Investments to allow for reliable and fast Internet are needed for online EBM tools to be accessed throughout India.
6.2 Integration into residency curricula and surgical culture
The best way to embed evidence into good clinical practice in India is to change surgical culture in general, for example, by training attending surgeons to serve as appropriate EBM role models for students. It would be beneficial for the SIOR's residency curriculum to provide a research methods course at the beginning of their training to familiarize students with EBM concepts. EBM training should be led by attending surgeons who can focus on day-to-day case studies as examples.[39,40] Instilling habits early on through experiential and culturally supported ways of learning can ensure that EBM is practiced successfully by individuals and surgical communities alike.[8,12,39,41–43]
All interviewees agreed that OE should be added to a resident's toolbox as a reliable learning tool. Furthermore, a weekly reading program or journal club can be integrated into residency curriculum to promote the practice of EBM. The reading program could include the weekly dissemination of 1 high-impact article to surgeons, which would be followed by group discussions. Providing an opportunity for residents to lead these meetings can foster greater learning and collaboration. Incorporating an EBM competency section into academic exams may also promote physicians to read current literature.
6.3 Study limitations
This study focused on a high-tech, private hospital in India and thus as a single-center study, it limited the generalizability of the findings beyond the context in which this study was conducted. Government hospitals in India are even more overloaded with patients and have very limited resources; they are at greater risk of being out of date with new practices and procedures. The study's findings present the best-case scenario of the implementation, usability, and capability of an EBM knowledge dissemination tool such as OE because the SIOR possessed high-quality resources in comparison to other Indian hospitals.
The small sample size of 44 surgeons enrolled in the study reduced the statistical power of the study. The results trended in the anticipated direction with the median open rate and click rate being higher for the group of surgeons receiving daily targeted mailers compared to the group receiving weekly mailers (14.3% vs. 0% for open rates, and 4.8% vs. 0% for click rates, respectively). However, the small sample size left the study vulnerable to finding no difference in effect. A larger trial might have found significant differences.
Determination of where the participants’ EBM knowledge came from was also a confounding variable—a causal relationship between OE articles and changing clavicle fracture practices could not be made given that this could have occurred due to an accumulated effect. Given these limitations, it was a pragmatic study which added OE as a resource onto what surgical residents may otherwise have been exposed to.
A similar study may be conducted on a larger scale across one Indian state as a cluster randomized trial to avoid contamination, using hospitals as the unit of randomization. Some hospitals would get weekly, general mailers while other hospitals would get daily, targeted mailers.
Although surgeons at the SIOR expressed positive attitudes toward OE as a knowledge-dissemination portal, it was not used to its full potential given numerous barriers. It is likely that other online EBM resources will face similar accessibility challenges. The capacity to use mobile applications such as OE within high-tech centers like the SIOR is available; however, without adequate investments in high-speed Internet, the full potential of such programs cannot be attained.
While providing the appropriate evidence-based resources through adequate Internet access is necessary, a paradigm shift in the overall methods of training future surgeons is crucial, especially with regard to the integration of EBM. Suitable training for senior role models and residents is needed for them to serve as key opinion leaders to ensure the sustainable integration of EBM practices into Indian surgical culture. Future studies should explore whether providing OE directly improves concordance between evidence and practice and whether it leads to better outcomes for patients over a longer period of time.
1. Montori V, Guyatt G. Progress in evidence-based medicine
. JAMA. 2008;300:1814–1816.
2. Sackett D, Rosenberg W, Gray J, et al. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312:71–72.
3. Smith R, Rennie D. Evidence based medicine—an oral history. JAMA. 2014;311:365–367.
4. Agarwal R, Kalita J, Misra U. Barriers to evidence-based medicine
practice in South Asia and possible solutions. Neurol Asia. 2008;13:87–94.
5. Hurwitz S, Tornetta P, Wright J. An AOA critical issue how to read the literature to change your practice: an evidence-based medicine
approach. J Bone Joint Surg. 2006;88:1873–1879.
6. Prasad K. Evidence-based medicine
in India. J Clin Epi. 2013;66:6–9.
7. Zaidi Z, Hashim J, Iqbal M, et al. Paving the way for evidence-based medicine
in Pakistan. JPMA. 2007;57:556–560.
8. Wilkinson E, Bosanquet A, Salisbury C, et al. Original Paper: Barriers and facilitators to the implementation of evidence-based medicine
in general practice: a qualitative study
. Eur J Gen Pract. 1999;5:66–70.
9. Grimshaw J, Eccles M, Lavis J, et al. Knowledge translation
of research findings. Implementation Sci. 2012;7:50.
10. Straus S, Tetroe J, Graham I. Defining knowledge translation
. CMAJ. 2009;181:165–168.
11. Al Omari M, Khader Y, Jadallah K, et al. Evidence-based medicine
among hospital doctors in Jordan: awareness, attitude and practice. J Eval Clin Pract. 2009;15:1137–1141.
12. Al-Almaie S, Al-Baghli N. Barriers facing physicians practicing evidence-based medicine
in Saudi Arabia. J Continuing Educ Health Professions. 2004;24:163–170.
13. Al-Ansary L, Khoja T. The place of evidence-based medicine
among primary health care physicians in Riyadh region, Saudi Arabia. Fam Pract. 2002;19:537–542.
14. Booth A. Evidence-based practice and the developing world. J Health Inform Libraries. 2008;25:74–77.
15. Dans A, Dans L. The need and means for evidence-based medicine
in developing countries. Evid-Based Med. 2000;5:100–101.
16. Sabri A, Qayyum M. The problem of evidence-based medicine
in developing countries. CMAJ. 2006;175:62–162.
17. Siddiqi K, Newell J. Putting evidence into practice in low-resource settings. Bull World Health Organ. 2005;83:882.
18. Siddiqi K, Newell J, Robinson M. Getting evidence into practice: what works in developing countries? Int J Qual Health Care. 2005;17:447–454.
19. Adily A, Ward J. Enhancing evidence-based practice in population health: staff views, barriers and strategies for change. Aust Health Rev. 2005;29:469–477.
20. Dawes M, Summerskill W, Glasziou P, et al. Sicily statement on evidence-based practice. BMC Med Educ. 2005;5:1.
21. Wolfe A. Institute of medicine report: crossing the quality chasm: a new health care system for the 21st century. Policy Pol Nurs Pract. 2001;2:233–235.
22. Sadeghi-Bazargani H, Tabrizi J, Azami-Aghdash S. Barriers to evidence-based medicine
: a systematic review. J Eval Clin Pract. 2014;20:793–802.
23. Mittal R, Perakath B. Evidence-based surgery: knowledge, attitudes, and perceived barriers among surgical trainees. J Surg Educ. 2010;67:278–282.
24. Maier R. What the surgeon of tomorrow needs to know about evidence-based surgery. Arch Surg. 2006;141:317–323.
25. Our Process. Myorthoevidence.com. Available at: https://myorthoevidence.com/About/Process
. Accessed July 15, 2018.
26. Creswell J, Plano Clark V. Creswell J, Plano Clark V. Designing and Conducting Mixed Methods
Research. 2nd edThousand Oaks, CA: Sage Publications; 2011;82–85.
27. Creswell J, Plano Clark V, Gutmann M. Tashakkori A, Teddlie C, et al. Advanced mixed methods
research designs. Handbook of Mixed Methods
in Social and Behavioral Research. Thousand Oaks, CA: Sage Publications; 2003;209–240.
28. Crabtree B, Miller W. Crabtree B, Miller W. The dance of interpretation. Doing Qualitative Research. 2nd edThousand Oaks, CA: Sage Publications; 1999;127–143.
29. Ahmadi-Abhari S, Soltani A, Hosseinpanah F. Knowledge and attitudes of trainee physicians regarding evidence-based medicine
: a questionnaire survey in Tehran, Iran. J Eval Clin Pract. 2008;14:775–779.
30. Bhandari M, Montori V, Devereaux P, et al. Challenges to the practice of evidence-based medicine
during residents’ surgical training: a qualitative study
using grounded theory. Acad Med. 2003;78:1183–1190.
31. Green M, Ruff T. Why do residents fail to answer their clinical questions? A qualitative study
of barriers to practicing evidence-based medicine
. Acad Med. 2005;80:176–182.
32. Karthikeyan G, Pais P. Clinical judgement & evidence-based medicine
: time for reconciliation. Indian J Med Res. 2010;132:623.
33. Mozafarpour S, Sadeghizadeh A, Kabiri P, et al. Evidence-based medical practice in developing countries: the case study of Iran. J Eval Clin Pract. 2011;17:651–656.
34. O’Donnell C. Attitudes and knowledge of primary care professionals towards evidence-based practice: a postal survey. J Eval Clin Pract. 2004;10:197–205.
35. Tilburt J, Goold S, Siddiqui N, et al. How do doctors use information in real-time? A qualitative study
of internal medicine resident precepting. J Eval Clin Pract. 2007;13:772–780.
36. van Dijk N, Hooft L, Wieringa-de Waard M. What are the barriers to residentsʼ practicing evidence-based medicine
? A systematic review. Acad Med. 2010;85:1163–1170.
37. Dawes M, Sampson U. Knowledge management in clinical practice: a systematic review of information seeking behavior in physicians. Int J Med Inform. 2003;71:9–15.
38. Salgotra KR. Evidence based orthopaedics – Western evidence in Indian context. Med J Armed Forces India. 2012;68:314–315.
39. Kitto S, Petrovic A, Gruen R, et al. Evidence-based medicine
training and implementation in surgery: the role of surgical cultures. J Eval Clin Pract. 2010;17:819–826.
40. Kwaan M, Melton G. Evidence-based medicine
in surgical education. Clin Colon Rectal Surg. 2012;25:151–155.
41. Gorgon E, Barrozo H, Mariano L, et al. Research evidence uptake in a developing country: a survey of attitudes, education and self-efficacy, engagement, and barriers among physical therapists in the Philippines. J Eval Clin Pract. 2013;19:782–790.
42. Scott I, Heyworth R, Fairweather P. The use of evidence-based medicine
in the practice of consultant physicians. Results of a questionnaire survey. Aust N Z J Med. 2000;30:319–326.
43. Srinivasan M, Weiner M, Breitfeld P, et al. Early introduction of an evidence-based medicine
course to preclinical medical students. J Gen Intern Med. 2002;17:58–65.
44. Weglein D, Gugala Z, Simpson S, et al. Impact of a weekly reading program on orthopedic surgery residents’ in-training examination. Orthopedics. 2015;38:e387–e393.