- Question: What is the long-term impact of the Safer Anaesthesia From Education Obstetric Anaesthesia (SAFE-OB) course in the Republic of Congo and in Madagascar?
- Findings: Qualitative evidence of personal and organizational change was found at 12–18 months corroborated by quantitative evidence of knowledge and skill retention.
- Meaning: Because anesthesia is a contributor to maternal mortality in low- and middle-income countries (LMICs; 13.8% of deaths after cesarean delivery are directly attributable to anesthesia), the findings suggest the SAFE-OB course may have a role to play in decreasing maternal mortality.
Maternal mortality is unacceptably high in low- and middle-income countries (LMICs). Globally, 99% of all maternal deaths occur in LMICs.1 In 2015, the maternal mortality ratio in high-income countries (HICs) was 12 per 100,000 live births, compared with 239 per 100,000 live births in LMICs.2 In Madagascar and in the Republic of Congo, the maternal mortality ratio is 3533 and 4424 per 100,000 live births, respectively, almost twice the average reported for LMICs.
The cause of high maternal mortality worldwide is multifactorial,5 but poor-quality anesthesia care is a contributor. It is known that 2.8% of all maternal deaths, 3.5% of direct maternal deaths (those resulting from obstetric complications), and 13.8% of deaths after cesarean delivery are directly attributable to anesthesia.6 For women undergoing obstetric procedures in LMICs, the risk of death from anesthesia is 12 per 1000 women.6 Anesthesia-attributed deaths (eg, complications of airway management and spinal anesthesia) are considered preventable with appropriate training and resources.6,7 Therefore, quality improvement efforts targeted to optimize obstetric anesthesia care worldwide could help improve maternal mortality.
The Safer Anaesthesia From Education Obstetric Anaesthesia (SAFE-OB) course is an education course designed for use in LMICs by the Association of Anaesthetists of Great Britain and Ireland (AAGBI) and administered by the World Federation of Societies of Anaesthesiologists (WFSA).8 SAFE-OB aims to improve the quality of obstetric anesthesia care through a 3-day refresher course for trained anesthesia providers (physician and nonphysician). In most LMICs, there is no ongoing continuing medical education and no requirement from licensing authorities (where they exist) to participate in educational activities. The SAFE-OB course is highly interactive using various teaching modalities such as scenario teaching, skills sessions, and small-group discussion. Lectures are kept to a minimum. The course is designed for 32 participants and 6–8 faculty instructors to keep the faculty/participant ratio low to optimize learning through interaction. Short-term course outcomes have been published,9,10 but longer-term outcomes (12–24 months) are not reported. Generally, published data on the impact of educational courses in LMICs are sparse. A recent review of the topic admitted that, out of necessity, much of the article was based on unpublished data from the WFSA, personal communication with course organizers and participants, and the author’s own personal experience.8
We aimed to use a concurrent embedded mixed-methods design based on the Kirkpatrick11 model (Table 1) for evaluating educational interventions to assess the short- and long-term impact of SAFE-OB after 5 courses in 2 African countries: the Republic of Congo and Madagascar, between 2014 and 2016.
Mercy Ships Institutional Review Board (MS-2016-004) and the Republic of Congo and Madagascar Ministries of Health gave permission for the study. The requirement for written informed consent was waived, but interviews required verbal consent.
The Republic of Congo and Madagascar are both low-income countries and are ranked 137 and 197, respectively, of 197 countries on the United Nations Health Development Index.12 The population sizes are 4.6 and 24.3 million, respectively, with a physician anesthesia density of 0.19 and 0.23 per 100,000 population.13 In the Republic of Congo, only nonphysician anesthesia provider training programs are available (physicians travel abroad for anesthesia training), whereas in Madagascar, both physician and nonphysician anesthesia training programs are provided.
A team from Mercy Ships taught 5 SAFE-OB courses between April 2014 and April 2016: 1 in the Republic of Congo and 4 in Madagascar. The SAFE-OB course outline is shown in Supplemental Digital Content 1, Table, http://links.lww.com/AA/C877. Although SAFE-OB is primarily aimed at anesthesia providers (physician and nonphysician), up to 8 obstetricians/midwives per course were also invited to attend days 2 and 3 of the course. We made this adaptation at local request because multidisciplinary training is known to increase the effectiveness of quality improvement interventions and the ability to apply learning to a clinical setting.14,15 No travel expenses or per diems were given to participants. French is the national language of both the Republic of Congo and Madagascar. All course materials were translated into French but, because the majority of course instructors did not speak French, sequential translation was used for all sessions.
Study Design, Sample Size, and Selection
The study was based on the Kirkpatrick11 model for evaluating training programs (Table 1) and a mixed-methods analysis design as recommended for patient safety research.16 Qualitative and quantitative data were integrated using a concurrent embedded design.17 The priority in data collection was a qualitative approach. Secondary data collection utilized quantitative knowledge and skills assessments to triangulate the qualitative self-reported responses.
The sample size and sampling techniques were based on the primary qualitative study outcome. Therefore, for impact evaluation at 4 and 12–18 months, participants were chosen using purposive sampling. Purposive sampling is widely used in qualitative research to identify and select information-rich cases to maximize effective use of limited resources.18 This often involves identifying and selecting knowledgeable and experienced individuals17 who are available and willing to communicate experiences and opinions in an articulate and reflective manner.19 Therefore, participants who engaged in open and frank discussions with the faculty were respected by their peers and often elected as “spokesperson” during the course, and were chosen based on being designated opinion leaders.20 Sampling was also pragmatically based on geographical location (those living near the evaluation location) because involvement in the study was voluntary and no per diems or transport costs were paid.
Initial Assessment and Selection of Participants for Each SAFE-OB Course
In each country, we conducted anesthesia educational needs assessment consisting of hospital surveys and interviews with the Ministry of Health, Dean of the Medical School, and the local anesthesia societies. The hospital surveys used a tool adapted from the World Health Organization (WHO) Situational Analysis tool21 and the 2010 WFSA International Standards for Safe Practice of Anaesthesia.22 As a result, minor course adaptations were agreed concerning local medication, equipment availability, and timetabling (eg, the use of magnesium chloride instead of magnesium sulfate, the complete lack of bougies, and the need to start the course at the time the local hospitals usually start work). Due to the relative sizes of the 2 countries and Mercy Ships operational capacity, 1 course was planned for the Republic of Congo and 4 courses for Madagascar. All participants were selected by the country’s chief of anesthesia in consultation with local hospital directors because local engagement with hospital management is known to be important in implementation of learning.14,15 The Republic of Congo participants were nonphysician anesthesia providers and the only 2 physician anesthesia providers working in the Pointe-Noire and Dolisie areas, which are the 2 largest cities outside the capital city. Participants from Madagascar were a mix of physician and nonphysician anesthesia providers selected from almost all the major regions of the country.
Initial Impact Assessment (Kirkpatrick Levels 1 and 2)
Before each course, all anesthesia participants underwent baseline knowledge and skill assessment using the standard tools provided as part of the SAFE-OB course. Midwives and obstetricians did not complete these assessments because the test contained some specific anesthesia-related questions. Knowledge assessment consisted of 50 questions, each with a true or false response and the result given as a percentage of correct responses. Skill assessment used low-technology simulation. Participants were allocated (using block randomization) to demonstrate 1 of 4 skills: neonatal resuscitation, maternal resuscitation, intubation, or initial assessment of a sick pregnant mother. The participant was required to complete 10 predetermined steps, and the number of correct steps was given as a percentage. Immediately after the course, participants repeated the same knowledge and skill assessments (Kirkpatrick level 2). In addition, at the end of the course, participants completed a self-reported feedback questionnaire in French on their enjoyment of the course and the relevance to their practice (Kirkpatrick level 1). An English translation of the questionnaire is provided in Supplemental Digital Content 2, Appendix 1, http://links.lww.com/AA/C878.
Long-term Impact Assessment at 4 and 12–18 Months (Kirkpatrick Levels 1–4)
Evaluation of Self-Reported Behavior Change and Identification of Catalysts and Barriers to Change (Kirkpatrick Levels 3 and 4).
Semistructured interviews were conducted with 1 cohort of participants at 4 months and a second cohort at 12–18 months. To lessen the chance of interinvestigator bias, an interview guide was used (Supplemental Digital Content 3, Appendix 2, http://links.lww.com/AA/C879).
All courses were taught in French. All interviews were conducted in English, French, or Malagasy depending on the participant’s preference. T.R. conducted interviews in all 3 languages without the need for a translator; M.C.W. and K.L.C. in English or French without a translator, but one was present in case clarification of responses was needed; and N.H.C. conducted interviews in English translated to French using a Mercy Ships translator. Interviews took place either face-to-face or by telephone at a prearranged time. Interviews lasted 15–20 minutes and occurred remote from the hospital (hotel or community center). Interviews were recorded by a primary scribe (not the interviewer or translator) with pen and paper. M.C.W. and N.H.C. are physician anesthetists, and K.L.C. and T.R. organize medical courses in LMICs. All interviewers were trained, familiar with the SAFE-OB course content and the working conditions of the participants. All participants were over 18 years of age and gave voluntary verbal consent to participate in the interview.
The primary outcome (qualitative determination of personal and organizational change at 4 and 12–18 months) was determined from interview responses. Interview responses were anonymized and labeled numerically with the prefix “C” or “M” to indicate the Republic of Congo or Madagascar, respectively. Interview responses were analyzed by A.W. and M.C.W. using the standard 6 phases of inductive thematic analysis: (1) responses were read and reread to become familiar with the data and pay attention to initially emerging patterns and ideas; (2) generation of initial categories (codes) by collapsing the data into labels and creating the categories for further analysis; (3) collation of categories into defined themes that accurately depict the data; (4) review of themes and entire data set to check for completeness; (5) define and name each theme; and (6) decide which themes make meaningful contributions to understanding what is going on within the data. Phases 1–3 were conducted independently, and phases 4–6 (reviewing, defining, and interpreting) were done collaboratively. Disagreements were referred to T.R.
Secondary outcome data (quantitative evaluations of knowledge and skill retention over time) were analyzed visually using a frequency histogram to determine distribution, and paired participant data were analyzed using a 2-tailed paired t test. A Bonferroni correction was applied for multiple testing (2 hypotheses); therefore, the critical P level for significance was set at .025, so individual tests with P < .025 were considered significant.
From 2014 to 2016, 213 participants (166 anesthesia providers and 47 obstetricians and midwives) were trained in 5 SAFE-OB courses in the Republic of Congo and Madagascar. Only anesthesia providers were included in the analysis.
Kirkpatrick Levels 1 and 2: Reaction and Learning
All participants rated the course as enjoyable, relevant, and said they would recommend it to others.
One hundred fifty-four of 166 anesthesia providers completed knowledge and skill tests pre- and postcourse. Failure to arrive on time or leaving early accounted for the noncompletions. Knowledge and skill test results improved significantly from precourse to postcourse and showed sustained improvement at 4 and 12–18 months after. Details are given in Table 2.
Kirkpatrick Level 3: Change in Personal Practice
Semistructured interviews were conducted at 4 months with 46 participants, and at 12–18 months with 28 different participants of whom 19 of 46 and 16 of 28 participants, respectively, also completed repeat knowledge and skills tests. Those who underwent telephone interview did not complete repeat knowledge and skills tests.
Responses among participants at 4 months were almost identical to the responses at 1 year with 2 notable exceptions: neonatal resuscitation and airway management. Neonatal resuscitation was reported by 50% of interviewees at 4 months and airway management by 25% as seen in the following comments: “Before I was nervous about dealing with newborns, now I know just what to do, have confidence and certainty in my decisions” [M7]. “Before I wasn’t confident in caring for urgent cases of pregnant women, now I am sure of intubation techniques, lateral tilt, and protecting the airway” [M20]. However no interviewee at 12–18 months mentioned neonatal resuscitation or airway management when discussing the most important things they had learned and what changes they had made in their personal practice.
On thematic analysis of responses at 12–18 months, 4 areas of practice change were identified: management of obstetric hemorrhage, use of the “airway, breathing, circulation (ABC)” structured approach, management of eclampsia, and the use of spinal anesthesia. Responses were similar for participants from the Republic of Congo and Madagascar. Further details are reported in Table 3.
Kirkpatrick Level 4: Organizational Culture Change
Responses to questioning about areas of organizational culture change were similar between the Republic of Congo and Madagascar. From thematic analysis, 3 themes emerged that were universally present at both 4 and 12–18 months: improved teamwork, communication, and preparation, as shown in Table 4. In addition, many Malagasy participants explained that new protocols had been developed for the management of spinal anesthesia, eclampsia, commonly used drugs and doses, and always intubating with a pillow under the woman’s head. The chief obstetric anesthetist in Madagascar, assisted by another professor, led these changes at the university hospital and publicized them through the Malagasy Society of Anaesthetists. By contrast in the Republic of Congo, the only new protocols were related to the dose of magnesium and volume of fluid used in eclampsia management.
No negative consequences were reported as a result of the changes, although a teaching professor in Madagascar said “Teamwork has been improved but now when there is no pulse oximeter my students do not want to work because they are used to it” [M37]. This was because, as a result of the SAFE-OB course, the need for pulse oximetry as an essential component of safe anesthesia was better understood, and was being more frequently used than before the course.
Perceived Catalysts and Barriers to Change
Four themes (leader support, self-motivation, peer support, and credibility) were identified as perceived catalysts for change and 3 themes (resistance from peers, lack of senior support, and lack of resources) as barriers. At 4 months, the overwhelming perceived barrier to change reported by almost all participants was lack of resources (equipment and medication), closely followed by resistance from peers, and lack of senior support. However, at 12–18 months, the biggest perceived barrier was resistance from peers and then lack of senior support. Few reported lack of resources. A summary of catalysts and barriers is given in Table 5.
Our mixed-methods evaluation of the SAFE-OB course in the Republic of Congo and Madagascar shows evidence of self-reported personal and organizational practice change sustained at 1 year and corroborated by evidence of knowledge and skill retention. Perceived catalysts and barriers to change are identified with some changing over time. Lack of resources becomes less important, and self-motivation becomes more important for sustained behavior change over time.
Effective education programs need to be enjoyable and relevant. Our results indicate the course was both of these. The most common overall causes of maternal mortality in LMICs are hemorrhage and eclampsia,2 and these were reported as 2 of the 4 most important areas of change in personal practice. The most common causes of anesthesia-attributed maternal mortality globally are complications of airway management and spinal anesthesia,6 both of which were reported as areas of change in practice. However, airway management was only reported at 4 and not 12–18 months. This may be because most obstetric anesthesia in the Republic of Congo and in Madagascar is performed under spinal anesthesia. Most anesthesia providers are trained to use only spinal anesthesia for obstetric cases due to lack of supplies for intubation and general anesthesia. In Madagascar, laryngoscopes are only available in 67% of hospitals and functioning vaporizers in 43%.23 Perhaps airway management was interesting initially but then faded due to irrelevance/lack of opportunity to practice the technique. Similarly neonatal resuscitation was also identified as an important area of change at 4 months but not at 12–18 months. This might also be due to lack of opportunity because most neonatal resuscitation is performed by midwives, with anesthetists rarely providing assistance. This supports the idea of including midwives on the SAFE-OB course as the course content highlights areas of practice specifically relevant to them, such as neonatal resuscitation.
Our findings of knowledge and skill retention at 12–18 months are similar to other educational initiatives in LMIC studies.10,24,25 In particular, evaluation of the SAFE-OB course in Rwanda by Livingston et al10 showed practice changes similar to our study: systematic management of emergencies, improved teamwork, communication, and preparation; similar catalysts for change: support from colleagues and higher hospital authorities; and similar obstacles to change: lack of resources, resistance from colleagues, and lack of engagement from hospital administrators.
The literature on knowledge and skill retention after training courses varies. Some studies show that decay happens as early as 4 months and others not until 6–12 months.26,27 Many studies come from high-income settings where health care professionals are subjected to a huge array of mandatory training from fire safety to cardiopulmonary resuscitation, yet in reality many never use these skills routinely. Therefore, it is not surprising that knowledge and skills decay rapidly. However, if participants have regular opportunity to use the knowledge and skills learned, then evidence shows improved retention.28,29 In our study, in contrast to HICs, all participants were regularly dealing with obstetric emergencies such as eclampsia and hemorrhage on a weekly basis. Therefore, it was easy to put new knowledge into practice. The differences in outcomes between high- and low-income settings highlight the need for reinforcement of any new knowledge and skills through regular practice. Another explanation for the differences may be due to the rarity of educational opportunities in LMICs. It is possible that the 1 course attended might be easier to remember and put into practice without the “noise” of lots of other courses and information, but our study did not evaluate this hypothesis.
Lack of resources and hierarchy (or lack of senior support) are commonly identified barriers to quality improvement initiatives in LMICs.30–33 The novel finding in this study, which requires further hypothesis testing, is that lack of resources is seen as less of a perceived barrier over time. This could be because initially it is easy to cite lack of resources as a reason why change is difficult, rather than struggle through the effort required to implement change. This theory is backed up by those participants who managed to make changes because they identified self-motivation, hard work, and determination as important catalysts for change. In addition, studies from HICs with all the necessary resources still find change difficult to initiate and sustain even when well evidenced and mandated.32,34 These and other studies suggest that attitudes and beliefs (culture) may be equally if not more important than physical resources.30,32,34,35 The SAFE-OB style of teaching allows time for discussion, which overcomes other known barriers to change such as skepticism and misunderstanding30–33,36 and allows sharing of experiences to ensure contextual relevance.
This study has a number of limitations. The main limitation is that because there was no concurrent control group we cannot attribute causality of changes in personal and organizational practice to the SAFE-OB course. We used self-reported responses to assess personal and organizational change. The relationship between self-reported change and actual behavior change was not measured, but self-reported change may overestimate actual change. Self-reporting is also subject to bias—either overly positive responses to please the interviewers or overly negative responses to obtain more training. However, understanding how and why a behavior change necessitates participants self-reporting their experiences and perceptions.
Our technique of pragmatic purposive sampling based on geographical location and using “opinion leaders” who were willing to engage in open and frank discussion with the faculty may have led to bias reporting and limits the generalizability of our findings. However, qualitative interviewing to elucidate catalysts and barriers to change relies on respondents’ willingness to share experiences and perceptions in an articulate, expressive, and reflective manner.17–19 This is in contrast to quantitative probability-based research which uses random sampling to ensure generalizability, minimize bias and control for the influence of confounders.
Catalysts and barriers to personal practice and organizational change were analyzed together for all participants. Therefore, we are able to comment if differences exist between participant cadres or between the types of change (individual and system).
Interviews were not recorded and transcribed and thus subject to recorder bias, but this was minimized by using a second person as a scribe who understood the primary language used and was not reliant on the translation and also provided verification on the translation. The primary study methodology was qualitative, not quantitative; therefore, no sample size calculation was done to estimate the power to detect differences in knowledge and skill tests. The construct, validity, and reliability of the SAFE-OB knowledge and skills tests are unknown, and using a single skill is intuitively a narrow test of obstetric anesthetic competence. Different cohorts were evaluated at 4 and 12–18 months to minimize participants’ absence from their workplaces.
Despite these limitations, our study has a number of strengths. The study adds to the paucity of evidence on long-term impact of anesthesia education courses in LMICs. The study provides evidence of sustainable personal and organizational change at 12–18 months and novel insights into perceived barriers and catalysts for change.
In conclusion, this study demonstrates that for participants from the Republic of Congo and Madagascar, attending a SAFE-OB course is associated with improved knowledge and skills; and self-reported changes in personal practice and organizational culture sustained over time.
AVAILABILITY OF DATA AND MATERIALS
Because of the perceived sensitive nature of the data collected for this study, requests to access the data set from qualified researchers trained in human subject confidentiality protocols may be sent to the corresponding author and are subject to approval by Mercy Ships Institutional Review Board and the Ministry of Health in the Republic of Congo and Madagascar.
Name: Michelle C. White, MBChB.
Contribution: This author helped conceive the original idea; design the study; acquire the data; interpret and analyze the data; review the literature; write the first draft of the manuscript; and approve the final manuscript.
Name: Tsiferana Rakotoarisoa.
Contribution: This author helped acquire the data, critically review the manuscript, and approve the final manuscript.
Name: Nicola H. Cox, MBBS.
Contribution: This author helped acquire the data and critically appraise and approve the final manuscript.
Name: Kristin L. Close, MPH.
Contribution: This author helped acquire the data and critically appraise and approve the final manuscript.
Name: Joan Kotze.
Contribution: This author helped acquire the data and critically appraise and approve the final manuscript.
Name: Abigail Watrous, PhD.
Contribution: This author helped design the study; interpret and analyze the data; and critically appraise and approve the final manuscript.
This manuscript was handled by: Angela Enright, MB, FRCPC.
1. Alkema L, Chou D, Hogan D, et al. Global, regional, and national levels and trends in maternal mortality between 1990 and 2015, with scenario-based projections to 2030: a systematic analysis by the UN Maternal Mortality Estimation Inter-Agency Group. Lancet (London, England). 2016:387;462–474.
2. World Health Organization. Maternal health and mortality. Available at: https://www.who.int/gho/maternal_health/countries/en/#M
. Accessed December 13, 2018.
3. World Health Organization. Maternal Mortality Ratio, Madagascar, 2015. Available at: https://www.who.int/gho/maternal_health/countries/mdg.pdf?ua=1
. Accessed December 13, 2018.
4. World Health Organization. Maternal Mortality Ratio, Congo-Brazzaville, 2015. Available at: http://www.who.int/gho/maternal_health/countries/cog.pdf?ua=1
. Accessed December 13, 2018.
5. Walker IA, Wilson IH. Anaesthesia in developing countries–a risk for patients. Lancet. 2008;371:968–969.
6. Sobhy S, Zamora J, Dharmarajah K, et al. Anaesthesia-related maternal mortality in low-income and middle-income countries: a systematic review and meta-analysis. Lancet Glob Health. 2016;4:e320–e327.
7. Dalina AM, Inbasegaran K. Anaesthetic related maternal deaths in Malaysia–a review. Med J Malaysia. 1996;51:52–63.
8. Evans FM, Duarte JC, Haylock Loor C, Morriss W. Are short subspecialty courses the educational answer? Anesth Analg. 2018;126:1305–1311.
9. Enright A, Grady K, Evans F. A new approach to teaching obstetric anaesthesia in low-resource areas. J Obstet Gynaecol Can. 2015;37:880–884.
10. Livingston P, Evans F, Nsereko E, et al. Safer obstetric anesthesia through education and mentorship: a model for knowledge translation in Rwanda. Can J Anaesth. 2014;61:1028–1039.
12. United Nations Human Development Index, United Nations, 2018. Available at: http://hdr.undp.org/en/2018-update
. Accessed December 13, 2018.
13. Kempthorne P, Morriss WW, Mellin-Olsen J, Gore-Booth J. The WFSA global anesthesia workforce survey. Anesth Analg. 2017;125:981–990.
14. White MC, Millimouno FS. Observational study of three different methods of implementing the WHO surgical safety checklist in Guinea Global. Anesth Perioper Med. 2016:2;158–161.
15. White M, Close K. Lasting impact: insights from a surgical mission-based mentoring training programme in the Republic of Congo. BMJ Glob Health. 2016;1:e000102.
16. Brown C, Hofer T, Johal A, et al. An epistemology of patient safety research: a framework for study design and interpretation. Part 4. One size does not fit all. Qual Saf Health Care. 2008;17:178–181.
17. Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. 2011.Los Angeles: SAGE Publications
18. Patton MQ. Qualitative Research and Evaluation Methods. 2002.3rd ed. Thousand Oaks, CA: Sage Publications
19. Bernard HR. Research Methods in Anthropology: Qualitative and Quantitative Approaches. 2002.3rd ed. Walnut Creek, CA: Alta Mira Press;
20. Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.
21. World Health Organization. WHO integrated management for essential and emergency surgical care (IMEESC) toolkit.
22. Merry AF, Cooper JB, Soyannwo O, Wilson IH, Eichhorn JH. International standards for a safe practice of anesthesia 2010. Can J Anesth. 2010;57:1027–1034.
23. Baxter LS, Ravelojaona VA, Rakotoarison HN, et al. An observational assessment of anesthesia capacity in Madagascar as a prerequisite to the development of a national surgical plan. Anesth Analg. 2017;124:2001–2007.
24. Osei-Ampofo M, Tafoya MJ, Tafoya CA, Oteng RA, Ali H, Becker TK. Skill and knowledge retention after training in cardiopulmonary ultrasound in Ghana: an impact assessment of bedside ultrasound training in a resource-limited setting. Emerg Med J. 2018;35:704–707.
25. Tuyisenge L, Kyamanya P, Van Steirteghem S, Becker M, English M, Lissauer T. Knowledge and skills retention following emergency triage, assessment and treatment plus admission course for final year medical students in Rwanda: a longitudinal cohort study. Arch Dis Child. 2014;99:993–997.
26. Charalampopoulos D, Karlis G, Barouxis D, et al. Theoretical knowledge and skill retention 4 months after a European paediatric life support course. Eur J Emerg Med. 2016;23:56–60.
27. Yang CW, Yen ZS, McGowan JE, et al. A systematic review of retention of adult advanced life support knowledge and skills in healthcare providers. Resuscitation. 2012;83:1055–1060.
28. Fischer H, Strunk G, Neuhold S, et al. The effectiveness of ERC advanced life support (ALS) provider courses for the retention of ALS knowledge. Resuscitation. 2012;83:227–231.
29. Selden NR, Anderson VC, McCartney S, Origitano TC, Burchiel KJ, Barbaro NM. Society of neurological surgeons boot camp courses: knowledge retention and relevance of hands-on learning after 6 months of postgraduate year 1 training. J Neurosurg. 2013;119:796–802.
30. Aveling EL, McCulloch P, Dixon-Woods M. A qualitative study comparing experiences of the surgical safety checklist in hospitals in high-income and low-income countries. BMJ Open. 2013;3:e003039.
31. Conley DM, Singer SJ, Edmondson L, Berry WR, Gawande AA. Effective surgical safety checklist implementation. J Am Coll Surg. 2011;212:873–879.
32. Russ SJ, Sevdalis N, Moorthy K. A qualitative evaluation of the barriers and facilitators toward implementation of the WHO surgical safety checklist across hospitals in England: lessons from the “Surgical Checklist Implementation Project.” Ann Surg. 2015;261:81–91.
33. Gagliardi AR, Straus SE, Shojania KG, Urbach DR. Multiple interacting factors influence adherence, and outcomes associated with surgical safety checklists: a qualitative study. PLoS One. 2014;9:e108585.
34. Russ S, Rout S, Caris J, et al. Measuring variation in use of the WHO surgical safety checklist in the operating room: a multicenter prospective cross-sectional study. J Am Coll Surg. 2015:220;1–11.e14.
35. Aveling EL, Kayonga Y, Nega A, Dixon-Woods M. Why is patient safety so hard in low-income countries? A qualitative study of healthcare workers’ views in two African hospitals. Global Health. 2015;11:6.
36. Bergs J, Lambrechts F, Simons P, et al. Barriers and facilitators related to the implementation of surgical safety checklists: a systematic review of the qualitative evidence. BMJ Qual Saf. 2015;24:776–786.