KEY POINTS
Question: What is the extent of compliance with the World Health Organization Surgical Safety Checklist and the association of this compliance with perioperative surgical outcomes in Uganda’s referral hospitals?Finding: Compliance was low and varied across the different hospitals and there was no association between compliance and perioperative surgical outcomes.Meaning: While the compliance with the World Health Organization Surgical Safety Checklist was unacceptably low, the findings did not show any significant association between compliance and perioperative surgical outcomes. Hospital-associated adverse events involving patients has been estimated at 10%, many of which are preventable.1 2
The World Health Organization (WHO) drew lessons from aviation experience3 4 5 , 6
In the United Kingdom, hospital audit data showed high compliance rates, but direct observation suggested that actual performance may be suboptimal and therefore it remains unclear whether the benefits obtained were through actual completion of a checklist or from an increase in overall awareness of patient safety issues.7 8
Rationale
The purpose of the study was as follows:
To determine compliance with the WHO SSC
To determine the relationship between compliance with the WHO SSC and perioperative surgical outcomes in Uganda’s referral hospitals.
These data will be used as a reference for future research, monitoring, and evaluation of the WHO SSC, as well as for quality improvement projects in Uganda and Africa at large.
METHODS
Study Area and Setting
A multicentre prospective cohort study was conducted in 5 of Uganda’s referral hospitals over a 4-month period (April 2016 to July 2016).
The hospitals were chosen purposively based on geographic location and the presence of an anesthesiologist at the time of protocol development. We chose 5 referral hospitals, each representative of one of the regions in Uganda with bed capacities ranging from 350 to 1500. They are equipped to perform both major and minor surgery.
Study Population Figure.: Flow diagram showing hospital sampling. SSC indicates Surgical Safety Checklist; WHO, World Health Organization.
Ethical approval was sought from the Makerere University School of Medicine Research and Ethics Board, Uganda National Council of Science and Technology and the participating hospitals’ ethics boards. Written informed consent was obtained from all study participants. The study participants were enrolled systematically into the study by probability proportional to the size sampling. All patients undergoing surgical operations were recruited into the study regardless of their age, surgical specialty, and the urgency of the surgical operation provided consent to participate in the study was given. Of the 16 referral hospitals in Uganda, 5 hospitals met the inclusion criteria and participated in the study. The Figure illustrates this. It was explained to the hospital directors that participation in the study was free of repercussions such as denial of services from the Ministry of Health and other service providers or victimization regardless of the study findings. If a significant adverse event was identified by the researchers, this was reported to the hospital director who was then expected to handle the matter in confidence. Hospitals remained anonymous to prevent victimization of the hospitals as this was a compliance study. In a bid to prevent the Hawthorne effect, only the institutions’ directors were told the purpose of the study and the rest of the institution was blinded to the true purpose of the study. We found that the hospitals had used WHO SSC as is with no local modifications contrary to the suggestion from the WHO. The sample size was calculated using the modified Kish and Leslie method (1965) with a 95% confidence interval (CI) and powered at 80%. We considered that 50% of the study population had poor compliance with the SSC because we could not find previous studies assessing compliance with the SSC. This proportion was obtained from the WHO recommendation when no previous proportions are available. Because our study was done among several centers, we applied a WHO standard design effect of 2 which doubles the sample size. In addition, we added a 10% increase in sample size to cater for any losses to follow-up. The resulting sample size was 845.
Study Procedure
We trained nurses and intern doctors as research assistants. They were present in the operating room and observed the activities in the room throughout the surgery and completed a structured questionnaire regarding demographics, the 19 parameters and 3 domains of the WHO SSC, near misses, and intraoperative variables. Compliance was expressed as the percentage of items that were completed per checklist. Technical and intraoperative complications were recorded as well. The patients were followed up and questioned on the occurrence of complications at 24 hours and daily until discharge or 30 days postsurgery whichever came first. The nurses on the ward reported any immediate postoperative complications and the patient's charts were reviewed for documentation of complications. The complications were recorded as defined by the American College of Surgeons National Surgical Quality Improvement Program database. At the end of the follow-up period, the questionnaire was checked for completeness and accuracy before data entry.
This manuscript adheres to applicable Strengthening The Reporting of Observational Studies in Epidemiology (STROBE) guidelines.
Statistical Analysis
Data were analyzed using STATA version 13 (StataCorp.2013.Stata Statistical Software: Release 13, StataCorp LP, College Station, TX). The primary outcome was compliance with the WHO SSC. Analyzed data are presented in tables and text.
Descriptives.
Descriptives were summarized using means and standard deviations for the parametric continuous data. Categorical data were summarized using proportions and percentages.
Prevalence of compliance was calculated as follows: the numerator was the number of checked off items on the checklist. The denominator was the total number of expected checked off items. We expressed compliance as a percentage completion of the items comprising the checklist. There is a total of 19 items in the checklist; therefore, checklist compliance was defined as “noncompliance” (none of the 19 items checked off), “low compliance” (at least 1 to 9 items checked off), and “high compliance” (10–19 items checked off) and was expressed as a percentage of the number of the expected checked off items. We assigned each of the items of the checklist a score of 1, then calculated a percentage of the compliance by dividing the score attained out the 19 which is the expected score. We presented the mean percentage compliances with their corresponding 95% CIs.
Bivariate Analysis.
A binary logistic regression model was used to assess for the association of each of the predictors with mortality while a linear regression model was used to asses for the association with the length of hospital stay.
Multivariable Analysis.
All factors with a P value of .2 or less at bivariate analysis were simultaneously entered in multivariable linear and logistic regression models, respectively. Confounding was assessed for at a 10% or more difference between the unadjusted and adjusted models. We assessed for confounding after assessing for hospital effects and interaction effects and none were found. Statistical significance was defined as P < .05.
We performed linear and logistic regression analysis to determine the association between compliance with the SSC as the main predictor and length of hospital stay, perioperative adverse events, and 30-day mortality, respectively. Results of the linear regression were expressed as coefficient and those of logistic regression expressed as OR with a CI of 95%.
RESULTS
A total of 859 patients were enrolled in the study. Distribution of recruited patients among the different hospitals was fairly similar. Fifty-eight percent of the recruited patients were from Obstetrics and Gynaecology, mostly cesarean sections and were American Society of Anaesthesiologists physical status I and II. The average duration of surgery was 1 hour. Average length of hospital stay was 4 days. Of the total number of recruited patients, 4% (36) died within 30 days of follow-up.
The WHO SSC was initiated in 44% of the enrolled surgeries. Nurses initiated the WHO SSC in 28% of the surgeries and were present more frequently during the performance of the different domains of the WHO SSC. Table 1 shows the different team members present during the different domains of the WHO SSC. Of the recruited participants, 287 (33.9%) were elective surgeries while 559 (66.1%) were emergencies. 33.9% (287) surgeries were performed under general anesthesia while 66.1% (559) were performed under regional anesthesia. Mean overall compliance with the WHO SSC was 41.7% (95% CI, 39.7–43.8) ranging from 11.9% to 89.8% in the different hospitals. Overall compliance with “sign in” was 44% (95% CI, 43–45.6), with “time out” was 42.0% (95% CI, 39.4–44.6), and with “sign out” was 33.3% (95% CI, 30.7–35.9) (Table 2 ).
Table 1.: Team Members Present During the Different Phases of the Checklist
Table 2.: Compliance With the WHO SSC
There was no statistically significant association between compliance with the WHO SSC and perioperative surgical outcomes: 30-day mortality and length of hospital stay. There was a significant association between compliance with the WHO SSC and the incidence of pain and loss of consciousness. We also found that the odds of death were significantly higher compared to the rest in hospital 4 (Tables 3–5 ).
Table 3.: Association Between Compliance With WHO SSC and Adverse Events
Table 4.: Logistic Regression Showing Association Between Compliance With WHO SSC and 30-d Mortality
Table 5.: Linear Regression Showing the Association Between Compliance With WHO SSC and Length of Hospital Stay
The WHO SSC was performed in 43.5% of the enrolled surgeries and a copy was present in only 5% of the patient’s files of the recruited patients.
In about 80% of the enrolled surgeries, pulse oximeter placement was adhered to during the “sign in” component of the WHO SSC. Ascertaining whether the patient had a known allergy was the worst performed (4.4%). Antibiotic prophylaxis was not given in the recommended time as per WHO standards in about half (49.6%) of the enrolled surgeries. Instrument, needle, and sponge counts were done at “sign out” in less than half (41%) of the enrolled surgeries.
DISCUSSION
We found low levels of compliance across the hospitals with no significant association between this compliance and perioperative surgical outcomes. Mean compliance with the WHO SSC was about 40% with a wide variation across the different participating hospitals and across the African hospitals that found a 48.5% mean use of the WHO SSC with variation between 10% and 90%. Uganda’s compliance was 10%.9 5
The reasons for low compliance with the WHO SSC include lack of leadership, teamwork, and enforcement of the use of the checklist; lack of training and awareness on WHO SSC use, socioeconomic norms, and cultural barriers where hierarchy prevents some team members from initiating the WHO SSC, and resource limitations in terms of stationery, inadequate staffing, and time constraints.9 , 10 11
Van Schoten’s12
Compliance was better with the “sign in” and “time out” phases but worse with the “sign out” phase. From experience, priorities at the end of surgery differ for the different team members. This result is comparable to the findings from van Klei et al’s5 13 12
Previous studies have proposed direct and indirect mechanisms by which the checklist improves surgical outcomes. An indirect mechanism would be improved teamwork and communication among the entire surgical team which will likely improve safety in the perioperative period.14
Unlike in previous studies,15 16 , 17 18
Strengths of this study include the fact that a prospective multicentre cohort study reduces the risk of recall bias and improves the generalizability of the results. As one of the larger studies of its kind in sub-Saharan Africa, it adds valuable information to the pool of data on the WHO SSC. Our main limitation was the small sample size compared to similar studies that may have prevented recognition of some adverse events that occur less frequently. A sample size of about 3500 patients would have been sufficient to achieve this objective. The Hawthorne effect cannot be ruled out in a directly observed study such as ours. We did try, however, to reduce its effect by ensuring that the investigators were familiar to the surgical team and trained them to document their observations in a manner that would not raise suspicion although in doing this, we could have increased the risk of observer bias. We were unable to collect data on age and sex of the study participants, an oversight on our part and this limited our analysis.
CONCLUSIONS
We set out to find the extent of compliance with the WHO SSC and the association of this compliance with perioperative surgical outcomes. We found that compliance with its use in Uganda’s referral hospitals showed an association with the increased incidence of pain and loss of consciousness postoperatively. Following these results, we recommend that a larger study, well powered to detect an association of compliance with perioperative adverse events, is done to demonstrate generalizability of the results. We also recommend that further research associating pain scores with use of the WHO SSC be considered. E
ACKNOWLEDGMENTS
The authors thank the Government of Uganda through the Ministry of Health who sponsored the first author’s Master of Medicine in Anaesthesia and Critical care. Anaesthesia and Intensive Care consultants partially funded the study. Many thanks go to the following doctors; Dr Stephen Ssenyonjo Ttendo, Dr Joseph Kyobe Kiwanuka, Dr George Kateregga, Dr Davidson Ocen, Dr Amanda Akatukunda, Dr Fred Bulamba, and Ms Bernadette Kamaria who were of great help during the data collection process at the different study sites and Dr John Baptist Kiggundu who was instrumental in data management and result analysis. We are grateful to Dr Janat Tumukunde, Dr Peter Kaahwa Agaba, Dr Mary Theresa Nabukenya, Dr Emmanuel Timarwa Ayebale, Dr John Mark Kasumba, Professor Joan Kalyango, Dr Pauline Bakibinga, and Dr Abdhalah Ziraba for their invaluable contribution to this final report.
DISCLOSURES
Name: Elizabeth N. Igaga, MBChB.
Contribution: This author helped with designing the research, sought ethical approval, trained the data collectors and data entrants, wrote the report, and read and approved the final manuscript.
Name: Cornelius Sendagire, MBChB.
Contribution: This author helped with designing the study protocol, analyzing the data, writing the final report, and read and approved the final manuscript.
Name: Samuel Kizito, MBChB.
Contribution: This author helped with conceiving the study, obtaining ethical approval, cleaning and analyzing the data, as well as writing the final report, and read and approved the final manuscript.
Name: Daniel Obua, MBChB.
Contribution: This author helped with designing the study, obtaining ethical approval, writing the final report, and read and approved the final manuscript.
Name: Arthur Kwizera, MBChB.
Contribution: This author helped with designing the study, obtaining ethical approval, data collection, writing the final report, and read and approved the final manuscript.
This manuscript was handled by: Angela Enright, MB, FRCPC.
REFERENCES
1. Patel J, Ahmed K, Guru KA, et al. An overview of the use and implementation of checklists in surgical specialities - a systematic review. Int J Surg. 2014;12:1317–1323.
2. de Vries EN, Prins HA, Crolla RM, et al.; SURPASS Collaborative Group. Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med. 2010;363:1928–1937.
3. Weiser TG, Haynes AB, Dziekan G, Berry WR, Lipsitz SR, Gawande AA; Safe Surgery Saves Lives Investigators and Study Group. Effect of a 19-item surgical safety checklist during urgent operations in a global patient population. Ann Surg. 2010;251:976–980.
4. Haynes AB, Weiser TG, Berry WR, et al.; Safe Surgery Saves Lives Study Group. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360:491–499.
5. van Klei WA, Hoff RG, van Aarnhem EE, et al. Effects of the introduction of the WHO “Surgical Safety Checklist” on in-hospital mortality: a cohort study. Ann Surg. 2012;255:44–49.
6. Walker IA, Reshamwalla S, Wilson IH. Surgical safety checklists: do they improve outcomes? Br J Anaesth. 2012;109:47–54.
7. Russ S, Rout S, Caris J, et al. Measuring variation in use of the WHO surgical safety checklist in the operating room: a multicenter prospective cross-sectional study. J Am Coll Surg. 2015;220:1–11.e4.
8. Lilaonitkul M, Kwikiriza A, Ttendo S, et al. Implementation of the WHO Surgical Safety Checklist and surgical swab and instrument counts at a regional referral hospital in Uganda - a quality improvement project. Anaesthesia. 2015;70:1345–1355.
9. Kariyo P, Hightower J, Ndihokubwayo J, Tumusiime P, Mwikisa C. Challenges facing the introduction of the WHO surgical safety checklist: a short experience in African countries. Afr Health Monitor. 2013;36:9.
10. Ratcliffe A, Kintu A, Byamugisha J. Barriers to the use of the World Health Organisation Surgical Safety Checklist at Mulago National Referral Hospital. 2016.Mbale Uganda Association of Anesthesiologists of Uganda, Makerere University.
11. Jammer I, Ahmad T, Aldecoa C, et al.; European Surgical Outcomes Study (EuSOS) group. Point prevalence of surgical checklist use in Europe: relationship with hospital mortality. Br J Anaesth. 2015;114:801–807.
12. van Schoten SM, Kop V, de Blok C, Spreeuwenberg P, Groenewegen PP, Wagner C. Compliance with a time-out procedure intended to prevent wrong surgery in hospitals: results of a national patient safety programme in the Netherlands. BMJ Open. 2014;4:e005075.
13. Vogts N, Hannam JA, Merry AF, Mitchell SJ. Compliance and quality in administration of a Surgical Safety Checklist in a tertiary New Zealand hospital. N Z Med J. 2011;124:48–58.
14. Jager E, McKenna C, Bartlett L, Gunnarsson R, Ho Y-H. Postoperative adverse events inconsistently improved by the World Health Organization Surgical Safety Checklist: a systematic literature review of 25 studies. World J Surg. 2016;40:1842–1858.
15. Urbach DR, Govindarajan A, Saskin R, Wilton AS, Baxter NN. Introduction of surgical safety checklists in Ontario, Canada. N Engl J Med. 2014;370:1029–1038.
16. Rodgers A, Walker N, Schug S, et al. Reduction of postoperative mortality and morbidity with epidural or spinal anaesthesia: results from overview of randomised trials. BMJ. 2000;321:1493.
17. Kettner SC, Willschke H, Marhofer P. Does regional anaesthesia really improve outcome? Br J Anaesth. 2011;107suppl 1i90–i95.
18. Khuri SF, Daley J, Henderson W, et al. The National Veterans Administration Surgical Risk Study: risk adjustment for the comparative assessment of the quality of surgical care. J Am Coll Surg. 1995;180:519–531.
