Process evaluation: evidence-based quality indicators in clinical practice, scholarship and pedagogy

Evaluation is fundamental to evidence implementation; the Joanna Briggs Institute (JBI) model indicates that for evidence-based healthcare (EBHC) to be implemented, evaluation is a critical component to demonstrating measures of impact or sustainability of a change process.¹ The lack of robust evaluation studies has, historically, been one of the perennial criticisms of the EBHC field. Compounding the criticisms of failures in evaluation, is that quantitative reviews with meta-analysis might tell us (with a degree of precision or confidence) the effect size of an intervention for a given outcome, but this is not the same as evaluating the impact on the processes or delivery of clinical practice. In this editorial, we will address process evaluation within the context of evidence implementation.

Process evaluation can be understood in more than one way. First, from the Donabedian principles, process evaluation facilitates examination of how care has been provided. More specifically in this context, process evaluation informs the appropriateness, acceptability, completeness or degree of competency, for example measuring changes in waiting times before surgery, percentage of patients with diabetes who receive their annual foot inspection, et cetera. Process evaluation in this context uses quality indicators, for example measuring protocol adherence using audit of compliance to recommended care for immediate newborns. Using audit of nine process indicators (e.g. the percentage of healthcare staff that received education on essential newborn care; the percentage in which exclusive breastfeeding was initiated within 1 h of birth), Kitila et al.² show that their implementation effort resulted in higher compliance to recommended care. In this sense, evaluation of process is used as feedback, or ‘wake-up call’, to show areas for clinical improvement. Process evaluations informed by Donabedian mode of evaluating the quality of care are reported in each JBI Implementation Case Report, with several important examples in this issue.

Three implementation case studies in this issue include process evaluations, comparing current practice against evidence-based quality indicators for high-quality care. Each study included a baseline measure of current practice, followed by site-specific implementation strategies then a postimplementation audit to observe, measure and report the extent to which processes of care had changed. Successful examples of process evaluation have been published in medication administration,³ nursing metrics for prevention and management of oral mucositis in chemotherapy, including oral hygiene, pain management and patient education,⁴ and how procedures for nonpharmacological analgesic interventions are implemented for newborn infants in Sao Paulo.⁵

Process evaluation also plays an important role in health services research. The Medical Research Council has published guidance on performing process evaluations on complex interventions.⁶ In this guidance, a process evaluation is defined as ‘a study which aims to understand the functioning of an intervention, by examining implementation, mechanisms of impact and contextual factors. Process evaluation is complementary to, but not a substitute for, high-quality outcomes evaluation’.^6(p.8) Other authors have used this to develop and test process evaluation of evidence-based care pathways and other implementation strategies.⁷

While both forms of process evaluation fit within the JBI model for EBHC, the role within quality improvement as form of evidence generation is widely taught and disseminated across the JBI Collaboration. In JBI, evidence generation is described as a legitimate means of knowledge creation to inform policy and practice arising from the results of well designed studies grounded in any methodological position. The challenge for JBI authors, methodologists and Collaboration is to consider how process evaluation fits within evidence generation and informs other components of the model. Moving from a statement to method and methodology that integrates with the JBI model has yet to be explored.¹ Future issues of JBI Evidence Implementation may well be the test cases for the development of a primary research agenda that is collaborative, aligned with the JBI model, informs and concurrently draws from relevant methods groups, and contributes to the JBI mission and vision. For 20 years, JBI has built a reputation on clinical implementation, on ensuring systematic reviews are both high quality and focused on generating recommendations and lines of action that meet knowledge needs in policy and practice. It is important to know if a certain intervention is proven effective (proof of concept). It is equally important to know if an intervention works in a specific context (proof in context).

The challenge for JBI and other EBHC groups is to consider how to integrate the external evidence operationally and define scholarship within this context. As described by the Dutch Council for Health and Society, it is not just clinicians and policy makers that need to consider integration of evidence in decision making, and the precedent actions.⁸ This shift will have strategic and operational consequences and must be part of an emergent learning process. JBI must therefore consider, plan for and design systems by making use of local data from practice, and planning for integration (integrated meaning within the JBI Model a suitable pathway is developed that closes the loop from evidence generation through to dissemination and knowledge translation) of operational, methodological and scholastic research. Such a process would support and build on evaluation of implementation and can then be extended into a formalized guide from which pedagogical innovations that further the knowledge on process evaluation can be translated throughout the Collaboration as has been the case with the current JBI Implementation Handbook.⁹

Acknowledgements

Conflicts of interest

The authors report no conflicts of interest.