Secondary Logo

Validation of an Instrument to Measure the Results of Quality Assurance in the Operating Room

Gomes, José Augusto Pereira; Martins, Maria Manuela, PhD; Tronchin, Daisy, PhD; Fernandes, Carla Sílvia, PhD

Quality Management in Healthcare: April/June 2019 - Volume 28 - Issue 2 - p 103–107
doi: 10.1097/QMH.0000000000000205
Methods and Instruments
Free
SDC

Background and Objectives: This study aims to validate an instrument designed to measure care quality in the operating room.

Methods: We conducted a transversal exploratory descriptive study with the objective of analyzing the psychometric properties of the scale. The 13 items included in the scale result from 2 previously conducted studies, a literature review study and a Delphi study. The instrument was administered to health professionals with at least 2 years of experience in the operating room in 71 Portuguese hospitals.

Results: The sample consisted of 1019 professionals who worked in an operating room; among them were nurses and doctors. The results show that the scale is a reliable and valid instrument composed of 3 dimensions: assessment of the quality of care; evaluation of processes; and control systems.

Conclusion: The final version of the scale meets the validity requirements, revealing high potential to be used in research and clinical practice. The use of this scale will allow evaluation and comparison of the results among the different operating rooms.

School of Nursing, Porto, Portugal (Dr Martins); School of Nursing, University of São Paulo, Sao Paulo, Brazil (Dr Tronchin); and School of Health, University Fernando Pessoa (UFP), Porto, Portugal (Dr Fernandes). Mr Gomes is Senior Training Consultant at Lima, Portugal.

Correspondence: José Augusto Pereira Gomes, Rua do Cidral n° 28, 4490-562 Póvoa de Varzim, Portugal (japgomes@gmail.com).

The authors have no conflicts of interest to disclose.

The operating room is the revenue center of any hospital and maximizing its efficiency has important implications for cost savings, patient satisfaction, and team satisfaction.1 Health care is currently under great pressure to seek efficiency, quality, safety, and satisfaction of the needs of its users.2,3 To measure performance and efficiency of the operating room, hospitals need indicators that display and monitor key performance indicators.4 Efficient and accurate measuring systems serve as a useful tool, allowing managers to monitor and improve the performance of health services.5 One of the main challenges in measuring and monitoring performance of the operating room is to determine which indicators to include.1

Most institutions select different combinations of metrics, based on local requirements and standards, many of which are familiar: number of surgical procedures performed outside normal hours; same-day cancellation fee; accuracy of the start time of the first case (or the time of the incision); use of operating rooms; percentage of unplanned cancellations; accuracy of the duration of the case; turnover times; and staff costs.1

In a study conducted to identify indicators in the operating room, the authors identified the following among them: rates of idle time; delay of surgery time; time of exchange between surgical procedures; cancellation of surgery on the same day; the number of monthly surgical procedures in each room; verification of surgical patients; surgery site; history of allergy; use of antibiotics 60 minutes before the incision; expected surgical time; results of sterilization indicators; availability of surgical instruments and materials; and number of instruments.6

Ideally, the indicators should consist of data already available in the operating room information systems (readily measurable). However, there appears to be a variation among hospitals in terms of which data and indicators should be collected representing significant challenges to external benchmarking.4 The measurement of results should be a multidimensional structure that involves the various components that contribute differently to the overall performance of the operating room.5

Only with uniform data and clear definitions of indicators can hospitals proceed with benchmarking and sharing learning processes, leading to improvement in practices.4 Traditional performance measurements generally use primarily financial aspects to measure performance.5 At this level, a study was conducted in the United States looking for the answer to the following questions: What is the cost of 1 minute of operating room time? And what contributes to this cost? The authors report that the average cost of operating room time in 2014 was $36 to $37 per minute, incorporating $20 to $21 for direct costs, $13 to $14 for salaries and benefits, and $2.50 to $3.50 for surgical materials.7 The construction and use of performance measurement tools are useful because they not only increase the efficiency of evaluation but also save costs.5 In addition to the simple financial savings, improved operating room efficiency can have a myriad of side benefits, including increased customer safety, increased throughput, and better customer and team satisfaction.1

Aware of this need, this article intends to validate a scale on outcome indicators in the operating room (Resultados da Avaliação da Qualidade no Bloco Operatório (RAQBO) scale). We believe that this instrument can be used to monitor and compare health institutions and improve the quality of care provided in the operating room.

Back to Top | Article Outline

METHODS

We conducted a transversal exploratory descriptive study with the objective of validating the psychometric properties of a scale to measure the Quality of Care Results in the Operating Room, known as the RAQBO scale.

The data collection instrument included a first part consisting of sociodemographic and professional characterization, followed by 13 items of the scale. Collection of the scale was carried out through an online questionnaire during the months of January to May 2018. The inclusion criterion for the study was to be a nurse or doctor active in the operating room for more than 2 years in Portugal.

Construction of the items of the scale is based on 2 previous studies, a review study on instruments to evaluate the quality in the operating room8 and a study using expert opinion to validate the selected indicators.3 These 2 steps resulted in a list of 13 items. These are 13 statements to which the degree of agreement is established through a Likert-type scale with 5 possible answers: 1 = totally disagree; 2 = partially disagree; 3 = indifferent; 4 = partially agree; and 5 = totally agree.

After obtaining the data, a database was developed in SPSS (Statistical Package for the Social Sciences), version 24, for Windows, that allowed us to characterize the sample and to determine validity and reliability of the scale. For the psychometric study, recommendations of the literature on the subject were followed9: namely, use of the Kaiser-Meyer-Olkin (KMO) sampling adequacy test to compare the simple correlations and analysis of main components with orthogonal varimax rotation. A factor analysis was performed on the basis of the Kaiser rule, excluding factor loadings lower than 0.30 if they existed. A total explained variance greater than 40% was used and the internal consistency evaluation was assessed using Cronbach's α.

The study was conducted after authorization from the institution and favorable assessment by the Ethics Committee (code no. CES246-16). Throughout the study, anonymity and voluntary participation were maintained. The informed consent form was obtained through the electronic format, where the participant accepted or did not accept the conditions of the study.

Back to Top | Article Outline

RESULTS

The participants answered all the questions on the scale. The sample consisted of 1019 participants, of whom 26.6% were doctors and 73.4% were nurses. The majority of respondents (69.2%) were female, ranging from 24 to 68 years old (mean = 43 years, standard deviation = 9.7 years). The time of professional tenure in the operating room ranged from 2 to 42 years (mean = 14.6 years, standard deviation = 9.1 years).

In the present study, it was found that the participants' answers ranged across all the points of the scale (i.e., 1-5), demonstrating that the scale structure is appropriate to evaluate the construct under analysis (Table 1).

Table 1

Table 1

After the individual study of the items of the RAQBO scale, we now proceed to an analysis of the latent structure through a factor analysis in order to identify the factors underlying this evaluation. These factors allow understanding of the larger concepts behind the pattern found in the answers, which allow us to measure validity of the instrument for the intended purpose.

Validity of the factor analysis was assessed by the KMO adequacy measure, with an overall value of 0.94 (Table 2). This value is very high, just like the values for each item. We conclude that all the items can be used because they fit the structure defined by the other items. As a result, we can state that the factor of the correlation matrix is very good; in other words, it is appropriate to perform a factor analysis with these data.

Table 2

Table 2

The exploratory factor analysis was performed, and 3 factors were obtained that explain 67.55% of the total variance (Table 3). The 3-factor solution, as suggested by the Kaiser rule, explains an acceptable percentage of the variance, involves an acceptable number of factors, and is the best solution in terms of their interpretation and meaning.

Table 3

Table 3

Table 4

Table 4

Table 3 shows the matrix of main factors, obtained by the varimax rotation method and the Kaiser normalization, where the factor weights of the different items in each factor are indicated with reference to the commonalities. In the factor analysis, no item was excluded (all presented a correlation value of higher than 0.30). As a result, it was concluded that the 3 factors are sufficient to describe the underlying structure of the data.

After performing the rotation and before a satisfactory factor solution, meanings were assigned to the factors, taking into account the theoretical reference. Thus, the dimensions that make up the scale were named: assessment of the quality of care; evaluation of the processes; and control systems.

“Assessment of the quality of care” was the first dimension obtained. It consists of 6 items that refer to quality assessment methods, namely, construction of indicators, health gains, satisfaction assessment, complications, and return to the operating room. This first factor explains 53.33% of the total variance.

The second dimension, “evaluation of the processes,” is composed of 4 items that explain 7.28% of the variance. These items refer to the work processes, namely, the evaluation of waiting times, teamwork, and the profitability of the operating room.

“Control systems” was the last dimension obtained. Composed of 3 items, it explains 6.93% of the variance. This dimension is composed of items that refer to the importance of audits to monitor the results and the issues related to the performance monitoring of the professionals working in the operating room.

The matrix of Spearman correlations between the factors is shown in Table 4. All correlations are significant and positive, with a strong correlation between the factors of “assessment of the quality of care” and “evaluation of processes.”

To evaluate the quality of the factor model, we observed differences between the simple correlations and the correlations estimated by the factorial model with the 3 factors retained. There are 26 residues (33%) with an absolute value greater than 0.05, which indicates a good quality of the adjustment. In addition, the adjustment quality index or goodness-of-fit index is 0.913, which indicates a good quality of adjustment.

Finally, internal consistency was assessed using Cronbach's α coefficient. The value of the entire scale was 0.925, which is a very high value and shows a very strong internal consistency of the instrument. The Cronbach's α for each dimensions is very good (assessment of the quality of care: 0.901; evaluation of the processes: 0.819; control systems: 0.747).

Back to Top | Article Outline

DISCUSSION

The operating room is a complex environment that comprises multiple interactions, where it is completely unacceptable if performance does not approach perfection and therefore needs metrics to evaluate its efficiency.1,4,10–12 In the present study, the RAQBO scale, composed of 13 items, was validated to measure quality results in the operating room as perceived by a population of Portuguese health professionals. The value obtained from the KMO measure was 0.94, which is considered excellent.

As for the study of internal consistency through Cronbach's α coefficient, the value was excellent (0.925). Cronbach's α is indicated to calculate the internal consistency of a measuring instrument in the case that Likert or multiple-choice scales are adopted and whose categories have an ascending or descending order of values.13 Cronbach's α reflects the degree of covariance among the items in a scale; values greater than 0.7 are ideal.9,14,15 From the exploratory factor analysis with varimax rotation, the 13 items of the scale were grouped into 3 dimensions that account for 67.55% of the total variance.

The dimension “assessment of the quality of care,” consisting of 6 items, refers to the definition of some indicators and their importance. One of the main challenges in the development of an instrument for measuring and monitoring the performance of the operating room is to define which indicators should be included.4 The balance between cost reduction and efficiency ensures patient safety is a continuous challenge, and it is important to develop strategies that reduce inefficiencies and provide personalized metrics in real time.16 Monitoring through indicators presents benefits to the respective institutions, measuring the quality of care at the time of surgery, the shorter length of stay for specific procedures, and better patient perception of quality of care.17,18 However, Fixler and Wright4 contested some of these qualitative measures, as in the case of satisfaction assessment, since most of them are not validated.

The dimension “evaluation of the processes” integrates indicators of process evaluation that refer, as the name indicates, to the monitoring of the processes of the operating room, among them the waiting times between the surgical procedures. Some authors are more specific in relation to these metrics, namely, the number of cases cancelled, the average time of rotation, the average time of occupation, the time of beginning of the first case, the mean time of induction, the mean time of extubation, and the mean time until recovery, among others.16

Another topic addressed in this dimension is the monitoring of teamwork. Effective teamwork is critical to the quality of patient care and safety in the operating room. Teamwork behaviors below the ideal often cause adverse events or near misses.19,20 Monitoring the dynamics of the team in the operating room, the way the teams function, and the factors that facilitate or impede teamwork are essential for obtaining high-quality care.20

The last dimension addresses issues related to “control systems,” integrating audits and performance evaluation. Surgical audits were developed with the objective of evaluating and facilitating quality improvement. Audit is a quality assessment method that collects detailed clinical data that are used to improve the quality of care through feedback on its results, as well as to facilitate benchmarking among participating hospitals.21 These audits make it possible to evaluate the efficiency of the resources used and to identify areas for improvement.17

Despite its important social and economic positions, the health industry has lagged behind in the availability of key data on the process and outcomes of care.21 The assessment of care costs and their outcomes should be a priority, and this will only be possible through the incorporation of very precise metrics.

Back to Top | Article Outline

CONCLUSION

This effort in the development and validation of an instrument to measure the Quality of Care Results in the Operating Room (RAQBO scale) integrates a set of essential perceived performance indicators to measure, monitor, evaluate, and compare performance and efficiency.

The scale comprising 13 items is easy to apply, has good conceptual properties, and presents good reliability and validity. The instrument has 3 dimensions, which are as follows: quality of care assessment; process evaluation; and control systems; that allow us to evaluate the results in their entirety.

The indicators under evaluation provide managers with an assessment tool to ensure the safety and quality of care. It also allows identifying areas in need of improvement and establishing corrective measures, constituting a valuable tool for use in a hospital environment or in research to replicate in future studies. As a limitation of this instrument, we consider the fact that it is restricted to the questions of the result, as well as being tested in the Portuguese context.

Back to Top | Article Outline

REFERENCES

1. Rothstein D, Raval M. Operating room efficiency. Semin Pediatr Surg. 2018;27(2):79–85. doi:10.1053/j.sempedsurg.2018.02.004.
2. Sartini M, Spagnolo A, Panatto D, Perdelli F, Cristina M. Improving environmental quality in an operating room: clinical outcomes and economic implications. J Prev Med Hyg. 2013;54(2):75–79.
3. Gomes JA, Martins M, Tronchin D, Fernandes CS. A técnica de grupo focal na validação de conteúdo para avaliação da qualidade assistencial em bloco operatório. Braz J Surg Clin Res. 2017;21(2):88–93.
4. Fixler T, Wright J. Identification and use of operating room efficiency indicators: the problem of definition. Can J Surg. 2013;56(4):224–226. doi:10.1503/cjs.020712.
5. Lin Q, Liu L, Liu H, Wang D. Integrating hierarchical balanced scorecard with fuzzy linguistic for evaluating operating room performance in hospitals. Expert Syst Appl. 2013;40(6):1917–1924. doi:10.1016/j.eswa.2012.10.007.
6. Wu Q, Huang L, Xing M, et al Establishing nursing-sensitive quality indicators for the operating room: a cross-sectional Delphi survey conducted in China. Aust Crit Care. 2017;30(1):44–52. doi:10.1016/j.aucc.2016.04.003.
7. Childers C, Maggard-Gibbons M. Understanding costs of care in the operating room. JAMA Surg. 2018;153(4):e176233. doi:10.1001/jamasurg.2017.6233.
8. Gomes JA, Martins M, Fernandes CS. Instrumentos para avaliar a qualidade e segurança no bloco operatório—revisão integrativa. Cogitare Enferm. 2016;21(5). doi:10.5380/ce.v21i5.45640.
9. Mokkink LB, Terwee CB, Patrick DL, et al The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19:539. doi:10.1007/s11136-010-9606-8.
10. Tanaka M, Lee J, Ikai H, Imanaka Y. Development of efficiency indicators of operating room management for multi-institutional comparisons. J Eval Clin Pract. 2013;19(2):335–341. doi:10.1111/j.1365-2753.2012.01829.x.
11. Weerakkody RA, Cheshire N, Riga C, et al Surgical technology and operating-room safety failures: a systematic review of quantitative studies. BMJ Qual Saf. 2013;22(9):710–718. doi:10.1136/bmjqs-2012-001778.
12. Civil I. Operating room culture affects patient outcomes, and we should operate accordingly. ANZ J Surg. 2018;88(4):264–265. doi:10.1111/ans.14390.
13. Echevarría-Guanilo ME, Gonçalves N, Romanoski PJ. Propriedades Psicométricas de instrumentos de medidas: bases conceituais e métodos de avaliação. Texto Contexto—Enferm. 2017;26(4):e1600017. doi:10.1590/0104-07072017001600017.
14. Souza AC, Alexandre NC, Guirardello E. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol Serv Saúde. 2017;26(3):649–659. doi:10.5123/s1679-49742017000300022.
15. Cunha CC, Netob OP, Stackflethc P. Principais métodos de avaliação psicométrica da confiabilidade de instrumentos de medida. Rev Aten Saúde. 2016;14(49):98–103. doi:10.13037/rbcs.vol14n49.3671.
16. Gabriel R, Gimlich R, Ehrenfeld J, Urman R. Operating room metrics score card-creating a prototype for individualized feedback. J Med Syst. 2014;38(11):144. doi:10.1007/s10916-014-0144-8.
17. Perkins J, Chiang T, Ruiz A, Prager J. Auditing of operating room times: a quality improvement project. Int J Pediatr Otorhinolaryngol. 2014;78(5):782–786. doi:10.1016/j.ijporl.2014.02.010.
18. Cardoen B, Demeulemeester E, Beliën J. Operating room planning and scheduling: a literature review. Eur J Oper Res. 2010;201(3):921–932. doi:10.1016/j.ejor.2009.04.011.
19. Russ S, Rout S, Sevdalis N, Moorthy K, Darzi A, Vincent C. Do safety checklists improve teamwork and communication in the operating room? A systematic review. Ann Surg. 2013;258(6):856–871. doi:10.1097/SLA.0000000000000206.
20. Hull L, Arora S, Kassab E, Kneebone R, Sevdalis N. Assessment of stress and teamwork in the operating room: an exploratory study. Am J Surg. 2011;201(1):24–30. doi:10.1016/j.amjsurg.2010.07.039.
21. Govaert J, van Bommel A, van Dijk W, van Leersum N, Tollenaar R, Wouters M. Reducing healthcare costs facilitated by surgical auditing: a systematic review. World J Surg. 2015;39(7):1672–1680. doi:10.1007/s00268-015-3005-9.
Keywords:

health care quality assurance; surgical procedures; process and results evaluation; operating room

© 2019Wolters Kluwer Health | Lippincott Williams & Wilkins