Cataract surgery is the most common operation performed in the elderly, with more than 1.3 million cases per year in the United States alone (1,2). Advances in cataract surgical care have allowed virtually all cataract surgery to be conducted in an outpatient setting using local anesthesia. Major medical complications to patients are rare; adverse events occur in about 3% of patients and are limited to transient bradycardia or hypertension (1). In the light of this safety, many centers are reevaluating the rationale for costly intraoperative care processes (such as the provision of monitored sedation by anesthesiologists) to increase cost efficiency (3–5). What outcomes can be used to track the impact of these changes on the quality of cataract care?
Patient satisfaction ratings provide a means to evaluate and monitor quality of health care (6), especially in settings where adverse events are rare. Investigators have used simple ratings of patient satisfaction to assess the quality of their cataract care (7,8) or to evaluate specific components of that care (9–12). However, simple ratings of patient satisfaction can often be imprecise (13–15) or may not be measuring satisfaction at all (16). Ratings of patient satisfaction must show that they yield the same score if conditions remain the same (i.e., the measurement is reliable) and that they truly reflect patients’ satisfaction with their care and not just a feeling of gratitude or relief (i.e., the measurement must be valid) (6).
In 1997, Dexter et al. (13) developed the Iowa Satisfaction with Anesthesia Scale (ISAS) comprised of 11 items. Their scale exhibited good reliability and validity when applied to outpatients undergoing a variety of procedures under monitored care at a tertiary care setting. However, this questionnaire has yet to be tested in community hospitals or on cataract patients alone.
The purpose of this study was to evaluate the ISAS as a patient satisfaction questionnaire in the cataract clinic of a small community hospital in Northeastern Ontario, Canada. We hoped to establish whether the ISAS could be feasibly administered and to examine its reliability and validity when applied exclusively to patients undergoing cataract excision and lens implantation under topical local anesthesia with monitored sedation care given by an anesthesiologist.
The study was approved by the Ethics Committee of the North Bay General Hospital.
Cataract care at the North Bay General Hospital is conducted in a clinic location separate from the main operating room. An anesthesiologist assesses each patient and administers IV sedation as needed while monitoring vital signs during the procedure. All patients receive topical intraocular local anesthesia. After the operation, patients recover under the care of a nurse and are discharged home as soon as they have met standard discharge criteria (Aldrete score of 10 with minimal pain and nausea).
The ISAS is a written questionnaire consisting of 11 items (Appendix 1). Three items ask patients to rate their pain. Despite their similar wording, these questions tap different aspects of the patient’s experience (13). Six items ask patients to report on what else they may have felt during their anesthetic experience. Two items ask patients to directly rate their anesthetic experience. Each item uses the same six-point response format (disagree very much, disagree moderately, disagree slightly, agree slightly, agree moderately, agree very much) scored from +1 to +6 (where appropriate, scores of negatively worded items were reversed). The best possible score for each item is 6 and the final questionnaire score is reported as the mean score of all 11 items (max score = 6).
We obtained permission to use the scale from the original author. We preserved the original wording of the questionnaire instructions and all items and response sets. The questionnaire was formatted on single-sided paper in 16-pt font. The instructions were written on the first page followed by the questionnaire presented as two questions per page.
Five additional items were added to the end of the questionnaire that served as validity indicators for the ISAS. These included: 1. the degree of agreement to the statement, “There is nothing I would change in the care I just received”; 2. an evaluation of the quality of their intraoperative care on a scale of 1 to 6; 3. any advice that would make their care better; 4. a patient satisfaction rating (visual analog scale [VAS] score from 1 to 10); and 5. preference for more, the same, less, or no sedation if they were to have the operation again.
A pretest of this questionnaire was conducted on 10 patients. This pretest indicated that patients needed help in filling out this questionnaire because of infirmity of advanced age, blurred vision from the surgery, pain and discomfort, or the lingering effects of sedation. As a result, we chose to make the questionnaire either self-administered or interviewer-administered (SS) according to the preference of the patient.
We recruited every patient arriving for surgery on 2 days per week when a research assistant was available. We excluded patients who were non-English speaking, had psychiatric or mental incapacity (inability to provide consent), required general anesthesia, or who refused to participate. The two ophthalmologists, five (of eight) anesthesiologists, and all nursing staff agreed to participate. The manner of sedation was decided by each anesthesiologist.
All patients meeting inclusion criteria were asked to provide consent. Demographic information was gathered in a brief preoperative interview. Immediately after surgery, the anesthesiologist and circulating nurse were asked to complete the ISAS according to how they thought the patient would answer it. As the patient readied him/herself for discharge, he/she was asked to complete the ISAS. The time required to complete the questionnaire was recorded. The same research assistant (SS) collected all data and administered the questionnaire to all patients.
A sample size of 300 was chosen to allow multivariable regression analysis (17,18). We also based our sample size on the reported standard deviation of the ISAS of 0.87 (ó = 0.09) (13). A sample size of 300 allowed the ISAS score determined for all cataract patients to have a 95% confidence limit interval of 0.02.
First, to establish feasibility, the participation rate was tracked (patients who agreed to complete the questionnaire divided by the total patients asked to participate). Second, questionnaire response rate was measured: the proportion of completed ISAS questionnaires less the number of incomplete or illegible questionnaires and missing items over all questionnaires. Third, the average time to complete the questionnaire as well as the proportion of patients who required assistance were recorded. Finally, we estimated the cost of each questionnaire (printing costs and interviewer costs).
The internal consistency of the items of the ISAS (whether they were all measuring the same thing) was estimated using Cronbach’s α. To obtain estimates of test-retest reliability, 26 patients were administered the questionnaire twice; a Spearman correlation coefficient, intraclass correlation coefficient, and the weighted kappa between first and second questionnaire scores were calculated. The latter two statistics measured the concordance between the two ratings.
Construct validity of the ISAS was assessed in several ways. One, the patient’s ISAS score was correlated with that predicted by their anesthesiologist and circulating nurse. Two, the overall ISAS score (corrected to exclude that item’s score) was correlated with its own two single item ratings of global satisfaction (item 2: “I would have the same anesthetic,” and item 8: “I was satisfied with my anesthesia care”) as well as the satisfaction VAS score. Three, the ISAS score was correlated with alternate ratings of the overall quality of care; a single item evaluative rating of quality of care and a single rating on whether they agreed with the statement: “There was nothing I would change in the care I just received.”
As an additional measure of construct validity, we evaluated the ability of the ISAS to discriminate appropriately between patient groups. We examined the ability of the ISAS to predict patients who had higher ratings of satisfaction or quality, who wanted changes in their care, who had any advice or suggestions to improve care, or who would have preferred more sedation the next time.
All scores on ISAS and patient ratings were reported as means (standard deviation). Cronbach’s α, Spearman, weighted kappa, and intraclass correlation coefficients were used as statistics of reliability; Spearman, Pearson, and Kendall’s tau b correlation coefficients as statistics of validity. Two-tailed unpaired t-tests for tests of significance between patient groups were used where necessary to establish discriminant validity (i.e., differences in mean ISAS scores in patients who rated the quality of their experience as <6). P < 0.05 was used as the level of significance. All statistics were calculated using STATA.
We postulated the following a priori criteria that would indicate that the ISAS had achieved acceptable standards as a reliable and valid questionnaire. Cronbach’s α was expected to exceed 0.6 to allow group comparisons (17). Test-retest reliability and correlations with alternate measures of satisfaction (construct validity) were also expected to exceed 0.5.
We approached 366 patients arriving for surgery between January and May 2003. Thirty-eight patients were excluded because of language, communication difficulties, or mental illness; 22 did not participate (12 felt too rushed; 10 refused). Thus, 306 patients were included in the study.
Seventy percent of all patients were between 60 and 80 yr of age (Table 1); there were slightly more men than women. Most patients were Canadian born, 70% spoke English as a first language, and 70% had at least a high school education. Ninety percent of patients were ASA physical status score II or III and, of these, the majority had hypertension, diabetes, and/or heart disease. Forty percent of patients were undergoing cataract surgery on their second eye.
Participation rate was 93% (306 of 328 patients). We obtained complete ISAS questionnaires from all 306 patients who agreed to participate and there were no incomplete questionnaires. The average time to complete the questionnaire was 5.3 min (sd = 1.37). Eighty-one percent of patients required assistance with completing the questionnaire. If the clinic clerical staff assumed the duties of assisting in all questionnaires (at $25.00/hour) and each questionnaire cost $0.60, the cost of each questionnaire can be estimated as $2.80 (Canadian dollars).
The mean ISAS score for all patients was 5.60 (sd = 0.48) and the median score was 5.73 (range, 3.3–6.0). The statistic of skewness of 1.56 and a kurtosis 5.65 indicated that the data were not normally distributed. The single item quality of care rating (“Looking back on my surgery, I would say the quality of the experience was poor … perfect”) had lower mean score (5.1), better variability (sd = .89), and also had better statistics of normality (skewness 0.73; kurtosis 2.72) compared with the mean ISAS score. The other four single item ratings were substantially more skewed compared with the ISAS.
Using responses from 306 patients, the coefficient α for the ISAS was 0.68. Removing the contribution of 1 or more items to the ISAS score only served to increase α to a maximum of 0.73 (data not shown).
Test-retest reliability was obtained from 26 patients who were administered the ISAS. The average time between these 2 occasions was 4 h (range, 3–9 h). The Spearman correlation, weighted kappa, and intraclass correlation coefficient between the mean ISAS score on the two occasions were 0.67, 0.48, and 0.57, respectively (indicating moderate reliability) (19). Test-retest reliability of three alternate single item ratings of quality and satisfaction was also measured. The two items within the ISAS (items 2 and 8) that directly rated anesthetic care and quality of care yielded Spearman correlations (weighted kappa, intraclass correlation coefficient) of 0.51 (0.50, 0.53) and 0.0 (0.0, 0.0), respectively. The item asking patients to rate overall quality of their care on a 6-point scale had a test-retest reliability of 0.66 (0.45, 0.52). Thus, the test-retest reliability of the ISAS was equal or superior to single item alternate ratings of quality and satisfaction.
Correlational (“criterion”) validity (i.e., correlations between the ISAS and alternative ratings of patient care) were poor to fair (Table 2). ISAS scores did not correlate with the anesthesiologists’ or nurses’ ratings of the ISAS (Spearman rho = 0.00 and 0.11). Correlations of the ISAS score with the patient’s VAS score of satisfaction, or with whether patients wanted the same anesthetic again, or with 2 single item ratings of the quality of care were fair (rho = 0.24–0.35). The ISAS did not correlate with whether a patient had any suggestions to improve their care (rho = −0.14). It had moderate correlations with whether a patient agreed that there was nothing in their care that they would change (rho = 0.39).
The ISAS had excellent construct validity. When patients were dichotomized into groups based on high or low ratings on alternate measures of patient satisfaction or quality, the ISAS score matched these ratings. Furthermore, the ISAS was higher in patients who had higher ratings on these scales (Table 3). For example, patients who had scored <90% on their VAS satisfaction scale had significantly lower scores on their ISAS: 5.34 versus 5.67 (P < 0.0001), and this difference increased in patients who scored <80% on the satisfaction VAS: 5.12 versus 5.64 (P < 0.0001). Similarly, patients who rated their quality of care as 5 or less on the 6-point evaluative scale score, had lower ISAS: 5.32 versus 5.68 (P < 0.0001), and this difference was greater in those patients who rated their quality poorer (4 or less): ISAS 4.98 versus 5.64 (P < 0.0001).
The ISAS also demonstrated construct validity by distinguishing appropriately between patient responses on alternate ratings of quality of care (Table 3). ISAS scores were significantly lower for patients who did not strongly agree with the statement, “There is nothing I would change in the care I just received” compared with those that strongly agreed with that statement: 4.76 versus 5.67 (P < 0.0001). There were also significant differences in ISAS scores in patients who had any advice or suggestions to assist future care of patients (5.08 versus 5.63, P < 0.0001) and in patients who wanted more sedation if they were to have the same surgery again (4.85 versus 5.66, P < 0.0001).
We have shown that the ISAS questionnaire is a feasible, reliable, and valid tool for measuring patient satisfaction in patients undergoing cataract surgery with monitored care in a community hospital. Summed scores from the ISAS have sufficient reliability to allow group comparisons. In addition, the ISAS has excellent construct validity; mean ISAS scores were lower in patients who had less than perfect alternate ratings of quality and satisfaction.
In 1999, Dexter et al. (13) used psychometric techniques to develop the ISAS to measure patient satisfaction with monitored anesthesia care. To ensure content validity, items were suggested by staff (anesthesiologists, nurses, and administrators) and ratified with patients. When self-administered on 92 patients undergoing various outpatient procedures (53% undergoing cataract care) at a tertiary care center, they had a response rate of 82%. Patient responses to the ISAS demonstrated excellent reliability; α = 0.8; rsq (test-retest) = 0.74. There was also reasonable evidence of validity; correlations with provider ratings were good (rsq = 0.23) as were correlations with a single item rating of satisfaction (tau = 0.41). They proposed that this scale could be useful to assess the satisfaction of patients of their monitored anesthesia care or even in patients after general anesthesia.
Psychometricians, however, warn that questionnaire instruments may not retain their feasibility, reliability, and validity and should be retested when applied to different settings or circumstances (17). Therefore, we thought it necessary to establish that the ISAS retained its measurement properties when applied to cataract care exclusively in a Canadian community hospital. We also expanded on Dexter et al.’s original finding by confirming that the ISAS could discriminate appropriately among patients who rate their satisfaction and quality differently.
When administered as an interviewer-assisted questionnaire, we obtained a 100% response rate with no missing items from 306 patients. Although an interviewer can bias ratings positively by making patients more reluctant to criticize their care (20), many of our patients were not willing to complete the questionnaire alone or were often not capable of doing so. In addition, many patients were initially confused about how to choose between strongly (dis)agree versus slightly (dis)agree. Therefore, without an interviewer to clarify and read questions, there would have been a substantial loss of data. In our view, the improvement in feasibility resulting from the use of an interviewer in postcataract patients more than offsets the slight positive bias in our study attributable to that interviewer.
The closed question format of the ISAS made it easy to administer in the outpatient setting. Additional comments were spontaneously provided by some patients and many of these comments provided interesting insight into the quality of care at the hospital. However, because an open question format would take more time to administer, adopting open questions would make a questionnaire more cumbersome to use and likely too costly for monitoring patient satisfaction. We believe that, at $2.80 per ISAS questionnaire, the ISAS remains an affordable quality assurance tool for most cataract centers.
The reliability of the ISAS, estimated by two methods, was acceptable but not high. Cronbach’s α, a measure of internal consistency, was 0.68. This statistic should be viewed as the theoretical upper limit of reliability (17); values more than 0.6 are necessary for the scale to have sufficient reliability to allow group comparisons. Test-retest reliability also indicated moderate levels of agreement in the ratings obtained on two occasions. Both measures of reliability were less than that found in Dexter et al.’s original study. We suspect that lower reliability in our study is a result of differences in study population (Canadian, cataract patients only, older) and survey methods (use of an interviewer, longer time between retests). Although not major, these differences in reliability reinforce the necessity of retesting the reliability of questionnaire instruments in alternate populations.
Unlike Dexter et al. (13), we found that the ISAS scores predicted by nurses and anesthesiologists correlated poorly with actual scores by patients. The inability of health care providers to predict the priorities and perceptions of their patients has been a consistent finding in other studies (21–23). In cataract surgery, perceiving patient dissatisfaction may be inherently more difficult because there are often little overt signs of patient distress. The higher correlations found by Dexter et al. may be explained partially by their heterogeneous case mix that included more painful surgical procedures (e.g., oocyte retrieval).
The ISAS scores demonstrated excellent construct validity by being lower in groups of patients expected to be less satisfied. This ability of the ISAS to discriminate appropriately between patients expected to have differences in satisfaction was not examined by Dexter et al. Myles et al. (24) also found appropriate and significant differences in the Quality of Recovery score between patients who scored highly in a VAS recovery score to those who did not. The ISAS was also able to distinguish among patients who wanted more sedation, who had suggestions, or wanted changes in their care. Our findings strongly suggest that the ISAS score does, in fact, reflect a patient’s positive or negative evaluation of their experience.
In summary, we have demonstrated that the ISAS has sufficient reliability and validity to measure patient satisfaction with monitored sedation during cataract surgery under topical intraocular anesthesia at a Canadian community hospital. Such a tool has important implications for monitored cataract care. Examining what causes differences in ISAS scores can provide insights into what makes patients dissatisfied; tracking ISAS scores can evaluate the impact of changes in care on patients themselves.
The authors acknowledge the support and assistance of the study anesthesiologists: Drs. Sean Travers, Rajalaxmi Wong, and Steven Bodley; the surgeons: Drs. Ken Kesty and Jon Spencer; and the nurses and support staff of the cataract clinic at the North Bay General Hospital.
1. Schein O, Katz J, Bass E, et al. The value of routine preoperative medical testing before cataract surgery. N Engl J Med 2000;342:168–75.
2. Javitt J, Kendix M, Tielsch J, et al. A geographic variation in utilization of cataract surgery. Med Care 1995;33:90–105.
3. Rosenfeld S, Litinsky S, Snyder D, et al. Effectiveness of monitored anesthesia care in cataract surgery. Ophthalmology 1999;106:1256–61.
4. Pecka S, Dexter F. Anesthesia providers’ interventions during cataract extraction under monitored anesthesia care. AANA J 1997;65:357–60.
5. Dexter F. Application of cost-utility and quality-adjusted life years analyses to monitored anesthesia care for sedation only. J Clin Anesth 1996;8:286–8.
6. Fung D, Cohen M. Measuring patient satisfaction with anesthesia care: a review of current methodology. Anesth Analg 1998;87:1089–98.
7. Fan YP, Boldy D, Bowen D. Comparing patient satisfaction, outcomes and costs between cataract day surgery and inpatient surgery for elderly people. Aust Health Rev 1997;20:27–39.
8. Reshef DS, Reshef SH. Post-operative cataract surgery satisfaction in a rural Kenyan clinic. J Cataract Refract Surg 1997;23:277–83.
9. Frey K, Sukhani R, Pawlowski J, et al. Propofol versus propofol-ketamine sedation for retrobulbar nerve block: comparison of sedation quality, intraocular pressure changes, and recovery profiles. Anesth Analg 1999;89:317–21.
10. Rose K, Waterman H, Toon L, et al. Management of day surgery patients with cataract attending a peripheral ophthalmic clinic. Eye 1999;13:71–5.
11. Herrick I, Gelb A, Nichols B, Kirkby J. Patient-controlled propofol sedation for elderly patients: safety and patient attitude toward control. Can J Anaesth 1996;43:1014–8.
12. Cruise C, Chung F, Yogendran S, Little D. Music increases satisfaction in elderly outpatients undergoing cataract surgery. Can J Anaesth 1997;44:43–8.
13. Dexter F, Aker J, Wright J. Development of a measure of patient satisfaction with monitored anesthesia care. Anesthesiology 1997;87:865–73.
14. Brown DL, Warner ME, Schroeder D, Offord K. Effect on intra-operative anesthetic events on post-operative patient satisfaction. Mayo Clin Proc 1997;72:20–5.
15. Klock PA, Roizen M. More or better: educating patients about the anesthesiologist’s role as peri-operative physician. Anesth Analg 1996;83:671–2.
16. Keep P, Jenkins J. From the other end of the needle: the patient’s experience of routine anaesthesia. Anaesthesia 1978;33:830–2.
17. Nunally JC. Psychometric theory. 2nd ed. New York: McGraw-Hill, 1978.
18. Harrell F, Lee K, Matchar D, Reichert T. Regression models for prognostic prediction: advantages, problems, and suggested solutions. Cancer Treat Rep 1985;10:1071–7.
19. Landis J, Koch G. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 1977;33:363–74.
20. Streiner D, Norman G. Health measurement scales: a practical guide to their development and use. 1st ed. Oxford: Oxford University Press, 1994.
21. Millis S, Jain SS, Eyles M, et al. Assessing physicians’ interpersonal skills: do patients and physicians see eye-to-eye? Am J Phys Med Rehabil 2002;81:946–51.
22. Lampic C, Sjoden P. Patient and staff perceptions of cancer patients’ psychological concerns and needs. Acta Oncol 2000;39:9–22.
23. Fung D, Cohen M. What do outpatients value most in their anesthesia care? Can J Anaesth 2001;48:12–9.
24. Myles PS, Hunt JO, Nightingale CE, et al. Development and psychometric testing of a quality of recovery score after general anesthesia and surgery in adults. Anesth Analg 1999;88:83–90.