Stimpfel, Amy Witkoski PhD, RN; Rosen, Jennifer E. MPH, MSN; McHugh, Matthew D. PhD, JD, MPH, RN, FAAN
Excellence in nursing care has been associated with positive outcomes for both patients and the nurses who care for them.1 One standard for identifying organizational excellence in nursing is Magnet Recognition® from the American Nurses Credentialing Center (ANCC).2 The basis for Magnet Recognition includes 4 key domains: transformational leadership; structural empowerment; exemplary professional practice; and new knowledge, innovations, and improvements.2 A 5th domain, empirical outcomes, is embedded within each domain. Excelling in these areas not only supports nurses, which results in better retention and higher job satisfaction,3-5 but also appears to influence important patient outcomes like mortality.6,7
For 2 decades, researchers have sought to empirically evaluate whether differences in the work environment of nurses—the principal hospital bedside care provider—make a difference for patient outcomes.3 Magnet® designation provides a useful mechanism for evaluating this premise and, more significantly, an important tool for changing work environments if the better outcomes associated with Magnet hospitals can be attributed to their better professional practice environments (PPEs). In 1994, Aiken et al6 compared the mortality rates of Medicare patients in reputational Magnet hospitals versus non-Magnet hospitals and determined that Magnet hospitals had lower mortality rates even after controlling for hospital characteristics.
Recently, McHugh et al7 reported that patients cared for in Magnet hospitals had lower odds of mortality and failure-to-rescue than did patients in non-Magnet hospitals. Their work and others8,9 have suggested that a key factor in differentiating Magnets from non-Magnets is a PPE.
Despite the mounting evidence associating Magnet hospitals with superior outcomes for patients and nurses, fewer than 9% (n = 390) of American hospitals have undergone the voluntary accreditation process, which is both time and resource intensive.10 Given the select group of Magnet hospitals, comparing Magnet hospitals with all non-Magnet hospitals may lead to concerns of bias because some of the comparison hospitals could be dissimilar. For example, Magnet hospitals are disproportionately large, academic teaching hospitals located in urban areas. Research that accounts for the dissimilarities between Magnet and non-Magnet hospitals can give us increased confidence in the expectation for positive nurse and patient outcomes.
In the absence of a randomized controlled trial to assess the causal effects of Magnet status on outcomes, a variety of approaches have been developed to overcome some of the limitations of observational data. Two of these methods, matching and causal mediation analysis, are used in this article. Matching is used to achieve balance in covariates between Magnet hospitals and the matched non-Magnet hospitals, thereby reducing biases that may account for differences in quality of care.11 Mediation is used to identify important intermediate variables existing in causal pathways between the independent (eg, Magnet status) and dependent (eg, quality) variables.12
The purpose of this article was 2-fold; 1st, we aimed to compare a matched set of Magnet and non-Magnet hospitals to evaluate the difference in nurse-reported quality of care; in the 2nd aim, we sought to determine if the PPE mediates the relationship between Magnet Recognition and quality of care.
Design and Sample
This retrospective, observational study used a secondary analysis of cross-sectional data. Two data sets were merged using common hospital identifiers: (1) a multistate survey of nurses from California, New Jersey, Pennsylvania, and Florida13 and (2) the American Hospital Association’s annual survey of hospitals.14 The sampling approach was previously described by Aiken and colleagues.13 The nurse survey13 contained items related to nurses and their work environment, including workload, education, job satisfaction, work environment, demographics, quality of care, and patient safety indicators. Hospital-level data were merged with the publicly available list of Magnet hospitals, obtained from the ANCC Web site.10 The final sample of hospitals (N = 551) included Magnet (n = 56) and non-Magnet (n = 495) hospitals.
Quality of Care
Quality of care was the main dependent variable of interest, measured using a single item in the nurse survey, “How would you describe the quality of nursing care delivered to patients in your unit?” with options including excellent, good, fair, or poor. The individual nurse responses were aggregated to create a hospital-level measure of the proportion of nurses who reported “excellent” quality for analysis. This measure has good predictive validity and corresponds well with hospital quality indicators such as mortality, failure-to-rescue, and patient satisfaction.15
Magnet Recognition was the main independent variable of interest in this study. The ANCC data from 2007 were used to categorize hospitals in the sample as Magnet or non-Magnet. Hospitals that had attained Magnet Recognition in 2006 as well as newly recognized Magnet institutions in 2007 were considered Magnet institutions in this sample. We included the newly recognized Magnet organizations because it takes roughly a year to complete the accreditation process, during which the requirements to achieve Magnet status would likely be in place.7
The Practice Environment Scale of the Nursing Work Index16 was used to measure the PPE. This instrument is composed of 5 subscales including nurse participation in hospital affairs; nursing foundations for quality of care; nurse manager ability, leadership, and support of nurses; staffing and resource adequacy; and nurse-physician relations. This instrument has demonstrated good reliability and validity in both domestic and international samples of nurses16 and has been endorsed by the National Quality Forum.17 A hospital-level mean of the subscales was used for analysis.
Hospital Structural Characteristics
Several structural characteristics were used in our matching and regression analyses building on previous Magnet research.6,7 We used continuous variables for the total numbers of beds as an indicator of hospital size, the percentage of the hospital’s inpatient Medicare and Medicaid populations (separately),18 Herfindahl-Hirschman Index (HHI)19 for market competition, and case mix index (CMI).20 The CMI is a measure based on the average annual cost of a diagnosis-related group for a hospital’s Medicare patients.20 A higher CMI for a hospital indicates a sicker patient population, requiring more expensive interventions. We used dichotomous or ordinal variables for technology status, measured as their capability to perform organ transplantation and/or open heart surgery or not; teaching status, measured by the presence and extent of medical students, residents, and fellows; ownership; whether the hospital is nonprofit or for profit; and core-based statistical area, a measure of population density.
After descriptive statistics were conducted, ordinary least squares regression models were used to examine the relationship between Magnet status and quality of care, before and after controlling for hospital characteristics. Next, the Magnet hospital sample was matched with non-Magnet hospitals. The goal of matching is to create balance on the measured covariates such that Magnet and non-Magnet hospitals look similar to each other in all respects except for their Magnet status. We began by calculating a distance measure for each hospital (a propensity score) based on a logistic regression of the covariates of Magnet status. We then evaluated multiple matching algorithms to determine which approach yielded the best balance of the covariates. To evaluate balance, we assessed standardized difference in means that are nonsensitive to sample size and graphical displays including Q-Q plots, jitter plots, and histograms.21,22 After assessing multiple matching approaches (1:k nearest neighbor matching with and without replacement, optimal matching, genetic matching), we found that 1:1 nearest neighbor matching without replacement (N = 112; 56 Magnet and 56 non-Magnet hospitals) achieved greater balance of covariates between Magnet hospitals and the matched non-Magnet hospitals. All matching and balance assessment was conducted using the MatchIt package23 in the R statistical program. After matching, we used ordinary least squares regression models to evaluate the effect of Magnet status on quality of care. We included the same covariates in this model to account for any residual covariate imbalance.
For our 2nd aim, we sought to understand whether and the degree to which better nurse practice environments explained, that is, mediated, the relationship between Magnet status and nurse-reported quality of care. A set of sensitivity analyses were conducted to evaluate how robust our findings were to an unobserved confounding variable.24 We used the R package “mediate” to evaluate mediation.12 All analyses were conducted in Stata version 11 (StataCorp, College Station, Texas) and R version 2.15.1. Statistical significance was established at the P < .05 level for 2-tailed tests. Institutional review board approval was obtained for the parent study at the researcher’s institution.
We began our analysis for the 1st aim by comparing the structural characteristics of all study hospitals with the non-Magnet hospitals and Magnet hospitals. In Table 1, the 1st column shows characteristics for all 551 hospitals. The next 2 columns show the Magnet (n = 56) and non-Magnet (n = 495) hospitals for comparison. The differences of greatest magnitude in these 2 groups were bed size (398 vs 264), high-technology status (70% vs 43%), and ownership (100% vs 81%) for Magnet versus non-Magnet hospitals, respectively. The propensity, or likelihood, of being a Magnet hospital given the hospital characteristics differed considerably between Magnet (0.24) and non-Magnet (0.09) hospitals before matching.
The 4th column of Table 1 shows the matched non-Magnet hospitals (n = 56). The matched non-Magnet hospitals looked considerably more like Magnet hospitals with respect to hospital characteristics. For example, the average bed count of matched non-Magnet hospitals was 389 compared with 264 for all non-Magnet hospitals—much more similar to 398, the average number of beds of Magnet hospitals. Similarly, technology and teaching status improved when the sample was matched for non-Magnet hospitals. The percentage of patients on Medicaid and Medicare was similar even before matching, suggesting potentially comparable patient populations. To illustrate the improvement in the balance of the covariates, we show the standardized difference in means of the covariates in all the hospitals (before matching) compared with the matched hospitals in Figure 1.
Table 2 displays the results from the prematching and postmatching regression models. In the prematching models, Magnet status was significantly associated with quality of care. In unadjusted models, Magnet status was associated with an additional 11% (SE, 0.02; P < .001) of nurses reporting excellent quality. After adjusting for the hospital characteristics, Magnet hospitals were still associated with having an additional 10% (SE, 0.02; P < .001) of their nurses reporting excellent quality.
After matching, we found similar, albeit attenuated, results indicating the influence of the hospital characteristics. Magnet status remained significantly associated with nurses’ reports of excellent quality of care. In the fully adjusted model, there was an additional 7% (SE, 0.02; P = 0.003) of nurses in Magnet hospitals who reported excellent quality of care compared with non-Magnet hospitals.
Our 2nd aim was to explore whether the nurse practice environment explained quality of care differences observed between Magnet and non-Magnet hospitals. Results from the mediation model found, on average, an additional 6.4% (P = 0.00) of nurses in Magnet hospitals reported excellent quality compared with non-Magnet hospitals. Most of the increase (4.2 of the 6.4 points, or 63%) could be attributed to the effect of the superior practice environment, which in turn affects quality of care.
The sensitivity analysis, which sought to assess the robustness of our results, is graphically displayed in the Figure, Supplemental Digital Content 1, http://links.lww.com/JONA/A280. The sensitivity analysis is conducted to gauge the degree of violation of the key assumption of mediation; that is, there are no omitted confounders affecting both the practice environment and quality of care. The plot indicates that an omitted confounder must explain about 24% of the remaining variance in the outcome model and about 53% of the remaining variance in the mediation model, for a product of 13% of the total variance, in order for the mediation effect to be zero. This analysis provides an estimate of how sensitive our mediation model is to the possibility of potentially unmeasured or omitted variables that could influence our outcome. For example, it is possible that nurse staffing or skill mix may influence how nurses reported quality of care.
Our study found that a significantly higher proportion of the nurses in Magnet-recognized hospitals reported that the quality of care in their unit was excellent compared with nurses working in a matched set of non-Magnet hospitals. We found that the nurses’ PPE mediated the relationship between Magnet status and quality of care. This suggests that Magnet hospitals produce better quality of care through their superior practice environments.
Our findings point to the practice environment as being a key driver of the Magnet effect on quality of care. Because all organizations, Magnet and otherwise, have the potential to enrich their practice environment, every organization stands to benefit from improving the organization of nursing care. Drawing from core Magnet principles, organizations can model and build a quality practice environment. Examples include having a visible and accessible chief nurse, encouraging and including nurses in decision making in their unit and throughout the organization, and supporting nursing practice and engaging in interdisciplinary patient care. Moreover, enhancing features of the practice environment are readily modifiable unlike most structural features of a hospital. As policies aimed at enhancing hospital care and improving quality are established, the contributions of nursing are warranted in all stages of planning, development, and implementation.
When situating our findings in the broader healthcare quality literature, this study adds new insight into how nursing influences quality. Many of the recent efforts to improve quality and enhance transparency in healthcare, such as value-based purchasing, have been dominated by physician services and medical outcomes.25 Less effort has been focused on enhancing nursing care to improve outcomes, despite evidence that several nursing care features (eg, practice environment, nurse staffing, and shift length) have already been found to be related to patient satisfaction, 1 element of value-based purchasing.26,27 Our study highlights that the overall quality of care can be optimized when nurses work in a positive environment, with adequate resources and support at the organizational level.
Early Magnet researchers largely described specific features of a work environment that attracted and retained nursing staff, enabling nurses to provide quality care.28,29 As the literature around Magnet has grown and evolved, researchers have operationalized quality of care in different ways, that is, who and what define quality. More recently, particular measures such as mortality,6,7,30 patient falls,31 and pressure ulcers,32 among others, have been studied. The findings from this study complement much of the existing literature that has found specific quality outcomes to be better in Magnet hospitals by using a newly validated measure of overall quality of care.15 Future work on this topic could confirm our findings using alternative measures of quality and employ similar analytic methods, when feasible.
The results of this study should not be considered causal given the cross-sectional nature of the data. Longitudinal studies of Magnet attainment would be valuable in confirming the relationships found in this cross-sectional analysis. The main concern here would be determining temporality, that is, whether the better practice environments observed in Magnet hospitals and, in our findings, accounting for a significant proportion of their quality difference result from the Magnet process or are already present to some degree. Previous research suggests that the Magnet process is indeed transformative, but this is nevertheless an important area for further study.7,33 Our approach, however, strengthens the basis for causal inference by improving our ability to identify the effect of Magnet hospitals because we created balance on measured covariates between Magnet and non-Magnet hospitals. A caveat, however, is that although we matched on a comprehensive set of variables, it is possible that important unmeasured confounders were omitted. Lastly, common-method bias, or same-source bias, is a concern in organizational research because of the potential correlation between variables; however, there is disagreement regarding the degree of upward bias that actually occurs.34 In previous work using nurse survey data aggregated to the hospital level, a split-sample approach was carried out to evaluate the level of same-source bias, which was found to be negligible.35 Our study also used aggregated independent and dependent variables, which likely reduced the same-source bias.36
Nurses’ reports of quality of care were significantly associated with Magnet status, with higher quality reported by nurses in Magnet hospitals compared with matched non-Magnet hospitals. The PPE mediates the relationship between Magnet status and quality of care. Hospitals that invest in improving the nursing work environment have the potential to benefit from increased quality of care for their patients and families.
1. Aiken LH, Clarke SP, Sloane DM, Sochalski J, Silber JH. Hospital nurse staffing and patient mortality, nurse burnout, and job dissatisfaction. JAMA. 2002; 288 (16): 1987–1993.
2. American Nurses Credentialing Center. The Magnet Model Components and Sources of Evidence. Silver Spring, MD: American Nurses Credentialing Center; 2008.
3. Kelly LA, McHugh MD, Aiken LH. Nurse outcomes in Magnet® and non-Magnet hospitals. J Nurs Adm. 2011; 41 (10): 428–433.
4. Hess R, Desroches C, Donelan K, Norman L, Buerhaus PI. Perceptions of nurses in Magnet® hospitals, non-Magnet hospitals, and hospitals pursuing Magnet status. J Nurs Adm. 2011; 41 (7-8): 315–323.
5. Upenieks VV. The interrelationship of organizational characteristics of Magnet hospitals, nursing leadership, and nursing job satisfaction. Health Care Manag (Frederick). 2003; 22 (2): 83–98.
6. Aiken LH, Smith HL, Lake ET. Lower Medicare mortality among a set of hospitals known for good nursing care. Med Care. 1994; 32 (8): 771–787.
7. McHugh MD, Kelly LA, Smith HL, Wu ES, Vanak JM, Aiken LH. Lower mortality in Magnet hospitals. Med Care. 2013; 51 (5): 382–388.
8. Upenieks VV. Assessing differences in job satisfaction of nurses in Magnet and non-Magnet hospitals. J Nurs Adm. 2002; 32 (11): 564–576.
9. Ulrich BT, Buerhaus PI, Donelan K, Norman L, Dittus R. Magnet status and registered nurse views of the work environment and nursing as a career. J Nurs Adm. 2007; 37 (5): 212–220.
11. Stuart EA. Matching methods for causal inference: a review and a look forward. Stat Sci. 2010; 25 (1): 1–21.
12. Imai K, Keele L, Tingley D. A general approach to causal mediation analysis. Psychol Methods. 2010; 15 (4): 309–334.
13. Aiken LH, Cimiotti JP, Sloane DM, Smith HL, Flynn L, Neff DF. Effects of nurse staffing and nurse education on patient deaths in hospitals with different nurse work environments. Med Care. 2011; 49 (12): 1047–1053.
15. McHugh MD, Stimpfel AW. Nurse reported quality of care: a measure of hospital quality. Res Nurs Health. 2012; 35: 566–575.
16. Lake ET. Development of the Practice Environment Scale of the Nursing Work Index. Res Nurs Health. 2002; 25 (3): 176–188.
18. Hsieh H, Clement DG, Bazzoli GJ. Impacts of market and organizational characteristics on hospital efficiency and uncompensated care. Health Care Manage Rev. 2010; 35 (1): 77–87.
19. Henke RM, Maeda JL, Marder WD, Friedman BS, Wong HS. Medicare and commercial inpatient resource use: impact of hospital competition. Am J Manage Care. 2013; 19 (6): e28–e248.
20. Park SH, Blegen MA, Spetz J, Chapman SA, De Groot H. Patient turnover and the relationship between nurse staffing and patient outcomes. Res Nurs Health. 2012; 35 (3): 277–288.
21. Ho DE, Imai K, King G, Stuart EA. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Anal. 2007; 15 (3): 199–236.
22. Imai K, King G, Stuart EA. Misunderstandings between experimentalists and observationalists about causal inference. J R Stat Soc A. 2008; 171 (2): 481–502.
23. Ho DE, Imai K, King G, Stuart EA. MatchIt: nonparametric preprocessing for parametric causal inference. J Stat Software. 2011; 42 (8): 1–28.
24. Ten Have TR, Joffe MM. A review of causal estimation of effects in mediation analyses. Stat Methods Med Res. 2012; 21 (1): 77–107.
25. Dunton N, Gajewski B, Klaus S, Pierson B. The relationship of nursing workforce characteristics to patient outcomes. Online J Issues Nurs. 2007; 12 (3). Manuscript 3.
26. Stimpfel AW, Sloane DM, Aiken LH. The longer the shifts for hospital nurses, the higher the levels of burnout and patient dissatisfaction. Health Aff (Millwood). 2012; 31 (11): 2501–2509.
27. Kutney-Lee A, McHugh MD, Sloane DM, et al. Nursing: a key to patient satisfaction. Health Aff (Millwood). 2009; 28 (4): w669–w677.
28. Kramer M, Schmalenberg CE. Best quality patient care: a historical perspective on Magnet hospitals. Nurs Adm Q. 2005; 29 (3): 275–287.
29. Kramer M, Schmalenberg CE. Development and evaluation of essentials of Magnetism tool. J Nurs Adm. 2004; 34 (7/8): 365–378.
30. Lake ET, Staiger D, Horbar J, et al. Association between hospital recognition for nursing excellence and outcomes of very low-birth-weight infants. JAMA. 2012; 307 (16): 1709–1716.
31. Lake ET, Shang J, Klaus S, Dunton NE. Patient falls: association with hospital Magnet status and nursing unit staffing. Res Nurs Health. 2010; 33 (5): 413–425.
32. Goode CJ, Blegen MA, Park SH, Vaughn T, Spetz J. Comparison of patient outcomes in Magnet® and non-Magnet hospitals. J Nurs Adm. 2011; 41 (12): 517–523.
33. Stolzenberger KM. Beyond the Magnet award: the ANCC magnet program as the framework for culture change. J Nurs Adm. 2003; 33 (10): 522–531.
34. Conway JM, Lance CE. What reviewers should expect from authors regarding common method bias in organizational research. J Bus Psychol. 2010; 25: 325–334.
35. Kutney-Lee A, Lake ET, Aiken LH. Development of the hospital nurse surveillance capacity profile. Res Nurs Health. 2009; 32 (2): 217–228.
36. Verran JA, Gerber RM, Milton DA. Data aggregation: criteria for psychometric evaluation. Res Nurs Health. 1995; 18 (1): 77–80.