Since the Institute of Medicine reports To Err Is Human 1 and Crossing the Quality Chasm 2 demonstrated the prevalence and impact of medical errors, patient safety and quality improvement have become areas of focus in U.S. health care. Attention from policy makers, payers, and groups such as the Agency for Healthcare Research and Quality (AHRQ), National Patient Safety Agency, and World Health Organization has resulted in numerous safety initiatives at the national and institutional levels. Most of these initiatives have targeted physicians, administrators, and other health care professionals. Less attention has been paid to medical students, a population that plays a crucial role in carrying the patient safety agenda forward.
Despite calls to include longitudinal patient safety and quality improvement instruction in undergraduate medical education,3–5 not all medical schools teach these topics to students. Among those that do, curricula vary in format, content, rigor, and evaluation methods.6 Further, many formal educational efforts focus on the acquisition of safety knowledge. Few tools have been developed to help researchers and educators understand how students’ actual clinical experiences may affect their attitudes about patient safety and their intentions regarding their future safety-related behavior.
There is growing awareness that learners are influenced not only by what is taught in the formal curriculum but also by unspoken, powerful messages and beliefs conveyed through interactions with superiors and educators during their clinical rotations. Work on this “hidden curriculum” has shown that there are frequently stark differences between what medical educators say and what they do (knowingly or unknowingly) when acting as role models for students and, in turn, between what students are explicitly taught and what they actually learn.7–9 The resulting culture in the clinical learning environment may deliver messages that oppose the content of the formal curriculum, professional values, and even practices for safe patient care.
Organizations’ cultural factors—safety culture, teamwork culture, error disclosure culture, and professionalism—are distinct from safety knowledge and are important for understanding aspects of the health care workplace environment that have been linked with patient safety and quality of care. For example, staff members’ perceptions of safety culture and teamwork culture are associated with their perceptions of how well their units are providing safe patient care.10,11 Physicians’ perceptions of error disclosure culture are associated with their intent to disclose a hypothetical error to a patient.12 Students who experience unprofessional behavior may be willing to do things they believe are wrong in order to avoid poor evaluations and to fit in with their clinical teams.13 Taken together, these four cultural factors can predict important patient safety outcomes.
We sought to understand how these cultural factors affect medical students. To do this, we developed the Medical Student Safety Attitudes and Professionalism Survey (MSSAPS) to measure students’ perceptions of these cultural factors in their clinical rotations. In this study, we examined the survey’s reliability and its construct and concurrent validity. We also tested the following two hypotheses:
- Medical students’ perceptions of safety culture, teamwork culture, error disclosure culture, and professionalism will be positively correlated with their perceptions of patient safety on their clinical rotations.
- Medical students’ perceptions of these four cultural factors will be positively correlated with their intended behavior concerning patient safety issues.
Our overall goal was to demonstrate that medical students could provide valuable information about their experiences with respect to the cultural factors we identified, and that this information could have important educational and clinical implications.
Developing the MSSAPS
We examined the literature to identify published survey items that we could use to measure medical students’ perceptions of their clinical rotation experiences with respect to four cultural factors—safety culture, teamwork culture, error disclosure culture, and professionalism—and overall patient safety, as well as students’ intended safety-related behaviors. For safety culture and teamwork culture, we adapted items from the Safety Attitudes Questionnaire (SAQ).11 For error disclosure culture, we adapted items from our recently published survey.12 For professionalism, we adapted items from the SAQ11 and a survey on medical student moral distress in clinical learning environments.14 For patient safety, we adapted the overall patient safety grade from the AHRQ’s Hospital Survey on Patient Safety Culture15 and included five safety behavioral intent items—one adapted from a survey about student safety perceptions16 and four that we created.
In creating the safety behavioral intent items, members of our team—two medical students (J.L., S.W.), two patient safety experts (E.T., D.B.), and one psychometrician (J.E.)—chose commonly accepted safety behaviors, such as participation in quality improvement activities and reporting and encouraging others to report errors and patient safety issues.
We pilot tested the first draft of the survey by e-mailing a link to the survey, which was hosted by SurveyMonkey (www.SurveyMonkey.com), to 38 medical students at two medical schools. We asked these students to (1) complete the draft survey and (2) indicate which items they found to be “unclear, confusing, or difficult to answer.” We received feedback from 14 students and made substantive changes to the instructions and item wording based on their input.
In the MSAPPS as administered in this study, we asked students to think about “all of their rotations” when responding. Thirty-four cultural factor items and five safety behavioral intent items were rated using a five-point Likert scale (ranging from 1 = disagree strongly to 5 = agree strongly, with NA = not applicable). Overall patient safety was measured on a five-point scale (ranging from A = excellent to E = failing) in response to the following item: “What overall grade would you give all of your rotations on patient safety?” We also included items to collect demographic information (age, gender, and anticipated residency specialty) as well as eight additional items not relevant to this study. We reverse-coded negatively worded items during data analysis.
Participants and survey administration
Graduating fourth-year medical students at three U.S. MD-granting medical schools were eligible to participate in this study. All three schools were located in urban areas; one school was state funded and the other two were private. The number of faculty at the schools ranged from approximately 1,000 to more than 9,000. This study was approved by the institutional review board of the University of Texas Health Science Center at Houston.
In February to April 2012, we administered the MSSAPS to consenting students as part of their fourth-year capstone courses focusing on “transition to residency.” The capstone course was mandatory at two sites and voluntary at the third. At each study site, students were introduced to the questionnaire during patient safety sessions included in the course. Students were invited via e-mail to participate in the Web-based survey. The survey took most respondents 10 to 15 minutes to complete, according to SurveyMonkey time stamps (when the survey was first accessed and last modified).
Our hypotheses allowed us to test concurrent validity—that is, whether a predictor is correlated with an outcome at the same point in time. Concurrent validity is important to establish because it shows that a link exists between a predictor (i.e., a cultural factor) and a criterion (i.e., overall patient safety grade or a future behavioral intent item). Additionally, we collected data to determine whether the MSSAPS also had evidence for reliability and construct validity. Because the survey items we adapted from other sources were developed predominantly for physicians and staff, we first conducted confirmatory factor analysis (CFA) to ensure that these items functioned in a similar psychometric manner for medical students. CFA provides evidence for construct validity by indicating whether survey items measured the intended factors (in this study, safety culture, teamwork culture, error disclosure culture, and professionalism). We chose the CFA approach rather than exploratory factor analysis because we had a priori beliefs, based on data with other populations,11,12 about how items would behave statistically (i.e., which items belonged with certain factors).
We determined the adequacy of CFA results on the basis of statistical and practical aspects of model fit.17 We examined the χ2/df ratio, Tucker–Lewis Index (TLI),18 comparative fit index (CFI),19 and root mean square error of approximation (RMSEA).20 The χ2/df ratio should be < 3, TLI and CFI values should be ≥ 0.90, and RMSEA values should be ≤ 0.05.21,22 We then computed an initial CFA model for each cultural factor. On the basis of parameter estimates and examination of survey item content, we modified the initial model into a revised model and then tested the revised model. We also assessed the reliability of each factor with Cronbach’s alpha (should be > 0.70). By conducting these analyses, we were able to have confidence in which survey items would be needed to effectively measure the study’s cultural factors.
We then tested our two hypotheses by examining the associations via Spearman correlations between the cultural factors and the patient safety measures, which were (1) safety behavioral intent (consisting of five separate items about the participant’s intent regarding future behavior) and (2) overall patient safety grade (one item representing the participant’s overall assessment of patient safety observed during clinical rotations).
We also examined percent positive scores for each cultural factor because this is a typical way to report culture scores.11,12 Percent positive scores allow medical educators to see variability within and among schools, allowing them to identify areas of concern. We computed percent positive by adding the numbers of participants who agreed slightly or strongly with the items comprising each factor and then dividing the sum by the total number of participants who answered the items for that factor.
Of the 367 fourth-year students invited to participate in the survey, 228 (62%) responded. Response rates varied by school, as follows: 164/200 (82%) at School A; 49/145 (34%) at School B; and 15/19 (79%) at School C. Of the 188 students who responded to the demographic questions, 96 (51%) were male, and 92 (49%) were female. The majority (94%; n = 177) indicated their age was in the range of 25 to 34 years.
MSSAPS performance characteristics: Evidence of construct validity and reliability
The initial CFA model for safety culture had an adequate CFI value but was lacking on the other indices. The regression weights for the last 2 of the factor’s 10 items were lower than the other items, so we excluded items 9 and 10. We ran a revised model with items 1 through 8 only and obtained much better model fit (Δ χ2 was statistically significant at P < .001) and improved values for χ2/df ratio, RMSEA, CFI, and TLI (Table 1; see Table 2 for items). Cronbach’s alpha for the 8-item factor was 0.89.
The initial CFA model for teamwork culture had poor values for the indices. The regression weights for the last 2 of the factor’s 8 items were lower than for the other items, so we excluded items 7 and 8. The revised model, which included items 1 through 6, had a significantly improved fit: Δ χ2 was statistically significant at P < .001, and we obtained better values for the χ2/df ratio, RMSEA, CFI, and TLI (Table 1; see Table 2 for items). Cronbach’s alpha for the six-item factor was 0.81.
Error disclosure culture.
The initial CFA model results indicated poor values for the χ2/df ratio, RMSEA, CFI, and TLI. The regression weights for the last 2 of the factor’s 6 items were lower than the other 4 items. The content for items 5 and 6—importance of disclosing errors—was also different from that of items 1 through 4, which focused on disclosure behavior. As a result, we tested a revised CFA model with just items 1 through 4. The revised model had a significantly better fit than the initial model, with a statistically significant Δ χ2 (P < .001) as well as values for χ2/df ratio, RMSEA, CFI, and TLI that met expectations (Table 1; see Table 2 for items). Cronbach’s alpha for the 4-item factor was 0.75.
We used a one-factor solution to test the initial CFA model for professionalism. The CFI value (0.90) was the only value to meet expectations, and the regression weights for the last 3 of the 10 professionalism items were lower than those for the other 7. Given this finding, and the determination that the last 3 items focused on participants’ comfort expressing concerns whereas the first 7 focused on observed professional behaviors, we created a revised model that replaced the single professionalism factor with two factors: experiences with professionalism (the first 7 items), and comfort expressing professional concerns (the last 3 items; see Table 2). The df did not change between the initial and revised models, so we could not compute Δ χ2, but as shown in Table 1, the χ2 value for the revised model was reduced by more than half (from 131.5 to 64.7). Additionally, the χ2/df ratio and CFI and TLI values met expectations. Although the RMSEA was > 0.05, it improved from the initial model. Cronbach’s alpha values for experiences with professionalism and comfort expressing professional concerns were 0.87 and 0.84, respectively.
CFA results overview.
In summary, CFA identified five cultural factors comprising 28 items: safety culture (8 items), teamwork culture (6 items), error disclosure culture (4 items), experiences with professionalism (7 items), and comfort expressing professional concerns (3 items). On the basis of the CFA results, we accepted the revised model as depicting each cultural factor (see Table 2).
Hypothesis testing: Medical students’ perceptions of cultural factors and safety
We report the response frequencies for the cultural factor items in Table 2 and for the safety behavioral intent items in Table 3. As described above, we calculated Spearman correlations between the five cultural factors, the five safety behavioral intent items, and overall patient safety grade as observed during clinical rotations. We found that each of the cultural factors was correlated with overall patient safety grade (r = 0.37–0.69, P < .05) and at least one of the safety behavioral intent items (Table 4). Safety culture was the strongest predictor of the four patient-safety-related behavioral intent items.
Percent positive scores for cultural factors
There was substantial variation in percent positive scores across participating institutions, with statistically significant differences in safety and teamwork cultures (Table 5). Safety culture positive perception scores ranged from 28% (95% confidence interval [CI], 13%–43%) to 64% (95% CI, 30%–98%), while teamwork culture percent positive scores ranged from 47% (95% CI, 32%–62%) to 74% (95% CI, 66%–81%). Error disclosure culture percent positive scores were low, ranging from 10% (95% CI, 0%–20%) to 27% (95% CI, 20%–35%). Percent positive scores were also low for experiences with professionalism, ranging from 7% (95% CI, 0%–15%) to 23% (95% CI, 16%–30%). Of note, individual items revealed that among responding students, 159/213 (75%) agreed that a team member had made disparaging remarks about a patient, and 140/213 (66%) agreed that a team member had been disrespectful to someone below himself/herself in the team hierarchy (see Table 2). Percent positive scores were low for comfort expressing professional concerns, with scores ranging from 17% (95% CI, 5%–29%) to 38% (95% CI, 8%–69%). Individual items revealed that at least one-quarter of responding students did not feel comfortable expressing concerns to superiors about patient safety (52/213; 25%) or their own mistreatment (91/213; 43%). Finally, percent positive for overall patient safety grade ranged from 38% (95% CI, 23%–53%) to 79% (95% CI, 54%–100%).
Medical students’ experiences during clinical rotations have an important influence on their attitudes toward patient safety and their future behaviors. Negative experiences with respect to safety culture, teamwork culture, error disclosure culture, and professionalism—including both experiences with professionalism and comfort expressing personal concerns—have the potential to contradict lessons learned during formal patient safety curricula. In this study, we have provided initial evidence for the reliability and validity of the MSSAPS in measuring medical students’ perceptions of the above-mentioned cultural factors as experienced during clinical rotations. Our study demonstrates that medical students’ clinical experiences are associated with their safety attitudes and intended behaviors. On the basis of our findings, we believe the MSSAPS can serve as an important tool for measuring these aspects of clinical experiences and targeting areas that need improvement in student/trainee education and in faculty development to provide trainees with effective role models.22
Our study shows that the MSSAPS is psychometrically sound and provides descriptive data that can be used to promote curricular and cultural change. For example, less than one-third of students from any of the three participating schools had positive perceptions of the error disclosure culture or professional behaviors they witnessed during their clinical rotations. Students reported they did not receive education about error disclosure or see errors routinely disclosed to patients and families; such experiences could lead them to internalize the message that disclosure is not an institutional priority. The low percentage of students with positive perceptions of experiences with professionalism suggests that students witnessed disrespectful behaviors among their team members (residents and attending physicians) and between their teams and their patients. The positive perception scores for comfort expressing professional concerns were also low, suggesting that many students had difficulty speaking up about unprofessional behaviors (e.g., mistreatment of themselves and others) or concerns about patient safety. In comparison, students viewed safety and teamwork cultures more positively, perhaps because of relatively long-standing efforts by hospitals to measure and make improvements in these areas.23–31 Importantly, each of the five cultural factors was correlated with students’ perceptions of overall patient safety for their clinical rotations (overall patient safety grade), and all of the factors except error disclosure culture were correlated with at least three of the five safety behavioral intent items. Further, the school with the lowest overall patient safety grade also had the lowest positive perception scores on the five cultural factors. Overall, these findings provide initial evidence for the validity of the MSSAPS and indicate that perceptions of patient safety may be shaped not only by safety and teamwork cultures but also by experiences with professionalism.32
Our study is the first to simultaneously measure medical student attitudes about teamwork culture, safety culture, error disclosure culture, and professionalism. It builds on prior studies of student experiences with unprofessional behavior and the hidden curriculum’s impact on patient safety33–35 by demonstrating quantitative correlations between students’ lived experiences in the hidden curriculum (represented by the cultural factors), their perceptions of patient safety, and their intentions regarding their future safety-related behaviors. Students frequently report experiencing and assimilating into a culture of unprofessional behaviors36 that likely undermine the explicit teachings of medical school curricula. Despite national attention and focused efforts to tackle unprofessional behav ior, a recent study documented a sobering persistence of medical student mistreatment.37 Given the importance of patient safety improvements, our results linking measures of professionalism (i.e., experiences with professionalism and comfort expressing professional concerns) with patient safety measures may provide additional motivation to change medical culture in light of the emphasis that health care professionals, regulators, payers, and the public have placed on improving patient safety.
The MSSAPS can be used for internal benchmarking purposes. Given the concurrent validity evidence for the instrument, students’ patient safety perceptions should improve as clinical rotation sites make improvements in the areas mentioned above. Educators could also administer the MSSAPS after individual rotations to identify problematic clinical experiences. Organizations could use percent positive scores to determine where to prioritize improvement efforts. For example, on the basis of the scores in this study, all three participating institutions could focus on improvements related to error disclosure culture and professionalism (both experiences with professionalism and comfort expressing concerns; see Table 5). School B could also target safety culture and teamwork culture as areas for improvement. Tracking percent positive scores longitudinally would allow schools to conduct year-to-year comparisons of changes in perceptions. In this way, the MSSAPS could be used at the institution level to identify aspects of clinical rotations and overall culture that may deviate from the mission of training safe, professional physicians.
Additionally, the MSSAPS could be useful for external benchmarking and to help medical schools and teaching hospitals answer the call to create cultures of respect that promote ethical, safe behavior.38 By targeting medical students—who are often sensitive to behavioral modeling—this survey could serve as a much-needed tool for measuring the effectiveness of institutional efforts to address disrespectful, unprofessional behavior.
This study has several limitations. First, it includes only the experiences of graduating fourth-year medical students from three institutions. Additional work across other medical schools and years of study is required to better understand the generalizability of our results. Second, further validity testing should be done, specifically to probe whether student responses (i.e., perceptions) relate to actual clinical behavior, because some students may not have had enough experience to distinguish between professional and unprofessional behaviors or between safe and unsafe acts. Third, the voluntary participation in School C’s capstone course (and therefore in the MSSAPS) may introduce bias. Fourth, there has traditionally been a lack of student exposure and experience with topics like medical error disclosure, which may limit the validity of student responses to corresponding survey items. Finally, additional research is needed to determine to what degree incremental changes in clinical curricula based on MSSAPS results would be reflected in subsequent administrations of the survey.
In conclusion, this study provides initial evidence for the reliability and validity of the MSSAPS, a survey instrument that can be used to determine whether medical students are having clinical experiences in positive patient safety learning environments. Our data suggest that a large proportion of students are exposed during their clinical rotations to behaviors that do not exemplify best practices for safe and professional patient care. However, we believe that by building on our findings that unprofessional behaviors in clinical learning environments are linked to students’ perceptions of patient safety, educators can help create a sense of urgency and encourage more leaders and clinical educators to address both explicit and hidden patient safety curricula—and the institutional cultures that affect them.
Acknowledgments: Dr. Bell’s work in humanism was supported through an Arnold P. Gold Foundation Professorship. The authors acknowledge course directors Katharine K. Treadway, MD, and Margaret Uthman, MD, for facilitating access to their students.
1. Institute of Medicine. Committee on Quality of Health Care in America. To Err Is Human: Building a Safer Health System. 2000 Washington, DC National Academy Press
2. Institute of Medicine. Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. 2001 Washington, DC National Academy Press
3. DesHarnais SI, Nash DB. Reforming way medical students and physicians are taught about quality and safety. Mt Sinai J Med. 2011;78:834–841
4. Berwick DM, Finkelstein JA. Preparing medical students for the continual improvement of health and health care: Abraham Flexner and the new “public interest.” Acad Med. 2010;85:S56–S65
5. Lucian Leape Institute Roundtable on Reforming Medical Education. . Unmet Needs: Teaching Physicians to Provide Safe Patient Care. 2010 Boston, Mass Lucian Leape Institute at the National Patient Safety Foundation
6. Wong BM, Etchells EE, Kuper A, Levinson W, Shojania KG. Teaching quality improvement and patient safety to trainees: A systematic review. Acad Med. 2010;85:1425–1439
7. Chuang AW, Nuthalapaty FS, Casey PM, et al. To the point: Reviews in medical education—taking control of the hidden curriculum. Am J Obstet Gynecol. 2010;203:316.e1–e6
8. Inui TS. A Flag in the Wind: Educating for Professionalism in Medicine. 2003 Washington, DC Association of American Medical Colleges
9. Hafferty FW, Franks R. The hidden curriculum, ethics teaching, and the structure of medical education. Acad Med. 1994;69:861–871
10. Etchegaray JM, Thomas EJ. Comparing two safety culture surveys: Safety Attitudes Questionnaire and Hospital Survey on Patient Safety. BMJ Qual Saf. 2012;21:490–498
11. Sexton JB, Helmreich RL, Neilands TB, et al. The Safety Attitudes Questionnaire: Psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. 2006;6:44
12. Etchegaray JM, Gallagher TH, Bell SK, Dunlap B, Thomas EJ. Error disclosure: A new domain for safety culture assessment. BMJ Qual Saf. 2012;21:594–599
13. Feudtner C, Christakis DA, Christakis NA. Do clinical clerks suffer ethical erosion? Students’ perceptions of their ethical environment and personal development. Acad Med. 1994;69:670–679
14. Wiggleton C, Petrusa E, Loomis K, et al. Medical students’ experiences of moral distress: Development of a Web-based survey. Acad Med. 2010;85:111–117
16. Leung GK, Patil NG. Patient safety in the undergraduate curriculum: Medical students’ perception. Hong Kong Med J. 2010;16:101–105
17. Reise SP, Widaman KF, Pugh RH. Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance. Psychol Bull. 1993;114:552–566
18. Bentler PM, Bonnet DG. Significance tests and goodness of fit in the analysis of covariance structures. Psychol Bull. 1980;88:588–606
19. Bentler PM. Comparative fit indexes in structural models. Psychol Bull. 1990;107:238–246
20. Steiger JH, Lind J. Statistically based tests for the number of common factors. May 1980 Iowa City, Iowa Paper presented at: Annual Meeting of the Psychometric Society
21. Bollen KA. Structural Equations With Latent Variables. 1989 New York, NY Wiley
22. Kline RB. Principles and Practice of Structural Equation Modeling. 2005 New York, NY The Guilford Press
23. Bell SK, Moorman DW, Delbanco T. Improving the patient, family, and clinician experience after harmful events: The “when things go wrong” curriculum. Acad Med. 2010;85:1010–1017
24. Vigorito MC, McNicoll L, Adams L, Sexton B. Improving safety culture results in Rhode Island ICUs: Lessons learned from the development of action-oriented plans. Jt Comm J Qual Patient Saf. 2011;37:509–514
25. Carney BT, West P, Neily JB, Mills PD, Bagian JP. Improving perceptions of teamwork climate with the Veterans Health Administration medical team training program. Am J Med Qual. 2011;26:480–484
26. Sexton JB, Berenholtz SM, Goeschel CA, et al. Assessing and improving safety climate in a large cohort of intensive care units. Crit Care Med. 2011;39:934–939
27. Paine LA, Rosenstein BJ, Sexton JB, Kent P, Holzmueller CG, Pronovost PJ. Assessing and improving safety culture throughout an academic medical centre: A prospective cohort study. Qual Saf Health Care. 2010;19:547–554
28. O’Leary KJ, Wayne DB, Haviley C, Slade ME, Lee J, Williams MV. Improving teamwork: Impact of structured interdisciplinary rounds on a medical teaching unit. J Gen Intern Med. 2010;25:826–832
29. Huang DT, Clermont G, Kong L, et al. Intensive care unit safety culture and outcomes: A U.S. multicenter study. Int J Qual Health Care. 2010;22:151–161
30. Pronovost PJ, Berenholtz SM, Goeschel C, et al. Improving patient safety in intensive care units in Michigan. J Crit Care. 2008;23:207–221
31. Thomas EJ, Sexton JB, Neilands TB, Frankel A, Helmreich RL. The effect of executive walk rounds on nurse safety climate attitudes. A randomized trial of clinical units. BMC Health Serv Res. 2005;5:28
32. Leape LL, Shore MF, Dienstag JL, et al. A culture of respect, part 1: The nature and causes of disrespectful behavior by physicians. Acad Med. 2012;87:1–8
33. Walton MM. Hierarchies: The Berlin Wall of patient safety. Qual Saf Health Care. 2006;15:229–230
34. Brainard AH, Brislen HC. Viewpoint: Learning professionalism: A view from the trenches. Acad Med. 2007;82:1010–1014
35. Lucey C, Souba W. Perspective: The problem with the problem of professionalism. Acad Med. 2010;85:1018–1024
36. Reddy ST, Farnan JM, Yoon JD, et al. Third-year medical students’ participation in and perceptions of unprofessional behaviors. Acad Med. 2007;82(10 suppl):S35–S39
37. Fried JM, Vermillion M, Parker NH, Uijtdehaage S. Eradicating medical student mistreatment: A longitudinal study of one institution’s efforts. Acad Med. 2012;87:1191–1198
© 2014 by the Association of American Medical Colleges
38. Leape LL, Shore MF, Dienstag JL, et al. A culture of respect, part 2: Creating a culture of respect. Acad Med. 2012;87:1–6