Dyrbye, Liselotte N. MD, MHPE; Schwartz, Alan PhD; Downing, Steven M. PhD; Szydlo, Daniel W.; Sloan, Jeff A. PhD; Shanafelt, Tait D. MD
Psychological distress is pervasive among medical students.1–4 Previous studies have documented that as many as one-quarter of medical students are depressed1 and that half experience symptoms of burnout.5,6 The majority report quality of life (QOL) substantially below that of the age-matched general population,6 as well as profound degrees of stress7 and substantial daytime fatigue.8 Psychological distress can adversely affect competency and professionalism9–14 and may contribute to illicit drug use,15 marital discord,16 poor physical health/self-care,17 suicidal ideation,18 and serious thoughts of dropping out of medical school.19 The strong dose–response relationship between burnout and mental QOL with future suicidal ideation18 and serious thoughts of dropping out of medical school19 suggest that students with the greatest degree of distress are at greatest risk for taking action that has serious consequences.
Few distressed medical students seek help on their own initiative.20,21 This creates a challenge for administrators as they try to support students and often results in students' distress going unrecognized and untreated. As distress during medical school predicts postgraduate mental health problems, the cycle of distress can continue unabated.3 Once in practice, physicians' distress can affect the quality of care they provide.22,23
One response to these challenges is to screen medical students for distress, which offers the potential benefit of prompt identification of at-risk students with greater possibility for early intervention. Although such screening has been advocated,9,24 efforts have been hampered by lack of an appropriate screening instrument. Existing instruments (e.g., Center for Epidemiologic Studies Depression Scale, Maslach Burnout Inventory [MBI], Perceived Stress Scale [PSS], Epworth Sleepiness Scale [ESS]) are long and typically measure only one form of distress (e.g., depression, burnout, fatigue, stress). Because distress can present in a variety of ways,25 using only one of these tools would fail to identify many distressed students.
Given the lack of a practical, suitable screening instrument, we recently developed a brief assessment tool designed to identify the subset of medical students with severe distress who warrant individual attention. We have previously reported the methods used to develop the Medical Student Well-Being Index (MSWBI) and its basic psychometric properties.26 In the present study, we evaluate the efficacy of the MSWBI to identify medical students with severe distress.
The primary cohort for this study includes the 2,248/4,287 medical students (52.4%) from seven U.S. medical schools who responded to a survey we conducted in 2007.18 (The participating medical schools were the Mayo Medical School, Uniformed Services University of the Health Sciences, University of Alabama School of Medicine, University of California, San Diego, School of Medicine, University of Chicago Pritzker School of Medicine, University of Minnesota Medical School, and University of Washington School of Medicine.) We repeated all analyses using data from a separate confirmatory cohort consisting of the 2,682/4,400 medical students (61%) at these institutions who completed the survey in 2009.27 In both 2007 and 2009, participation was elective and anonymous, and each school's institutional review board approved the study.
The 2007 and 2009 surveys included questions about demographic characteristics as well as suicidal ideation (i.e., “During the past 12 months have you had thoughts of taking your own life?”) and serious thoughts of dropping out of medical school (i.e., “During the past 12 months have you had any thoughts of dropping out of medical school?” and, if so, “How seriously did you consider dropping out of medical school?”). The surveys also incorporated the MSWBI and standardized instruments to screen for common dimensions of distress.
The 2007 and 2009 surveys included the MBI,28–34 the Primary Care Evaluation of Mental Disorders (PRIME-MD),35,36 and the eight-question Medical Outcomes Study Short-Form Health Survey (SF-8)37,38 to measure burnout, symptoms of depression, and QOL, respectively (in aggregate, 32 items). The SF-8 relies on age- and gender-norm-based scoring methods to calculate mental and physical QOL summary scores.37,38 The 2007 survey also included the Epworth Sleepiness Scale (ESS)8,39 and the Perceived Stress Scale (PSS)40–42 (an additional 18 items). We selected these instruments on the basis of their psychometric properties8,28–42 and their extensive use in previous samples of medical students or residents.4–8,22,43–46
Medical Student Well-Being Index
We have previously reported the methods used to develop the seven-item MSWBI and our preliminary evaluation of its psychometric properties.26 Briefly, we identified the domains of burnout, depression, stress, fatigue, and mental and physical QOL for inclusion; generated items; and developed the MSWBI using a process of literature review, expert input, review, and feedback by medical school deans and medical students, and correlation analysis from previously administered assessments. An independent group of experts judged the items to be clear, relevant, and representative, with acceptable interrater reliability. The final MSWBI questions are shown in Table 1. All questions are answered using a simple yes/no format. One point is assigned for each “yes”; summary scores on the seven-item index range from 0 to 7 (lowest to highest risk for severe distress).
Proxy measures for students in severe distress
Because distress among medical students has multiple dimensions25 and there is no single gold standard for defining severe distress, we evaluated the MSWBI's performance in multiple ways. First, we evaluated the MSWBI's ability to identify students with low mental QOL, which we defined as having an SF-8 mental component score ≥1/2 standard deviation (SD) below the age- and gender-matched population norm.37 We selected this outcome as the primary dependent variable because it allowed benchmarking against population data and evaluates a clinically meaningful effect size.47
Second, we evaluated the MSWBI's ability to identify students who, within the previous 12 months, had suicidal ideation (those who responded “yes”) or had serious thoughts of dropping out of medical school (defined as those who reported that they had somewhat seriously, seriously, very seriously, or extremely seriously considered dropping out of medical school) as secondary/alternative measures of distress because they represent clinically relevant outcomes that warrant individualized counseling.
We used basic descriptive statistics to summarize sample demographics. We compared subpopulations using Fisher exact test or Wilcoxon/two-sample t test procedures, as appropriate. We carried out all hypothesis testing using a 5% type I error rate and a two-sided alternative.
We assessed the efficacy of the MSWBI by calculating the sensitivity, specificity, and likelihood ratios (LRs) associated with exact and threshold MSWBI scores. We constructed receiver operating characteristic (ROC) curves to demonstrate the relationship between sensitivity and false-positive rate at each MSWBI threshold score for each outcome of interest.
We used multiple confirmatory multivariable methods to test the MSWBI's stability as a screening instrument for severe distress. First, we performed forward and backward stepwise logistic regression to identify which MSWBI item(s) best detects the medical student with low mental QOL, suicidal ideation, or serious thoughts of dropping out. We evaluated the fitness of our model using 500 randomly generated bootstrapping samples to conduct forward and backward stepwise regression. We used generalized estimating equations to control for clustering by school. Next, using the 2007 data only, we built an exploratory multivariable model to determine whether adding to the MSWBI one or more of the 50 items from the MBI, PRIME-MD, SF-8, ESS, or PSS would improve the MSWBI's ability to detect students with low mental QOL. Lastly, we used a hypothetical cohort to explore the practicality of using the MSWBI at various cut points. We conducted all analyses using SAS version 9 (SAS Institute, Cary, North Carolina).
We have previously reported the demographic characteristics, prevalence of burnout and depressive symptoms, and QOL of the 2,248/4,287 medical students (52.4%) who responded to the survey in 2007 (the primary cohort for this study).18 Briefly, among respondents, 51.8% (1,159/2,236) were men, 54.9% (1,229/2,237) were 25 to 30 years old, 54.3% (1,217/2,241) were single, and 11% (266/2,248) had children. Respondents were less likely than the surveyed population to be male (51.8% versus 55.0% [2,355/4,285]), 25 to 30 years old (54.9% versus 58.2% [2,470/4,242]), and non-Caucasian (26.2% [579/2,213] versus 28.6% [1,184/4,139]; all P < .04).
Nearly half of the respondents (49.6%, 1,069/2,154) had burnout, and almost as many (46.5%, 1,037/2,228) screened positive for depressive symptoms. The mean SF-8 mental component score was 41.82 for women and 45.01 for men. Among the respondents, 41.3% (899/2,178) had low mental QOL, 11.2% (249/2,230) had suicidal ideation within the previous 12 months, and 10.9% (243/2,222) had serious thoughts of dropping out of medical school within the previous 12 months.
Ability to detect low mental QOL
Table 1 shows the MSWBI items endorsed by the students in the 2007 cohort. Students with low mental QOL were more likely to endorse each of the seven individual MSWBI items (all P < .001) and a greater total number of items than were students without low mental QOL (mean: 3.7 versus 1.5; P < .0001). As Figure 1 illustrates, SF-8 mental component scores decreased as the number of MSWBI items endorsed increased (P < .0001 for both genders), suggesting that the MSWBI can stratify students' mental QOL. At MSWBI scores of ≥3 for men or ≥4 for women, mean SF-8 mental component scores were ≥1/2 SD lower than those of the age- and gender-matched population (45.01 for men and 40.325 for women).37 With increasing number of MSWBI items endorsed, the odds of low mental QOL increased; for example, students who endorsed four items had a 3.81 (95% confidence interval 2.94–4.93) increased odds of low mental QOL.
Table 2 shows the sensitivity and specificity of different threshold scores for identifying students with low mental QOL. The threshold score provides a way to estimate the risk of distress in a group of students scoring at or above a given level (e.g., score is ≥4) and is most useful for setting a cutoff score to identify a subset of students for further evaluation. At a threshold score of ≥4, the specificity of the MSWBI for detecting students with low mental QOL was 87.7%, and the sensitivity was 59.2%. The LR for low mental QOL among students with a score <4 would be 0.47 as compared with 4.79 for those with a score ≥4. Figure 2 illustrates the sensitivity (true positive rate) versus 1 − specificity (false-positive rate) for having low mental QOL based on the threshold score; the area under the ROC curve is 0.833 for low mental QOL.
Table 3 shows the efficacy of the MSWBI using exact scores, which provides a way to determine an individual student's risk of distress. The LR of low mental QOL ranges from 0.09 to 18.49 for exact scores as they increase from 0 to 7. Using a 40% prevalence of low mental QOL (i.e., approximately the level of distress in the current sample and in other studies1,6) as the pretest probability, the exact score can lower the probability of low mental QOL to 5.7% for a student who scores 0 on the MSWBI but raise it to 92.5% for a student who scores 7 on the MSWBI.
There was virtually identical sensitivity and specificity with overlapping confidence intervals for identifying students with low mental QOL when we limited analysis to only first- and second-year students or only third- and fourth-year students (data not shown).
Ability to detect suicidal ideation or serious thoughts of dropping out of medical school
Students with suicidal ideation or serious thoughts of dropping out were more likely to endorse each of the seven MSWBI items and to endorse a greater total number of items than were those students without suicidal ideation or serious thoughts of dropping out (all P < .0001). We also found a dose–response relationship between increasing number of MSWBI items endorsed and the odds of suicidal ideation (odds ratio [OR] 0.08–14.82) or of serious thoughts of dropping out (OR 0.04–11.34). As Figure 2 shows, the area under the ROC curve is 0.762 for suicidal ideation and 0.791 for serious thoughts of dropping out. LRs and posttest probabilities, using a prevalence of 10% for recent suicidal ideation and 10% for serious thoughts of dropping out1,6,9,20,24,48 as the pretest probability, for exact MSWBI scores are shown in Table 3. The posttest probability of suicidal ideation increases from 1.1% to 61.4%, and the posttest probability of serious thoughts of dropping out increases from 0.6% to 55.0% as the exact MSWBI score increases from 0 to 7.
Next, we explored the prevalence of suicidal ideation and serious thoughts of dropping out among students who had MSWBI scores ≥4 but did not have low mental QOL—students who would be considered a “false-positive” based on their SF-8 mental component scores alone. These students, compared with students who had MSWBI scores <4, were nearly five times more likely to have suicidal ideation (42/183 [23.0%] versus 78/1,533 [5.1%]; P < .0001) and were nearly three times more likely to have serious thoughts of dropping out (27/183 [11.5%] versus 65/1,533 [4.2%]; P < .0001), indicating that they were indeed experiencing severe distress despite their SF-8 mental component scores. In such a scenario, when only students with MSWBI scores ≥4 who did not have low mental QOL or suicidal ideation were considered “false-positives,” the sensitivity of the MSWBI using a threshold ≥4 was 93.2%, whereas the specificity was 91.1%. Similarly, when only students with MSWBI scores ≥4 who did not have low mental QOL or serious thoughts of dropping out were considered “false-positives,” the sensitivity and specificity for the threshold ≥4 were 89.6% and 90.4%, respectively.
To confirm these findings, we repeated the analyses on a separate sample of 2,682 medical students who responded to the survey in 2009. Using this sample, we found that SF-8 mental component scores decreased as the number of MSWBI items endorsed increased (P < .0001 for both genders), further supporting our finding that the MSWBI can stratify students' mental QOL (see Supplemental Digital Figure 1, http://links.lww.com/ACADMED/A52). The MSWBI again stratified risk for all three outcomes with <10% differences in sensitivity and specificity at all score levels compared with the results we obtained using the 2007 dataset. For example, in the 2009 dataset, at a threshold score of ≥4 the sensitivity and specificity, respectively, were 68.61% and 80.65% for detecting low mental QOL, 74.62% and 63.67% for suicidal ideation, and 78.33% and 57.78% for serious thoughts of dropping out.
Multivariable logistic modeling
To explore whether the seven-item instrument could be shortened, we performed logistic regression to determine if each of the seven items in the MSWBI was an independent predictor of low mental QOL. Four of the seven MSWBI items (numbers 1, 3, 5, and 6; see Table 1) were independently associated with students having low mental QOL. This persisted after we controlled for clustering by school. The four-item model correctly predicted 83.6% of the cases in terms of low mental QOL. In our bootstrap analysis, these four MSWBI items remained in the multivariable logistic regression model 97% to 100% of the time. Next, using the 2007 dataset we conducted an exploratory multivariable model to evaluate whether adding any of the 50 items from the MBI, SF-8, PRIME-MD, ESS, and PSS would improve the seven-item MSWBI's ability to detect students with low mental QOL. Our analysis suggested that 14 of the 50 items were independent predictors of low mental QOL. In aggregate, however, these 14 items accounted for only 8.2% of the concordance (e.g., increased the accuracy of the model predicting low mental QOL from 83.6% to 91.8%) at a cost of tripling the length of the MSWBI.
Our similar analysis to determine association with our secondary/alternative measure of distress found that six of the seven MSWBI items were independently associated with suicidal ideation (numbers 2, 3, 4, 5, 6, and 7) or serious thoughts of dropping out (numbers 1, 2, 3, 4, 5, and 7). Both models correctly predicted students' status over 75% of the time.
The backward regression provided results similar to those of all of the forward stepwise models. In aggregate, each of the seven MSWBI items was independently associated with low mental QOL, suicidal ideation, or serious thoughts of dropping out, suggesting that the instrument could not be shortened.
Use of the MSWBI to screen a hypothetical cohort of students at an average-sized medical school
Finally, we modeled the outcome of screening a hypothetical cohort of medical students at an average-sized medical school (130 students per class, with total enrollment of 520) using the MSWBI, with the assumption that individualized follow-up would be provided to all students scoring ≥4. We assumed the prevalence of low mental QOL to be 40% (208/520) on the basis of previous publications.1 In this hypothetical cohort, 162/520 (31.2%) students would screen positive on the MSWBI (score ≥4); of these, 123/162 (75.9%) would have low mental QOL, and up to an estimated 86% would have low mental QOL, suicidal ideation, or serious thoughts of dropping out of medical school. The prevalence of a false-negative score (score <4 in students with low mental QOL, suicidal ideation, or serious thoughts of dropping out) would be an estimated 5% to 7%.
Medical students' varied manifestations of psychological distress (e.g., burnout, depression, low mental QOL, stress, fatigue), their reluctance to seek help, and the potential serious consequences of their distress underscore the need for a practical brief screening instrument that evaluates multiple dimensions of distress simultaneously and identifies the students in greatest need of individualized attention. To the best of our knowledge, this is the first large, multicenter study that provides validity data on such an instrument.
Among two large cohorts of medical students, the MSWBI identified students at risk for three clinically relevant outcomes (low mental QOL, suicidal ideation, or serious thoughts of dropping out of medical school) that warrant recognition and individualized attention. At a threshold score of ≥4, the MSWBI's specificity for detecting students in severe distress in the 2007 cohort ranged from 87.7% to 90.4%–91.0%, and its sensitivity ranged from 59.2% to 89.6%–93.2% (both the latter percentage ranges were derived from defining “false-positives” as not having low mental QOL or suicidal ideation/serious thoughts of dropping out—a reasonable approach because having suicidal ideation or serious thoughts of dropping out are both outcomes that warrant identification and response regardless of a student's SF-8 mental component score). Thus, at a threshold score of ≥4, the MSWBI's specificity is comparable to that of other established and widely used screening instruments (e.g., PRIME-MD,35,36 Patient Health Questionnaire [PHQ-9],49 Mood Disorder Questionnaire,50 Generalized Anxiety Disorder Scale51,52).
In the multivariable logistic regression, four of the seven MSWBI items remained independently associated with low mental QOL, six of seven with suicidal ideation, and six of seven with serious thoughts of dropping out. Each of the seven items was an independent predictor of at least one of these three outcomes, suggesting that the instrument cannot be shortened. Our findings speak to the heterogeneity of distress among medical students and support the concept that assessing only one domain is not adequate when screening for distress. In contrast, adding the 14 items from the standardized instruments that were independently associated with mental QOL tripled the length of the MSWBI but captured minimal additional ability to accurately predict distress. Collectively, our results suggest that screening for severe distress using a brief screening tool such as the MSWBI could facilitate rapid identification of students who need individualized interventions.
The MSWBI could be used by schools or by students themselves. Schools could use the MSWBI to estimate a group of students' or an individual student's risk of being in severe distress. For example, assuming a 40% prevalence of low mental QOL (approximately the level of distress in the 2007 and 2009 samples and other studies1,6), the exact score on the MSWBI can lower the probability of low mental QOL to 5.7% (MSWBI score of 0) or raise it to 92.5% (MSWBI score of 7). For suicidal ideation and serious thoughts of dropping out of medical school (using a prevalence of 10% for both outcomes1,6,9,20,24,48), the exact score can lower the probability to 1.1% (score of 0) or less or raise it as high as 61.4% (score of 7). Such information can guide administrators' decision making about which students need assistance beyond that offered by a generic wellness program and help them allocate limited resources. Students also have a role in monitoring and identifying their level of distress and seeking help when appropriate. The use of brief survey tools such as the MSWBI may help students promote their own wellness or recognize when they need help. Such skills are important to maintaining resilience throughout a career in medicine.
This study has several limitations. First, medical student distress is a multidimensional construct, and no gold standard exists for measuring it. We choose to examine three dimensions of distress—clinically meaningful changes in SF-8 mental component score, suicidal ideation, and serious thoughts of dropping out—as indicators of clinically relevant outcomes that warrant individualized attention. Although the MSWBI seems to be valuable for stratifying risk for these important variables, it may not stratify risk for all dimensions of distress. Nonetheless, the fact that the mean SF-8 mental component score incrementally decreased with each one-point increase in MSWBI score suggests that the MSWBI is a powerful risk stratification tool. Second, future studies are needed to determine the MSWBI's predictive validity and explore associations between MSWBI score and other important outcomes. Third, although we surveyed large samples of medical students from diverse private and public medical schools to obtain representative samples of U.S. students, the generalizability of these results to other samples of students is unknown—particularly to students outside the United States.
In summary, the results from this study suggest that the MSWBI is likely to be a helpful tool for schools interested in estimating the risk that either an individual student or group of students is in severe distress. Such information can guide decision making about which students need assistance beyond that offered to all students through a generic wellness program and help schools allocate limited resources. Further study is needed to explore how best to engage students in the screening process and whether screening with the MSWBI will lead to earlier intervention, less suffering, and fewer adverse consequences.
This work was supported by a professionalism grant from the Mayo Clinic Program in Professionalism and Bioethics and from intramural funds from the Mayo Clinic Department of Medicine, Division of Primary Care. Dr. Dyrbye receives salary support from the Mayo Clinic, Department of Medicine Program on Physician Well-Being and the Mayo Clinic College of Medicine Office of Medical Education. The sponsor had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.
Institutional review board approval was obtained from the Mayo Medical School, Uniformed Services University of the Health Sciences, University of Alabama School of Medicine, University of California, San Diego, School of Medicine, University of Chicago Pritzker School of Medicine, University of Minnesota Medical School, and University of Washington School of Medicine prior to surveying their students in 2007 and 2009. The University of Illinois at Chicago College of Medicine institutional review board approved the remainder of the study pertaining to instrument development and validation.
The abstract of an earlier version of this article was presented at the 2008 BMA-AMA-CMA International Conference on Doctors' Health: Doctors' Health Matters—Finding the Balance, London, England, November 2008, and at the University of Illinois at Chicago College of Medicine 10th Annual Master of Health Professions Education Summer Conference, Chicago, Illinois, July 2009.
2 Dahlin M, Joneborg N, Runeson B. Stress and depression among medical students: A cross-sectional study. Med Educ. 2005;39:594–604.
3 Tyssen R, Vaglum P, Gronvold NT, Ekeberg O. Factors in medical school that predict postgraduate mental health problems in need of treatment. A nationwide and longitudinal study. Med Educ. 2001;35:110–120.
4 Thomas NK. Resident burnout. JAMA. 2004;292:2880–2889.
5 Dyrbye LN, Thomas MR, Harper W, et al. The association of race with medical student well-being across the United States. Arch Intern Med. 2007;167:2103–2109.
7 Rohe DE, Barrier PA, Clark MM, Cook DA, Vickers KS, Decker PA. The benefits of pass–fail grading on stress, mood, and group cohesion in medical students. Mayo Clin Proc. 2006;81:1443–1448.
8 Johns MW. Reliability and factor analysis of the Epworth Sleepiness Scale. Sleep Med. 1992;15:376–381.
9 Clark DC, Zeldow PB. Vicissitudes of depressed mood during four years of medical school. JAMA. 1988;260:2521–2528.
10 Stewart SM, Lam TH, Betson CL, Wong CM, Wong AM. A prospective analysis of stress and academic performance in the first two years of medical school. Med Educ. 1999;33:243–250.
11 Thomas MR, Dyrbye LN, Huntington JL, et al. How do distress and well-being relate to medical student empathy? A multicenter study. J Gen Intern Med. 2007;22:177–183.
12 Hojat M, Mangione S, Nasca T, et al. An empirical study of decline in empathy in medical school. Med Educ. 2004;38:934–941.
13 Crandall SJ, Volk RJ, Loemker V. Medical students' attitudes toward providing care for the underserved. Are we training socially responsible physicians? JAMA. 1993;269:2519–2523.
14 Woloschuk W, Harasym PH, Temple W. Attitude change during medical school: A cohort study. Med Educ. 2004;38:522–534.
15 Baldwin DC Jr, Hughes PH, Conard SE, Storr CL, Sheehan DV. Substance use among senior medical students. A survey of 23 medical schools. JAMA. 1991;265:2074–2078.
16 Colford JM Jr, McPhee SJ. The ravelled sleeve of care. Managing the stresses of residency training. JAMA. 1989;261:889–893.
18 Dyrbye LN, Thomas MR, Massie FS, et al. Burnout and suicidal ideation among U.S. medical students. Ann Intern Med. 2008;149:334.
21 Tjia J, Givens JL, Shea JA. Factors associated with undertreatment of medical student depression. J Am Coll Health. 2005;53:219–224.
22 West WP, Huschka MM, Novotny PJ, et al. Association of perceived medical errors with resident distress and empathy: A prospective longitudinal study. JAMA. 2006;296:1071–1078.
23 Shanafelt TD, Balch CM, Dyrbye LN, Bechamps G, Russell T, Satele D, Rummans T, Swartz K, Novotny PJ, Sloan J, Oreskovich MR. Suicidal ideation among American surgeons. Arch Surg. 2011;146:64–67.
25 Dyrbye LN, Harper W, Durning S, et al. Patterns of distress in U.S. medical students. Med Teach. In press.
26 Dyrbye LN, Szydlo DW, Downing SM, Sloan JA, Shanafelt TD. Development and preliminary psychometric properties of a well-being index for medical students. BMC Med Educ. January 27, 2010;10:8.
27 Dyrbye LN, Massie FS Jr, Eacker A, et al. Relationship between burnout and professional conduct and attitudes among U.S. medical students. JAMA. 2010;304:1173–1180.
28 Rafferty JP, Lemkau JP, Purdy RR, Rudisill JR. Validity of the Maslach Burnout Inventory for family practice physicians. J Clin Psychol. 1986;42:488–492.
29 Lee RT, Ashforth BE. A meta-analytic examination of the correlates of the three dimensions of job burnout. J Appl Psychol. 1996;81:123–133.
30 Leiter M, Durup J. The discriminant validity of burnout and depression: A confirmatory factor analytic study. Anxiety Stress Coping. 1994;7:357–373.
31 Maslach C, Jackson SE, Leiter MP. Maslach Burnout Inventory Manual. 3rd ed. Palo Alto, Calif: Consulting Psychologists Press; 1996.
32 Arthur NM. The assessment of burnout: A review of three inventories useful for research and counseling. J Couns Dev. 1990;69:186–189.
33 Soderfeldt M, Soderfeldt B, Warg LE, Ohlson CG. The factor structure of the Maslach Burnout Inventory in two Swedish human service organizations. Scand J Psychol. 1996;37:437–443.
34 Toppinen-Tanner S, Ojajarvi A, Vaananen A, Kalimo R, Jappinen P. Burnout as a predictor of medically certified sick-leave absences and their diagnosed causes. Behav Med. 2005;31:18–27.
35 Spitzer RL, Williams JB, Kroenke K, et al. Utility of a new procedure for diagnosing mental disorders in primary care. The PRIME-MD 1000 study. JAMA. 1994;272:1749–1756.
36 Whooley MA, Avins AL, Miranda J, Browner WS. Case-finding instruments for depression. Two questions are as good as many. J Gen Intern Med. 1997;12:439–445.
37 Ware J, Kosinski M, Dewey J, Gandek B. How to Score and Interpret Single-Item Health Status Measures: A Manual for Users of the SF-8 Health Survey. Lincoln, RI: QualityMetric Incorporated; 2001.
38 Turner-Bowker DM, Bayliss MS, Ware JE Jr, Kosinski M. Usefulness of the SF-8 Health Survey for comparing the impact of migraine and other conditions. Qual Life Res. 2003;12:1003–1012.
39 Johns MW. A new method for measuring daytime sleepiness: The Epworth Sleepiness Scale. Sleep. 1991;14:540–545.
40 Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. 1983;24:385–396.
41 Cohen S, Williamson GM. Perceived stress in a probability sample of the United States. In: Spacapan S, Oskamp S, eds. The Social Psychology of Health. Newbury Park, Calif: Sage; 1988.
42 Katz J, Monnier J, Libet J, Shaw D, Beach S. Individual and crossover effect of stress on adjustment in medical student marriages. J Marital Fam Ther. 2000;26:341–351.
43 Guthrie E, Black D, Bagalkote H, Shaw C, Campbell M, Creed F. Psychological stress and burnout in medical students: A five-year prospective longitudinal study. J R Soc Med. 1998;91:237–243.
44 Shanafelt T, West C, Zhao X, et al. Relationship between increased personal well-being and enhanced empathy among internal medicine residents. J Gen Intern Med. 2005;20:559–564.
45 Margolis SA, Reed RL. Effect of religious practices of Ramadan on sleep and perceived sleepiness of medical students. Teach Learn Med. 2004;16:145–149.
46 Bellini LM, Baime M, Shea JA. Variation of mood and empathy during internship. JAMA. 2002;287:3143–3146.
47 Norman GR, Sloan JA, Wyrwich KW. The truly remarkable universality of half a standard deviation: Confirmation through another look. Expert Rev Pharmacoecon Outcomes Res. 2004;4:515–519.
48 Tyssen R, Vaglum P, Gronvold NT, Ekeberg O. Suicidal ideation among medical students and young physicians: A nationwide and prospective study of prevalence and predictors. J Affect Disord. 2001;64:69–79.
49 Kroenke K, Spitzer RL, Williams JB. The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med. 2001;16:606–613.
50 Hirschfeld RM, Williams JB, Spitzer RL, et al. Development and validation of a screening instrument for bipolar spectrum disorder: The Mood Disorder Questionnaire. Am J Psychiatry. 2000;157:1873–1875.
51 Kroenke K, Spitzer RL, Williams JB, Monahan PO, Löwe B. Anxiety disorders in primary care: Prevalence, impairment, comorbidity, and detection. Ann Intern Med. 2007;146:317–325.
52 Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: The GAD-7. Arch Intern Med. 2006;166:1092–1097.