Seasonal Variation in Family Member Perceptions of Physician Competence in the Intensive Care Unit: Findings From One Academic Medical Center : Academic Medicine

Secondary Logo

Journal Logo

Research Reports

Seasonal Variation in Family Member Perceptions of Physician Competence in the Intensive Care Unit

Findings From One Academic Medical Center

Stevens, Jennifer P. MD, MS; Kachniarz, Bart; O’Reilly, Kristin RN, MPH; Howell, Michael D. MD, MPH

Author Information
Academic Medicine 90(4):p 472-478, April 2015. | DOI: 10.1097/ACM.0000000000000553


Each July in the United States, an entirely new cohort of physicians, recently graduated from medical school, enters hospitals and intensive care units (ICUs) to begin their postgraduate clinical training. Both the popular and medical literature have raised concerns about whether the complete and overnight replacement of an experienced cohort of physicians with an inexperienced one might have measurable effects on patient outcomes.1–4 A review by Young and colleagues4 has shown that some of this concern may be appropriate: Four of the six studies deemed to be of high quality showed an increased risk of mortality during July, with the adjusted odds of death ranging from 1.08 to 1.34.

To date, while the literature has focused on patient outcomes such as risks of mortality,5–12 morbidity,8,13,14 medication error,6,12,15–19 and efficiency5,7,20–23 in July, to our knowledge no studies have examined the family’s experience and perception of any July phenomenon. However, a family of a patient in an ICU may have concerns about the competency and capabilities of junior and inexperienced physicians in the summer months as compared with other times of the year, given the attention in the popular and scientific literature to this topic. Further, families of patients in the ICU themselves grapple with significant morbidity from the experience of watching a loved one’s illness in the ICU,24 something that may be influenced by the level of discord with the primary team25 and may be influenced by concerns about the physicians’ expertise.

We sought to understand seasonal variation in how families of patients in the ICU perceive physician competence. We hypothesized that families of patients cared for in ICUs in the first months of new physicians, the summer months of July, August, and September, when physicians are their most inexperienced, might perceive a difference in competence as compared with families of patients in ICUs at later months of the year.


Design, setting, and patients

This was a cohort study of data previously collected from April 2008 through June 2011 at Beth Israel Deaconess Medical Center, an academic tertiary care center in Boston, Massachusetts. This institution had nine adult, closed ICUs, 77 adult ICU beds, and 649 total licensed beds. Three of these ICUs are medical ICUs; two of them have pulmonary and critical care fellows, and one does not. One medical ICU with a fellow has only second- and third-year residents, whereas the other two ICUs have both interns and residents. The surgical ICUs are staffed by surgery and anesthesia residents, surgery and anesthesia critical care fellows, and attendings, and the cardiac care unit is staffed by residents and interns without fellows. The cardiac surgery ICU is staffed without residents and, as described below, was excluded from analysis.

We included all family members who completed the Family Satisfaction with Care in the Intensive Care Unit (FS-ICU) instrument, described below. Only one FS-ICU could be completed per patient.

Our study was approved by the institutional review board at the Beth Israel Deaconess Medical Center. The requirement of informed consent and authorization for disclosure of protected health information was waived.

Survey instrument

The FS-ICU is a validated 34-item instrument designed to measure satisfaction with a family member’s care in the ICU.26 Detailed descriptions of its derivation and its multicentered, multicultural validation procedures have been published elsewhere.27–30

The details of the implementation of this instrument at our single institution have been described previously in the literature.28 In brief, as part of a larger quality improvement effort, our institution began administering the FS-ICU in April 2008. Families of survivors who were discharged or transferred from the ICU were administered the survey within 48 hours of discharge. Family members of nonsurvivors were mailed the survey at seven weeks after the patient’s death, with a single reminder letter four weeks later.

Outcomes and covariates

The primary outcome of this analysis was the perceived competence of the physicians in the ICU by family members of patients. This was measured on the basis of a single item of the FS-ICU survey that asks family members, “Skill and competence of ICU doctors: how well the doctors cared for your family member” and includes a five-point Likert response scale ranging from “excellent” to “poor.” Because responses were strongly skewed towards “excellent,” we dichotomized the variable into “competent” when the “excellent” rating was selected or “less than fully competent” when any other rating was chosen.

Other covariates of interest included patient-related factors (demographic characteristics, diagnosis, and comorbidities) and family-related factors (demographic characteristics of the proxy, relationship of the proxy, frequency of interaction between the proxy and patient). We also collected responses to the FS-ICU that have been shown to influence overall satisfaction of care in the ICU in our prior work, including disagreement in the family, living in the same city as the hospital, and living in the same household as the patient.28

We conducted two secondary analyses. First, we examined whether perceptions of competence varied by whether the patient survived. To test this hypothesis, we stratified our analyses by survivorship, extracted from the electronic medical record. Second, we examined whether the presence or absence of interns (first-year trainees) had any effect on competence, and we stratified our results by a binary covariate of presence of interns in the ICU.

Exposure variables

The exposure of interest was care provided during in the summer months (July through September), defined on the basis of when new physicians start their training (July). Other seasons were similarly defined in three-month intervals, with fall as October through December, winter as January through March, and spring as April through June. At our institution, within the eight adult ICUs included for analysis, there were two different styles of physician-in-training coverage: residents and interns or residents alone.

Statistical analyses

We performed univariable analyses using chi-square tests or Fisher exact tests for tests of difference to identify significant difference between patients admitted in the summer and all other patients. Univariable logistic regression was performed to understand all significant drivers of perception of physician competence. We conducted multivariable modeling using logistic regression using all covariates that met significance of 0.10 in univariable analyses and those variables that held clinical validity as potential confounders. Age of patient, age of proxy, and length of stay were transformed to meet the linearity assumption of logistic regression. We used area under the receiver operating characteristic curve to assess discrimination and used Hosmer–Lemeshow test of goodness-of-fit to test calibration and to ensure that the model was not overfit to the data.

Because we hypothesized that seasonal perceptions of competence might be different in families whose loved ones died than in those who survived, we evaluated our models for effect modification by survivorship, although these were considered secondary analyses as the survey method was different between these two groups. Further, as several ICUs at our hospital have interns present, we evaluated whether seasonal variation was more pronounced in those with interns as compared with those without interns among survivors.


The number of family members of survivors who were approached and declined was not recorded; among families of nonsurvivors, 1,282 were identified, 1,112 were approached, and 535 agreed to participate. The differences between responders and nonresponders among the families of nonsurvivors are included in Table 1. Of 815 families originally included, 48 (6%) of families were excluded subsequently because the primary outcome, the measure of “physician competence,” was not completed, and 96 families were excluded as their patients were admitted to the one ICU in our institution that did not have resident coverage.

Table 1:
Univariable Comparison Among Families of Nonsurvivors Who Responded to the Family Satisfaction With Care in the Intensive Care Unit Survey in Eight ICUs and Families of Nonsurvivors Who Did Not Respond, From a Study of Perceptions of Physician Competence and Season, Beth Israel Deaconess Medical Center, 2002–2011

All remaining patients and families were examined for any differences between those admitted to the ICUs in the summer as compared with those admitted any other time of the year. In univariable analyses, patients admitted in the summer were noted to be younger (mean age 62 versus 67 years, P = .003) and less likely to die (35% [65 of 185] versus 47% [297 of 630], P = .004). The families of patients admitted in the summer were more likely to have previous experience as a family member of someone in the ICU (68% [123 of 181] versus 59% [372 of 630], P = .03; 4 missing responses) and more likely to be the parent of the patient as compared with any other family relationship (12% [23 of 185] versus 6% [39 of 630], P = .005). Differences are summarized in Table 2. Forty-eight families did not provide data on the primary outcome of physician competence and were not included in subsequent analyses.

Table 2:
Differences Between Patients Admitted to Eight ICUs During Summer Months (July–September) and All Other Months, From a Study of Perceptions of Physician Competence and Season, Beth Israel Deaconess Medical Center, 2002–2011

In univariable analyses, families of patients admitted in the summer were more likely to perceive their physicians as competent as compared with those admitted in the winter (odds ratio [OR] = 1.9; 95% confidence interval [CI] 1.2–3.0; P = .009). Table 3 describes a number of unadjusted associations with judging a physician competent. In additional to the seasonal influence on perception of competence, whether or not a patient survived (OR = 0.7; 95% CI 0.5–0.9; P = .02) and being the adult child of the patient (OR = 0.7; 95% CI 0.5–1.0; P = .04) were more likely to be associated with judging a physician less than fully competent, whereas having a family member ventilated (OR = 1.5; 95% CI 1.1–2.1; P = .02) and the overall agreement within the family about care choices (OR = 1.7; 95% CI 1.3–2.4; P < .001) were associated with a greater odds of judging the physician team competent. Overall satisfaction was strongly collinear with finding physicians competent (OR = 11.6; 95% CI 8.0–17.0; P < .0001).

Table 3:
Univariable Associations With Family Perception of “Excellent” Competence of the Treating Physicians in the Intensive Care Unit, as Rated on the Family Satisfaction With Care in the Intensive Care Unit Survey, From a Study of Perceptions of Physician Competence and Season, Beth Israel Deaconess Medical Center, 2002–2011

In multivariable modeling, seasonal variation in perceptions of physician competence was adjusted for patient and proxy demographics (age, sex of patient, and age of proxy), severity of illness and comorbidities (using the individual-level case mix,31 presence of respiratory or oncologic comorbidities, and overall number of comorbidities), features of the admission (an ICU length of stay a week or more, whether the patient was on pressors or ventilated over his or her stay), overall satisfaction, the specific ICU within the hospital, and whether or not the family had agreement with the care plan and lived in the same city as the hospital. Seasonal variation persisted. Only families with patients admitted during the summer had an increased odds of judging a physician competent as compared with the winter (OR 2.4; 95% CI 1.3–4.4; P = .004); families of patients admitted in the other seasons were not (fall: OR 1.5; 95% CI 0.8–1.7; P = .20; spring: OR 1.4; 95% CI 0.8–2.3; P = .22). The full multivariable model is presented in Supplemental Digital Table 1 at

In secondary analyses, seasonal variation of physician competence was different between survivors and nonsurvivors of the ICU. When considered in multivariable analyses, families of survivors in summer remained at a 2.7-fold greater odds of judging their physicians competent as compared with families in the winter (OR 4.5; 95% CI 1.9–10.6; P = .007), whereas families of nonsurvivors showed no variation in perceptions of competence by season (univariable results by season presented in Figure 1). Among the survivors, the difference in unadjusted risk of judging the physician not fully competent did not vary significantly by presence or absence of interns (Figure 2) in multivariable modeling.

Figure 1:
Univariable risk of the family judging the treating physicians as fully competent as measured by the Family Satisfaction with Care in the Intensive Care Unit survey, stratified between survivors and nonsurvivors of eight intensive care units, Beth Israel Deaconess Medical Center, 2002–2011. Error bars represent the 95% confidence interval of each percentage.
Figure 2:
Mean unadjusted risks of judging the physicians’ competence “excellent” on the Family Satisfaction with Care in the Intensive Care Unit survey, stratified by presence or absence of interns in the intensive care units and restricted to patients and families of survivors of eight intensive care units, Beth Israel Deaconess Medical Center, 2002–2011. Error bars represent the 95% confidence interval of each percentage. The mean risks were not significantly different by season. Abbreviation: ICU indicates intensive care unit.


In this retrospective cohort study, we found that family members of patients in the ICU perceived physician competence to be highest in the summer months and lowest in the winter months, even when adjusting for patient- and family-level factors. In secondary analyses, this seasonal effect was only present for family members of patients who survived. The presence or absence of interns as compared with other trainees did not significantly influence the risk that a family judged the physician team less than fully competent.

Although not decisive, literature on the July effect has raised the specter that patients are at greater risk of death, complications, and longer lengths of stay when trainees are at their most junior.1,4,9 However, studies limited to more acutely ill patients have not found any mortality difference.5,32,33 Our study evaluated a different outcome than prior studies examining the July effect by focusing on family perceptions of physician competence rather than patient outcome. Family members are at increased risk of posttraumatic stress disorder, particularly those who function as the primary decision makers for their family members, women, and those with previous diagnoses of depression and anxiety.34–36 Although many family members reported high satisfaction with care in the ICU both in our study and in previous studies,28,36 prior research has documented the association between satisfaction and feeling supported during difficult decisions.37 We suggest that there may be a seasonal flux to perceptions of physician competence, which may lead to meaningful variation in family experiences in the ICU. Although families of survivors may have greater confidence in the physicians they speak with in summer months, this appears to drop off in winter months, in spite of the effect of nonsurvivors having greater baseline satisfaction of care noted by Wall and colleagues.38 Whether this leads to meaningful differences in health and psychological outcomes for families is an area for further study.

There are many possible explanations for our findings. Although some authors have suggested that the most senior physicians take vacation during the summer months,10 informal discussions within our hospital suggest that the most senior physicians assign themselves to work the ICUs during the summer in our hospital. Although certainly not definitive, one explanation for our findings is that senior physicians take a more assertive role in the management of unstable patients in the summer months versus other months. Alternatively, perhaps the seasonality of family satisfaction may be tied to the quality and frequency of handoffs between trainees, although it is not clear in what direction handoffs would affect care—trainees in the summer may put a great deal of time into their handoffs but not be well versed in the necessary and important clinical information, whereas trainees in the winter may be more skilled at succinct handoffs but may cut corners. Further investigation would be necessary to explore the relationship of handoffs to seasonal variations in family members’ perceptions of physician knowledge.

Our study had several strengths. We took an alternative approach to the question of the July effect, considering the family perspective rather than solely taking into account patient outcomes. Further, we took into account both clinical- and family-related data as potential confounders.

Our study did have several limitations. First, although we collected data about the response rate among nonsurvivors’ families and basic demographic information about those patients, we do not know the number of survivors’ families who declined to participate in the overall project. As a result, we are unable to draw conclusions about the directionality of response bias in our results. Further, the different data collection strategies between families of survivors and nonsurvivors differed, which may have led to different degrees of recall bias and may have led us to present these analyses as secondary analyses. However, the timing (delay of survey for families of nonsurvivors) paralleled the approach for development and validation of the FS-ICU instrument.26 Second, given the sample size, we were forced to group subjects’ admissions by season rather than by month. Many new residents are not scheduled to enter the ICU for the first time in July but, rather, later in the summer, so considering differences by season might be more appropriate than differences by month. Third, we presumed that year of training approximated experience with the ICU. Given the vast differences in training between interns and senior residents, this seems a generally valid assumption, but it fails to take into consideration the subtlety of different training programs (e.g., anesthesia residents who rotate through the surgical ICUs who may have different levels of experience with mechanical ventilation). These subtleties of experience could be investigated in future research. Fourth, families and patients were likely to have more similar experiences within a single ICU than between ICUs. Although responses likely cluster by ICU within our hospital, we were not able to pursue formal hierarchical modeling given the small number of clusters. Fifth, there may be unmeasured clinical- or family-level confounding.

We found no evidence to suggest that families perceive a decrease in physician competence during the summer months in the ICUs at our academic medical center. Quite the opposite, we found that families of survivors believed physicians to be the least competent in winter months. Future research should consider the implications of any decreased vigilance in winter months in trainee supervision and the influence of other potential drivers of variation such as seasonal variation in physician strain.


1. Cox L. The “July effect”: Worst month for fatal hospital errors, study finds. ABC News. June 3, 2010 Accessed September 15, 2014
2. Friedman RA. Their coats are white but their hands are green. New York Times. June 29, 2004 Accessed September 15, 2014
3. Aylin P, Majeed FA. The killing season—fact or fiction? BMJ. 1994;309:1690
4. Young JQ, Ranji SR, Wachter RM, Lee CM, Niehaus B, Auerbach AD. “July effect”: Impact of the academic year-end changeover on patient outcomes: A systematic review. Ann Intern Med. 2011;155:309–315
5. Barry WA, Rosenthal GE. Is there a July phenomenon? The effect of July admission on intensive care mortality and length of stay in teaching hospitals. J Gen Intern Med. 2003;18:639–645
6. Chow KM, Szeto CC, Chan MH, Lui SF. Near-miss errors in laboratory blood test requests by interns. QJM. 2005;98:753–756
7. Rich EC, Gifford G, Luxenberg M, Dowd B. The relationship of house staff experience to the cost and quality of inpatient care. JAMA. 1990;263:953–957
8. Anderson KL, Koval KJ, Spratt KF. Hip fracture outcome: Is there a “July effect”? Am J Orthop (Belle Mead NJ). 2009;38:606–611
9. Englesbe MJ, Fan Z, Baser O, Birkmeyer JD. Mortality in Medicare patients undergoing surgery in July in teaching hospitals. Ann Surg. 2009;249:871–876
10. Shuhaiber JH, Goldsmith K, Nashef SA. Impact of cardiothoracic resident turnover on mortality after cardiac surgery: A dynamic human factor. Ann Thorac Surg. 2008;86:123–130
11. Smith ER, Butler WE, Barker FG 2nd. Is there a “July phenomenon” in pediatric neurosurgery at teaching hospitals? J Neurosurg. 2006;105(3 suppl):169–176
12. Phillips DP, Barker GE. A July spike in fatal medication errors: A possible effect of new medical residents. J Gen Intern Med. 2010;25:774–779
13. Haller G, Myles PS, Taffé P, Perneger TV, Wu CL. Rate of undesirable events at beginning of academic year: Retrospective cohort study. BMJ. 2009;339:b3974
14. Ayas NT, Norena M, Wong H, Chittock D, Dodek PM. Pneumothorax after insertion of central venous catheters in the intensive care unit: Association with month of year and week of month. Qual Saf Health Care. 2007;16:252–255
15. Kestle JR, Cochrane DD, Drake JM. Shunt insertion in the summer: Is it safe? J Neurosurg. 2006;105(3 suppl):165–168
16. Garcia S, Canoniero M, Young L. The effect of July admission in the process of care of patients with acute cardiovascular conditions. South Med J. 2009;102:602–607
17. Hartley KL, Gedde SJ, Venkatraman AS, Feuer WJ, Ajuria-Londono L. Is it safe to have an ophthalmic emergency in July? Ophthalmic Surg Lasers Imaging. 2007;38:358–364
18. Walling HW, Veremakis C. Ordering errors by first-year residents: Evidence of learning from mistakes. Mo Med. 2004;101:128–131
19. Borenstein SH, Choi M, Gerstle JT, Langer JC. Errors and adverse outcomes on a surgical service: What is the role of residents? J Surg Res. 2004;122:162–166
20. Bakaeen FG, Huh J, LeMaire SA, et al. The July effect: Impact of the beginning of the academic cycle on cardiac surgical outcomes in a cohort of 70,616 patients. Ann Thorac Surg. 2009;88:70–75
21. Dhaliwal AS, Chu D, Deswal A, et al. The July effect and cardiac surgery: The effect of the beginning of the academic cycle on outcomes. Am J Surg. 2008;196:720–725
22. Englesbe MJ, Pelletier SJ, Magee JC, et al. Seasonal variation in surgical outcomes as measured by the American College of Surgeons–National Surgical Quality Improvement Program (ACS-NSQIP). Ann Surg. 2007;246:456–462
23. Rich EC, Hillson SD, Dowd B, Morris N. Specialty differences in the “July phenomenon” for Twin Cities teaching hospitals. Med Care. 1993;31:73–83
24. Paparrigopoulos T, Melissaki A, Efthymiou A, et al. Short-term psychological impact on family members of intensive care unit patients. J Psychosom Res. 2006;61:719–722
25. Abbott KH, Sago JG, Breen CM, Abernethy AP, Tulsky JA. Families looking back: One year after discussion of withdrawal or withholding of life-sustaining support. Crit Care Med. 2001;29:197–201
26. Wall RJ, Engelberg RA, Downey L, Heyland DK, Curtis JR. Refinement, scoring, and validation of the Family Satisfaction in the Intensive Care Unit (FS-ICU) survey. Crit Care Med. 2007;35:271–279
27. Stricker KH, Niemann S, Bugnon S, Wurz J, Rohrer O, Rothen HU. Family satisfaction in the intensive care unit: Cross-cultural adaptation of a questionnaire. J Crit Care. 2007;22:204–211
28. Hunziker S, McHugh W, Sarnoff-Lee B, et al. Predictors and correlates of dissatisfaction with intensive care. Crit Care Med. 2012;40:1554–1561
29. Heyland DK, Tranmer JEKingston General Hospital ICU Research Working Group. . Measuring family satisfaction with care in the intensive care unit: The development of a questionnaire and preliminary results. J Crit Care. 2001;16:142–149
30. Heyland DK, Rocker GM, Dodek PM, et al. Family satisfaction with care in the intensive care unit: Results of a multiple center study. Crit Care Med. 2002;30:1413–1418
31. Howell MD, Ngo L, Folcarelli P, et al. Sustained effectiveness of a primary-team-based rapid response system. Crit Care Med. 2012;40:2562–2568
32. Finkielman JD, Morales IJ, Peters SG, et al. Mortality rate and length of stay of patients admitted to the intensive care unit in July. Crit Care Med. 2004;32:1161–1165
33. Claridge JA, Schulman AM, Sawyer RG, Ghezel-Ayagh A, Young JS. The “July phenomenon” and the care of the severely injured patient: Fact or fiction? Surgery. 2001;130:346–353
34. Azoulay E, Pochard F, Kentish-Barnes N, et al.FAMIREA Study Group. Risk of post-traumatic stress symptoms in family members of intensive care unit patients. Am J Respir Crit Care Med. 2005;171:987–994
35. Gries CJ, Engelberg RA, Kross EK, et al. Predictors of symptoms of posttraumatic stress and depression in family members after patient death in the ICU. Chest. 2010;137:280–287
36. Heyland DK, Cook DJ, Rocker GM, et al. Decision-making in the ICU: Perspectives of the substitute decision-maker. Intensive Care Med. 2003;29:75–82
37. Lautrette A, Darmon M, Megarbane B, et al. A communication strategy and brochure for relatives of patients dying in the ICU. N Engl J Med. 2007;356:469–478
38. Wall RJ, Curtis JR, Cooke CR, Engelberg RA. Family satisfaction in the ICU: Differences between families of survivors and nonsurvivors. Chest. 2007;132:1425–1433

Supplemental Digital Content

© 2015 by the Association of American Medical Colleges