Seasonal Variation in Family Member Perceptions of Physician Competence in the Intensive Care Unit

Findings From One Academic Medical Center

Abstract

Purpose 

Researchers have found mixed results about the risk to patient safety in July, when newly minted physicians enter U.S. hospitals to begin their clinical training, the so-called “July effect.” However, patient and family satisfaction and perception of physician competence during summer months remain unknown.

Method 

The authors conducted a retrospective observational cohort study of 815 family members of adult intensive care unit (ICU) patients who completed the Family Satisfaction with Care in the Intensive Care Unit instrument from eight ICUs at Beth Israel Deaconess Medical Center, Boston, Massachusetts, between April 2008 and June 2011. The association of ICU care in the summer months (July–September) versus other seasons and family perception of physician competence was examined in univariable and multivariable analyses.

Results 

A greater proportion of family members described physicians as competent in summer months as compared with winter months (odds ratio [OR] 1.9; 95% confidence interval [CI] 1.2–3.0; P = .003). After adjustment for patient and proxy demographics, severity of illness, comorbidities, and features of the admission in a multivariable model, seasonal variation of family perception of physician competence persisted (summer versus winter, OR of judging physicians competent 2.4; 95% CI 1.3–4.4; P = .004).

Conclusions 

Seasonal variation exists in family perception of physician competence in the ICU, but opposite to the “July effect.” The reasons for this variation are not well understood. Further research is necessary to explore the role of senior provider involvement, trainee factors, system factors such as handoffs, or other possible contributors.

Each July in the United States, an entirely new cohort of physicians, recently graduated from medical school, enters hospitals and intensive care units (ICUs) to begin their postgraduate clinical training. Both the popular and medical literature have raised concerns about whether the complete and overnight replacement of an experienced cohort of physicians with an inexperienced one might have measurable effects on patient outcomes.^1–4 A review by Young and colleagues⁴ has shown that some of this concern may be appropriate: Four of the six studies deemed to be of high quality showed an increased risk of mortality during July, with the adjusted odds of death ranging from 1.08 to 1.34.

To date, while the literature has focused on patient outcomes such as risks of mortality,^5–12 morbidity,^8,13,14 medication error,^6,12,15–19 and efficiency^5,7,20–23 in July, to our knowledge no studies have examined the family’s experience and perception of any July phenomenon. However, a family of a patient in an ICU may have concerns about the competency and capabilities of junior and inexperienced physicians in the summer months as compared with other times of the year, given the attention in the popular and scientific literature to this topic. Further, families of patients in the ICU themselves grapple with significant morbidity from the experience of watching a loved one’s illness in the ICU,²⁴ something that may be influenced by the level of discord with the primary team²⁵ and may be influenced by concerns about the physicians’ expertise.

We sought to understand seasonal variation in how families of patients in the ICU perceive physician competence. We hypothesized that families of patients cared for in ICUs in the first months of new physicians, the summer months of July, August, and September, when physicians are their most inexperienced, might perceive a difference in competence as compared with families of patients in ICUs at later months of the year.

Method

Design, setting, and patients

This was a cohort study of data previously collected from April 2008 through June 2011 at Beth Israel Deaconess Medical Center, an academic tertiary care center in Boston, Massachusetts. This institution had nine adult, closed ICUs, 77 adult ICU beds, and 649 total licensed beds. Three of these ICUs are medical ICUs; two of them have pulmonary and critical care fellows, and one does not. One medical ICU with a fellow has only second- and third-year residents, whereas the other two ICUs have both interns and residents. The surgical ICUs are staffed by surgery and anesthesia residents, surgery and anesthesia critical care fellows, and attendings, and the cardiac care unit is staffed by residents and interns without fellows. The cardiac surgery ICU is staffed without residents and, as described below, was excluded from analysis.

We included all family members who completed the Family Satisfaction with Care in the Intensive Care Unit (FS-ICU) instrument, described below. Only one FS-ICU could be completed per patient.

Our study was approved by the institutional review board at the Beth Israel Deaconess Medical Center. The requirement of informed consent and authorization for disclosure of protected health information was waived.

Survey instrument

The FS-ICU is a validated 34-item instrument designed to measure satisfaction with a family member’s care in the ICU.²⁶ Detailed descriptions of its derivation and its multicentered, multicultural validation procedures have been published elsewhere.^27–30

The details of the implementation of this instrument at our single institution have been described previously in the literature.²⁸ In brief, as part of a larger quality improvement effort, our institution began administering the FS-ICU in April 2008. Families of survivors who were discharged or transferred from the ICU were administered the survey within 48 hours of discharge. Family members of nonsurvivors were mailed the survey at seven weeks after the patient’s death, with a single reminder letter four weeks later.

Outcomes and covariates

The primary outcome of this analysis was the perceived competence of the physicians in the ICU by family members of patients. This was measured on the basis of a single item of the FS-ICU survey that asks family members, “Skill and competence of ICU doctors: how well the doctors cared for your family member” and includes a five-point Likert response scale ranging from “excellent” to “poor.” Because responses were strongly skewed towards “excellent,” we dichotomized the variable into “competent” when the “excellent” rating was selected or “less than fully competent” when any other rating was chosen.

Other covariates of interest included patient-related factors (demographic characteristics, diagnosis, and comorbidities) and family-related factors (demographic characteristics of the proxy, relationship of the proxy, frequency of interaction between the proxy and patient). We also collected responses to the FS-ICU that have been shown to influence overall satisfaction of care in the ICU in our prior work, including disagreement in the family, living in the same city as the hospital, and living in the same household as the patient.²⁸

We conducted two secondary analyses. First, we examined whether perceptions of competence varied by whether the patient survived. To test this hypothesis, we stratified our analyses by survivorship, extracted from the electronic medical record. Second, we examined whether the presence or absence of interns (first-year trainees) had any effect on competence, and we stratified our results by a binary covariate of presence of interns in the ICU.

Exposure variables

The exposure of interest was care provided during in the summer months (July through September), defined on the basis of when new physicians start their training (July). Other seasons were similarly defined in three-month intervals, with fall as October through December, winter as January through March, and spring as April through June. At our institution, within the eight adult ICUs included for analysis, there were two different styles of physician-in-training coverage: residents and interns or residents alone.

Statistical analyses

We performed univariable analyses using chi-square tests or Fisher exact tests for tests of difference to identify significant difference between patients admitted in the summer and all other patients. Univariable logistic regression was performed to understand all significant drivers of perception of physician competence. We conducted multivariable modeling using logistic regression using all covariates that met significance of 0.10 in univariable analyses and those variables that held clinical validity as potential confounders. Age of patient, age of proxy, and length of stay were transformed to meet the linearity assumption of logistic regression. We used area under the receiver operating characteristic curve to assess discrimination and used Hosmer–Lemeshow test of goodness-of-fit to test calibration and to ensure that the model was not overfit to the data.

Because we hypothesized that seasonal perceptions of competence might be different in families whose loved ones died than in those who survived, we evaluated our models for effect modification by survivorship, although these were considered secondary analyses as the survey method was different between these two groups. Further, as several ICUs at our hospital have interns present, we evaluated whether seasonal variation was more pronounced in those with interns as compared with those without interns among survivors.

Results

The number of family members of survivors who were approached and declined was not recorded; among families of nonsurvivors, 1,282 were identified, 1,112 were approached, and 535 agreed to participate. The differences between responders and nonresponders among the families of nonsurvivors are included in Table 1. Of 815 families originally included, 48 (6%) of families were excluded subsequently because the primary outcome, the measure of “physician competence,” was not completed, and 96 families were excluded as their patients were admitted to the one ICU in our institution that did not have resident coverage.

Table 1: Univariable Comparison Among Families of Nonsurvivors Who Responded to the Family Satisfaction With Care in the Intensive Care Unit Survey in Eight ICUs and Families of Nonsurvivors Who Did Not Respond, From a Study of Perceptions of Physician Competence and Season, Beth Israel Deaconess Medical Center, 2002–2011

All remaining patients and families were examined for any differences between those admitted to the ICUs in the summer as compared with those admitted any other time of the year. In univariable analyses, patients admitted in the summer were noted to be younger (mean age 62 versus 67 years, P = .003) and less likely to die (35% [65 of 185] versus 47% [297 of 630], P = .004). The families of patients admitted in the summer were more likely to have previous experience as a family member of someone in the ICU (68% [123 of 181] versus 59% [372 of 630], P = .03; 4 missing responses) and more likely to be the parent of the patient as compared with any other family relationship (12% [23 of 185] versus 6% [39 of 630], P = .005). Differences are summarized in Table 2. Forty-eight families did not provide data on the primary outcome of physician competence and were not included in subsequent analyses.

Table 2: Differences Between Patients Admitted to Eight ICUs During Summer Months (July–September) and All Other Months, From a Study of Perceptions of Physician Competence and Season, Beth Israel Deaconess Medical Center, 2002–2011

In univariable analyses, families of patients admitted in the summer were more likely to perceive their physicians as competent as compared with those admitted in the winter (odds ratio [OR] = 1.9; 95% confidence interval [CI] 1.2–3.0; P = .009). Table 3 describes a number of unadjusted associations with judging a physician competent. In additional to the seasonal influence on perception of competence, whether or not a patient survived (OR = 0.7; 95% CI 0.5–0.9; P = .02) and being the adult child of the patient (OR = 0.7; 95% CI 0.5–1.0; P = .04) were more likely to be associated with judging a physician less than fully competent, whereas having a family member ventilated (OR = 1.5; 95% CI 1.1–2.1; P = .02) and the overall agreement within the family about care choices (OR = 1.7; 95% CI 1.3–2.4; P < .001) were associated with a greater odds of judging the physician team competent. Overall satisfaction was strongly collinear with finding physicians competent (OR = 11.6; 95% CI 8.0–17.0; P < .0001).

Table 3: Univariable Associations With Family Perception of “Excellent” Competence of the Treating Physicians in the Intensive Care Unit, as Rated on the Family Satisfaction With Care in the Intensive Care Unit Survey, From a Study of Perceptions of Physician Competence and Season, Beth Israel Deaconess Medical Center, 2002–2011

In multivariable modeling, seasonal variation in perceptions of physician competence was adjusted for patient and proxy demographics (age, sex of patient, and age of proxy), severity of illness and comorbidities (using the individual-level case mix,³¹ presence of respiratory or oncologic comorbidities, and overall number of comorbidities), features of the admission (an ICU length of stay a week or more, whether the patient was on pressors or ventilated over his or her stay), overall satisfaction, the specific ICU within the hospital, and whether or not the family had agreement with the care plan and lived in the same city as the hospital. Seasonal variation persisted. Only families with patients admitted during the summer had an increased odds of judging a physician competent as compared with the winter (OR 2.4; 95% CI 1.3–4.4; P = .004); families of patients admitted in the other seasons were not (fall: OR 1.5; 95% CI 0.8–1.7; P = .20; spring: OR 1.4; 95% CI 0.8–2.3; P = .22). The full multivariable model is presented in Supplemental Digital Table 1 at https://links.lww.com/ACADMED/A245.

In secondary analyses, seasonal variation of physician competence was different between survivors and nonsurvivors of the ICU. When considered in multivariable analyses, families of survivors in summer remained at a 2.7-fold greater odds of judging their physicians competent as compared with families in the winter (OR 4.5; 95% CI 1.9–10.6; P = .007), whereas families of nonsurvivors showed no variation in perceptions of competence by season (univariable results by season presented in Figure 1). Among the survivors, the difference in unadjusted risk of judging the physician not fully competent did not vary significantly by presence or absence of interns (Figure 2) in multivariable modeling.

Figure 1: Univariable risk of the family judging the treating physicians as fully competent as measured by the Family Satisfaction with Care in the Intensive Care Unit survey, stratified between survivors and nonsurvivors of eight intensive care units, Beth Israel Deaconess Medical Center, 2002–2011. Error bars represent the 95% confidence interval of each percentage.

Figure 2: Mean unadjusted risks of judging the physicians’ competence “excellent” on the Family Satisfaction with Care in the Intensive Care Unit survey, stratified by presence or absence of interns in the intensive care units and restricted to patients and families of survivors of eight intensive care units, Beth Israel Deaconess Medical Center, 2002–2011. Error bars represent the 95% confidence interval of each percentage. The mean risks were not significantly different by season. Abbreviation: ICU indicates intensive care unit.

Discussion

In this retrospective cohort study, we found that family members of patients in the ICU perceived physician competence to be highest in the summer months and lowest in the winter months, even when adjusting for patient- and family-level factors. In secondary analyses, this seasonal effect was only present for family members of patients who survived. The presence or absence of interns as compared with other trainees did not significantly influence the risk that a family judged the physician team less than fully competent.

Although not decisive, literature on the July effect has raised the specter that patients are at greater risk of death, complications, and longer lengths of stay when trainees are at their most junior.^1,4,9 However, studies limited to more acutely ill patients have not found any mortality difference.^5,32,33 Our study evaluated a different outcome than prior studies examining the July effect by focusing on family perceptions of physician competence rather than patient outcome. Family members are at increased risk of posttraumatic stress disorder, particularly those who function as the primary decision makers for their family members, women, and those with previous diagnoses of depression and anxiety.^34–36 Although many family members reported high satisfaction with care in the ICU both in our study and in previous studies,^28,36 prior research has documented the association between satisfaction and feeling supported during difficult decisions.³⁷ We suggest that there may be a seasonal flux to perceptions of physician competence, which may lead to meaningful variation in family experiences in the ICU. Although families of survivors may have greater confidence in the physicians they speak with in summer months, this appears to drop off in winter months, in spite of the effect of nonsurvivors having greater baseline satisfaction of care noted by Wall and colleagues.³⁸ Whether this leads to meaningful differences in health and psychological outcomes for families is an area for further study.

There are many possible explanations for our findings. Although some authors have suggested that the most senior physicians take vacation during the summer months,¹⁰ informal discussions within our hospital suggest that the most senior physicians assign themselves to work the ICUs during the summer in our hospital. Although certainly not definitive, one explanation for our findings is that senior physicians take a more assertive role in the management of unstable patients in the summer months versus other months. Alternatively, perhaps the seasonality of family satisfaction may be tied to the quality and frequency of handoffs between trainees, although it is not clear in what direction handoffs would affect care—trainees in the summer may put a great deal of time into their handoffs but not be well versed in the necessary and important clinical information, whereas trainees in the winter may be more skilled at succinct handoffs but may cut corners. Further investigation would be necessary to explore the relationship of handoffs to seasonal variations in family members’ perceptions of physician knowledge.

Our study had several strengths. We took an alternative approach to the question of the July effect, considering the family perspective rather than solely taking into account patient outcomes. Further, we took into account both clinical- and family-related data as potential confounders.

Our study did have several limitations. First, although we collected data about the response rate among nonsurvivors’ families and basic demographic information about those patients, we do not know the number of survivors’ families who declined to participate in the overall project. As a result, we are unable to draw conclusions about the directionality of response bias in our results. Further, the different data collection strategies between families of survivors and nonsurvivors differed, which may have led to different degrees of recall bias and may have led us to present these analyses as secondary analyses. However, the timing (delay of survey for families of nonsurvivors) paralleled the approach for development and validation of the FS-ICU instrument.²⁶ Second, given the sample size, we were forced to group subjects’ admissions by season rather than by month. Many new residents are not scheduled to enter the ICU for the first time in July but, rather, later in the summer, so considering differences by season might be more appropriate than differences by month. Third, we presumed that year of training approximated experience with the ICU. Given the vast differences in training between interns and senior residents, this seems a generally valid assumption, but it fails to take into consideration the subtlety of different training programs (e.g., anesthesia residents who rotate through the surgical ICUs who may have different levels of experience with mechanical ventilation). These subtleties of experience could be investigated in future research. Fourth, families and patients were likely to have more similar experiences within a single ICU than between ICUs. Although responses likely cluster by ICU within our hospital, we were not able to pursue formal hierarchical modeling given the small number of clusters. Fifth, there may be unmeasured clinical- or family-level confounding.

We found no evidence to suggest that families perceive a decrease in physician competence during the summer months in the ICUs at our academic medical center. Quite the opposite, we found that families of survivors believed physicians to be the least competent in winter months. Future research should consider the implications of any decreased vigilance in winter months in trainee supervision and the influence of other potential drivers of variation such as seasonal variation in physician strain.