Young, John Q. MD; Niehaus, Brian MD; Lieu, Sandra C.; O'Sullivan, Patricia S. EdD
Graduate medical education in the United States increasingly emphasizes longitudinal training in ambulatory care settings. This training typically occurs in continuity clinics, where residents are responsible for a panel of patients over an extended period of time. At the end of each academic year, a significant patient care transition occurs in these clinics when the most experienced residents leave because of graduation or advancement to a higher level of training. As a result, patients in the caseloads of departing residents are transferred.
Recent studies have identified challenges to patients' safety and residents' education posed by the academic year-end transfer.1,2 First, the year-end transfer often terminates a long-standing treatment relationship that can be experienced by patients as a significant loss leading to increased symptoms.3 This heightened acuity makes monitoring of higher-risk patients even more critical during the transition. Second, residents stationed in continuity clinics have often accumulated large panels of patients. As a result, the year-end transfer requires unique organizational processes capable of simultaneously transitioning a high volume of patients. Without such procedures in place, there is a much higher probability for patients or important clinical information “falling through the cracks” compared with other types of transfers. Third, patients are transferred from trainees with substantial experience to those with much less (outpatient) experience and skill. This experience gradient can be significant and warrant slower caseload growth, enhanced supervision, and specialized didactics.2
Given these features of the year-end transfer, it is important to develop initial caseloads for incoming residents that optimize learning and patient safety. Our literature search, using the Medical Subject Headings “internship and residency” and “ambulatory care” and variations thereof, found no studies specifically focused on methods for creating outpatient caseloads for continuity clinics. When constructing these initial caseloads for incoming residents, we know anecdotally that many training programs in various specialties largely preserve prior caseloads from outgoing residents, making only minor adjustments. These adjustments typically are limited to balancing the number of patients assigned, but they may also include removing patients deemed “poor learning cases” or redistributing patients deemed “demanding.” This approach has several advantages, including administrative ease and a year-end handoff that requires the incoming resident to receive sign-out from only one or a few outgoing residents. Disadvantages include the possibility that the panel, as inherited by the new trainee, may have become unbalanced during the departing resident's tenure because of that resident's clinical interests, practice style, or referral sources. Although other models may exist for creating initial outpatient caseloads, for the purpose of this study we will consider this model as a traditional one and a current standard practice.
Despite its ease, this traditional approach may generate unbalanced caseloads and have deleterious effects on learning and patient safety. The composition of caseloads can affect learning. Prior research has linked caseload composition and residents' satisfaction. Studies in pediatrics, internal medicine, and family medicine have shown that residents' satisfaction correlates with the diversity of patients in their outpatient caseloads.4–9 Measures of diversity in these studies included patients' age, gender, diagnosis, and the acuity of their illness. Other studies have shown that residents value clinics that offer an appropriate number of patients, smooth patient flow, and substantial continuity of care.7,8,10 Programs have begun to use procedure logs to ensure that trainees obtain sufficient exposure to specified tasks. Although these instruments help each trainee reach minimum thresholds, they do not necessarily affect the overall mix of patients treated. Unbalanced caseloads may meet minimum thresholds but still result in significant experience gaps for trainees. For example, if significant differences exist in the complexity and acuity of caseloads, some residents may not be sufficiently challenged to learn how to manage such cases, whereas others may be too stressed to learn.
Although the association between transfer of patient panel and patient safety and quality of care has not been studied yet in the outpatient setting, research in the inpatient setting does suggest that workload affects trainees' performance. Heavy inpatient workload is correlated with burnout, emotional exhaustion, and poor sleep,11–14 which in turn lead to more errors and less empathy.15–18 These factors may contribute to the higher mortality rate observed for patients admitted to inpatient ward teams with a high census compared with those with a low census.19
Thus, constructing balanced caseloads has the potential to enhance the quality of both training and patient care. Efforts to protect against the negative effects of excessive workload often focus on number of patients per clinician (e.g., “caps”) and work hours (e.g., duty hours restrictions). Yet, other factors, such as acuity and complexity, make important contributions to mental workload. Because a physician's capacity to process information is limited, performance decreases when task demands exceed that capacity, compromising both learning and safety.20–23
Human factors research has focused on how best to measure mental workload. For example, in aviation, where decisions require high mental acuity, workload assessments are necessary to make decisions about crew size, safety, and the effects of cockpit automation.24 Several methods exist for measuring workload, including physiological (e.g., heart rate and skin conductance), procedural (time spent on secondary tasks), and perceptual (e.g., rating scales).20 The National Aeronautics and Space Administration Task Load Index represents a widely accepted perceptual measure of workload that has been applied to numerous disciplines.25,26 For a given task, the tool asks respondents to rate the mental and physical demands, time pressure, effort required, stress, and degree of success. In developing Medicare's Resource-Based Relative Value Scale for reimbursement, Hsiao et al27,28 used qualitative and quantitative methods to define physician mental workload as comprising four similar dimensions: time required, mental effort, technical effort, and stress related to the risk for patient harm. Bertram et al29,30 further developed this concept to measure physicians' mental workloads in the ambulatory internal medicine setting, adding two domains: satisfaction with performance and difficulty/complexity. Subsequent research in ambulatory care settings confirmed that physicians' estimates of work are reliable and valid and correlate with these domains.31,32
The mental workload model captures a broader range of factors that affect learning and patient care than do those models that focus on census or work hours. In this article, we compare the workload-balancing method (WBM), a model adapted from mental workload theory, with the traditional method, using outpatient caseloads for psychiatry residents for purposes of demonstration. Our goal was to reduce intercaseload variability with respect to mental workload in order to promote patients' safety and residents' learning. A related goal was to ensure comparable exposure to selected experiences for each resident.
This descriptive study compares the initial outpatient continuity caseloads generated for incoming trainees by two different models: a mental workload model and the traditional model.
Setting and participants
The intervention took place in a psychiatric hospital based in the Department of Psychiatry at the University of California, San Francisco School of Medicine. The hospital has a paper chart system with an electronic billing and scheduling system. It is affiliated with multiple training programs, including clinical training programs in psychology and in adult, child and adolescent, and geriatric psychiatry. For this study we focused on the construction of caseloads in the medication management clinics, which typically have 400 patients and 16 residents. Psychiatry residents receive in this clinic most of their outpatient, longitudinal training in medication management. Residents enter the clinic at the beginning of their third year and remain for one to two years. This study received approval from the local institutional review board.
Prior to academic year 2006–2007, the residency program constructed initial caseloads in this clinic by reassigning en masse a caseload of a specific outgoing resident to an incoming resident. Minor adjustments ensured that the numbers of patients were comparable between caseloads. Anecdotal complaints by residents suggested that substantial intercaseload differences existed. A subsequent analysis showed that, at the extremes, some residents had more than twice the number of patient visits than did others, even though the numbers of patients in each caseload were roughly similar. In addition, a survey of residents revealed that the number of patients requiring more than 60 minutes of a clinician's time outside of clinic each month ranged from 1 to 16 per caseload. These discrepancies prompted an effort to develop a better method for constructing initial caseloads.
We adapted the NASA Task-Load Index and the work of Hsiao et al27,28 and Bertram et al29,30 to predict the mental workload for a given caseload. Our WBM included five of the six mental workload domains: time, mental effort, technical effort, risk-related stress, and complexity. We did not include the sixth domain—performance (satisfaction associated with task completion)—because we were measuring the experience of care for a patient over an extended period of time and not during a discrete visit or clinic. We also assumed that the workload of a given caseload was a function of the workload of each individual patient. Whereas prior studies have asked clinicians to rate mental workload for a given half-day clinic or for a specific procedure, we asked trainees to rate the workload associated with the overall care of each patient.
Table 1 illustrates the relationship between the five mental workload domains and the variables that our method used. The operational definitions were derived from literature and in-depth discussion with residents and faculty. Studies have shown that mental workload correlates with number of visits,29,32 clinician's experience,30,32 complexity of the patient's pathology,32 quality of care rated by the clinician29,30,32 and supervisor,30 acuity, collaboration demands, and amount of administrative tasks.32 A number of possible indicators of mental workload were considered, including acuity, complexity of patient pathology, amount of paperwork associated with a given patient, and numbers of patients, visits, time demands outside of clinic, cancellations, and no-shows. After several iterative brainstorming sessions with third-year residents, chief residents, and faculty, we concluded that time, mental and technical effort, risk-related stress, and difficulty were best captured by acuity, complexity, time demands outside of clinic, number of patients, number of visits, no-show and cancellation frequency, treatment refractoriness, comorbidity burden, and amount of collaboration required with other clinicians.
To define acuity, we considered a number of options, including assigning points for predefined factors that contribute to acuity. For simplicity, we decided to have outgoing residents categorize each patient in a binary fashion—acute or not acute (Table 1). Balancing acuity was thought to be especially important for educational reasons and to avoid harm to patients by residents overwhelmed with too many acute patients. For complexity and time demands outside of clinic, two principal measures were considered: the number of minutes of a clinician's time required outside of clinic, and scores on a five-point scale anchored to patient complexity and time required outside of clinic. Both were tested with residents. On the basis of residents' feedback during pretesting of the instrument, we concluded that it was difficult to accurately recall minutes per patient per month and that the five-point scale allowed residents to better assess their perceptions of time demands and complexity. In addition to number of patients, we chose the number of predicted visits per month (based on the prior year's utilization) to further capture the volume of work. Finally, we defined “amount of required collaboration” as a binary attribute: whether or not a patient had a psychotherapist in addition to the resident managing medications. This was thought to add workload by requiring regular communication and collaboration and to be an important competence for each resident to obtain. No-shows/cancellations, treatment refractoriness, and comorbidity burden were not included in the final model, because this information was not sufficiently reliable in the electronic billing system. A more elaborate model could include them.
In May, residents who would depart from the clinic on June 30 were asked to rate each of their patients on the mental workload variables of acuity, time demand outside of clinic, and collaboration requirements. These data were manually entered into an Excel spreadsheet linked to the hospital's electronic billing and scheduling software. We calculated each patient's average number of visits per month for the past year or, if care had started within the past year, since care began. The caseload construction spreadsheet was designed to automatically update the mental workload profile for an incoming resident's caseload when a patient was assigned to that resident. This allowed for real-time balancing of the caseloads with regard to these variables. This process required approximately two hours to construct panels for 16 residents. Table 2 is an example of what the mental workload profile would look like for individual residents as caseloads were being constructed. On the Wednesday prior to July 1, didactics were replaced by a three-hour face-to-face sign-out. All outgoing and incoming residents were required to attend. Incoming residents received a list of their new patients (with the names of the referring clinicians and written transfer summaries) and then obtained sign-out on each patient.
Outcome measures and data analysis
For academic years 2006–2007, 2007–2008, 2008–2009, and 2009–2010, we compared the caseloads constructed by the WBM with the caseloads that would have been constructed by the traditional practice of preserving prior residents' caseloads, with adjustments only made to equalize the number of patients per caseload. We reported each year separately to test the robustness of the WBM over several years. For each mental workload variable in Table 1, we calculated the mean and standard deviation for caseloads generated by the WBM versus the traditional method. To normalize the standard deviations of each variable for comparison, the coefficient of variation (CoV), defined as the standard deviation divided by the mean, was calculated for caseloads constructed by both models. In addition, for each of the four academic years, we averaged the CoV for the four variables to produce a summative representation of the WBM's overall performance compared with the traditional method.
Results for each year are presented in Table 3. For each mental workload factor measured, the WBM generated caseloads with much less intercaseload variation. When the CoV is averaged across the four mental workload factors, the WBM generated initial caseloads with 56%, 50%, 61%, and 58% less overall variation for the four successive academic years. To illustrate the real-world impact of reduced variation on residents, Table 4 portrays how the WBM and traditional models distributed acute patients for 15 residents in 2006–2007. In the WBM, the number of acute patients per caseload ranged from four to six (CoV = 0.15) compared with a range of zero to eight in the traditional method (CoV = 0.40). This reduced variation meant that the percentage of acute patients per caseload ranged from 22% to 33% rather than 0% to 44%.
No prior study has addressed how to construct incoming residents' initial outpatient continuity clinic caseloads that are balanced in the domains that affect mental workload and, therefore, learning and quality of care. Compared with the method traditionally used in our (and, as we know anecdotally, other) clinics, the WBM resulted in substantially less intercaseload variation with respect to number of acute patients, complexity/time demands outside of clinic, number of visits per month, and collaboration demands. We present the results for each year separately as proof that the WBM performed consistently better across all study years than did the traditional model.
The impact of workload balancing may be substantial. For example, the traditional method resulted in certain caseloads having a much higher proportion of acute patients and patients with higher complexity/time demands outside of clinic. These significantly more acute and complex caseloads may overtax residents, making them more prone to error and less able to learn. And because these initial caseloads are often constructed for interns or residents with little outpatient experience, the risks are amplified.1,2 In contrast, the WBM reduces variation between caseloads. It much more effectively balances those factors that contribute to mental workload, especially with regard to acuity, risk-related stress, complexity, and various sources of time demand. In so doing, this method safeguards against harm to patients and impaired learning due to cognitive overload. It also accomplishes important educational objectives by ensuring that each resident is exposed to acute, complex, and time-demanding patients, and collaborative care.
Despite the annual rebalancing, the traditional method generated similar intercaseload variation in each of the four years (CoA range = 0.26–0.30; Table 3). Thus, the traditional method did not perform better, even when the balanced method had been used to construct the initial caseloads at the beginning of the previous year. This finding suggests that caseloads, at least in our system, develop differently and diverge over the course of a single academic year. This divergence likely stems from the fact that trainees acquire many new patients during the academic year through specialized assessment clinics that draw from markedly different patient populations. This amplifies intercaseload variation because each trainee rotates through a different combination of assessment clinics.
Several potentially relevant factors are not measured in our model, including some related to complexity (comorbidity burden and treatment refractoriness), risk-related stress and time (no-show and cancellation rates). We initially wanted to incorporate comorbidity and no-show/cancellation rates, but our electronic billing systems did not at the time have reliable information for these variables. Other variables, such as treatment refractoriness and stress associated with a patient, could have been measured by asking outgoing residents to rate each patient in these domains. However, to limit the work required by outgoing clinicians and the associated data entry, we decided to defer the inclusion of these variables. Incorporating them in the future may lead to a model even more refined than our current WBM.
This method does have limitations. For many of the variables, the model assumes that outgoing residents can make accurate estimates. A number of studies indicate that physicians' estimates of work are reliable and valid.27–32 In fact, physicians' global estimates of work are highly correlated with the mental workload dimensions.28 It could be that a single global estimate would suffice. Although the method results in more balanced caseloads for each resident, it is not designed to match the mental workload of a given caseload with a resident's capacity for mental work. Residents may individually differ in the amount of mental workload that optimizes their learning and the quality of care they deliver. An ideal model would match workload with individual capacity. Moreover, physician characteristics also contribute to mental workload.31 For example, residents' styles of how frequently they decide to see their patients and how proactive they are about calling them will impact their experiences of the workload associated with specific patients. Further, patients in our setting could not be moved between clinics, which occur on different days. As a result, caseloads were redistributed among the four residents in each clinic, not among the 16 third-year residents in the program; the method, therefore, did not mitigate variation between clinics. Also, our model, while creating balanced initial caseloads, does not address how caseloads become unbalanced during a given trainee's tenure. Finally, the study itself does not measure the impact of the intervention on those outcomes of most interest, namely, the quality of the learning by residents and the care received by patients.
Although the model was developed and implemented for a psychiatry outpatient continuity clinic, it could easily be adapted to other disciplines. Some factors, such as number of visits and number of patients, are relevant to all disciplines. Other factors, such as acuity, complexity, and time demand outside of clinic, are relevant but may be defined differently depending on the specialty and other contextual factors. Of perhaps greater significance, this model could be expanded to achieve more ambitious goals, such as constructing caseloads that optimize learning in any number of dimensions, by, for example, balancing a caseload's mix of diagnoses. Given how much intercaseload variation develops during a trainee's rotation, this model could also be adapted to address how new patients are assigned to a resident's panel throughout the academic year, rather than just at the outset. Finally, although the patient panels in psychiatry clinics are smaller than those in primary care clinics, the principles used in this model are scalable to patient panels of all sizes.
Further research is required to describe current practices across specialty and programs, verify which factors are most important to balance, and ascertain the impact on learning, quality of care, patient outcomes, and clinicians' productivity and satisfaction. Because these are outcomes of interest to any system of care, this method may have application to how organized systems of care can improve the retention, satisfaction, and performance of physicians.
The authors wish to thank Amy Berlin, MD, Derius Carrol, Cynthia Chappel, Andrea DeRochi, LCSW, Alicia Martinez, Anne Saggio, and Kimberly Terry for their important contributions to the development and implementation of the intervention.
This study was approved by the institutional review board of UCSF as “exempt.”
1 Young JQ, Eisendrath SJ. Enhancing patient safety and resident education during academic year-end transfer of outpatients: Lessons from the suicide of a psychiatric patient. Acad Psychiatry. In Press.
2 Young JQ, Wachter RM. Academic year-end transfers of outpatients from outgoing to incoming residents: An unaddressed patient safety issue. JAMA. 2009;302:1327–1329.
3 Mischoulon D, Rosenbaum JF, Messner E. Transfer to a new psychopharmacologist: Its effect on patients. acad psychiatry. 2000;24:156–163.
4 Blankfield RP, Kelly RB, Alemagno SA, King CM. Continuity of care in a family practice residency program. Impact on physician satisfaction. J Fam Pract. 1990;31:69–73.
5 Dyck RJ, Azim HF. Impact of an outpatient psychiatry rotation on family medicine residents and interns. Can J Psychiatry. 1982;27:644–647.
6 Rubenstein H, Levitt C. Family practice clinics. Survey of family practice residents' attitudes. Can Fam Physician. 1993;39:1895–1902.
7 Schultz KW, Kirby J, Delva D, et al. Medical students' and residents' preferred site characteristics and preceptor behaviours for learning in the ambulatory setting: A cross-sectional survey. BMC Med Educ. 2004;4:12.
8 Sisson SD, Boonyasai R, Baker-Genaw K, Silverstein J. Continuity clinic satisfaction and valuation in residency training. J Gen Intern Med. 2007;22:1704–1710.
9 Haney EM, Nicolaidis C, Hunter A, Chan BK, Cooney TG, Bowen JL. Relationship between resident workload and self-perceived learning on inpatient medicine wards: A longitudinal study. BMC Med Educ. 2006;6:35.
10 Lofgren RP, Mladenovic J. How reorganizing a general medicine clinic affected residents' and patients' satisfaction. Acad Med. 1990;65:604–608.
11 Sargent MC, Sotile W, Sotile MO, Rubash H, Barrack RL. Stress and coping among orthopaedic surgery residents and faculty. J Bone Joint Surg Am. 2004;86A:1579–1586.
12 Stucky ER, Dresselhaus TR, Dollarhide A, et al. Intern to attending: Assessing stress among physicians. Acad Med. 2009;84:251–257.
13 Martini S, Arfken CL, Balon R. Comparison of burnout among medical residents before and after the implementation of work hours limits. Acad Psychiatry. 2006;30:352–355.
14 Gopal R, Glasheen JJ, Miyoshi TJ, Prochazka AV. Burnout and internal medicine resident work-hour restrictions. Arch Intern Med. 2005;165:2595–2600.
15 Shanafelt TD, Bradley KA, Wipf JE, Back AL. Burnout and self-reported patient care in an internal medicine residency program. Ann Intern Med. 2002;136:358–367.
16 Landrigan CP, Rothschild JM, Cronin JW, et al. Effect of reducing interns' work hours on serious medical errors in intensive care units. N Engl J Med. 2004;351:1838–1848.
17 Feddock CA, Hoellein AR, Wilson JF, Caudill TS, Griffith CH. Do pressure and fatigue influence resident job performance? Med Teach. 2007;29:495–497.
18 West CP, Huschka MM, Novotny PJ, et al. Association of perceived medical errors with resident distress and empathy: A prospective longitudinal study. JAMA. 2006;296:1071–1078.
19 Ong M, Bostrom A, Vidyarthi A, McCulloch C, Auerbach A. House staff team workload and organization effects on patient outcomes in an academic general internal medicine inpatient service. Arch Intern Med. 2007;167:47–52.
20 Weinger MB, Herndon OW, Zornow MH, Paulus MP, Gaba DM, Dallen LT. An objective methodology for task analysis and workload assessment in anesthesia providers. Anesthesiology. 1994;80:77–92.
21 Weinger MB, Herndon OW, Gaba DM. The effect of electronic record keeping and transesophageal echocardiography on task distribution, workload, and vigilance during cardiac anesthesia. Anesthesiology. 1997;87:144–155.
22 Weinger MB, Englund CE. Ergonomic and human factors affecting anesthetic vigilance and monitoring performance in the operating room environment. Anesthesiology. 1990;73:995–1021.
23 Sweller J. Cognitive load during problem-solving—Effects on learning. Cogn Sci. 1988;12:257–285.
24 Wiener EL, Nagel DC. Human Factors in Aviation. San Diego, Calif: Academic Press; 1988.
25 Becker AB, Warm JS, Dember WN, Hancock PA. Effects of jet engine noise and performance feedback on perceived workload in a monitoring task. Int J Aviat Psychol. 1995;5:49–62.
26 Young G, Zavelina L, Hooper V. Assessment of workload using NASA Task Load Index in perianesthesia nursing. J Perianesth Nurs. 2008;23:102–110.
27 Hsiao WC, Yntema DB, Braun P, Dunn D, Spencer C. Measurement and analysis of intraservice work. JAMA. 1988;260:2361–2370.
28 Hsiao WC, Braun P, Yntema D, Becker ER. Estimating physicians' work for a resource-based relative-value scale. N Engl J Med. 1988;319:835–841.
29 Bertram DA, Hershey CO, Opila DA, Quirin O. A measure of physician mental work load in internal medicine ambulatory care clinics. Med Care. 1990;28:458–467.
30 Bertram DA, Opila DA, Brown JL, et al. Measuring physician mental workload: Reliability and validity assessment of a brief instrument. Med Care. 1992;30:95–104.
31 Feldman R, Hillson SD, Wingert TD. Measuring the dimensions of physician work. Med Care. 1994;32:943–957.
32 Orozco P, Garcia E. The influence of workload on the mental state of the primary health care physician. Fam Pract. 1993;10:277–282.