Cannon, Grant W. MD; Keitz, Sheri A. MD, PhD; Holland, Gloria J. PhD; Chang, Barbara K. MD, MA; Byrne, John M. DO; Tomolo, Anne MD, MPH; Aron, David C. MD, MS; Wicker, Annie B.; Kashner, T Michael PhD, JD, MPH
Medical education involves a spectrum of learning activities characterized by educational experiences that ideally are organized around the evolving needs of the learner. The progression from undergraduate medical school training through graduate medical education involves graduated levels of medical knowledge, clinical experiences, and responsibilities. The design of effective educational experiences requires an understanding of the factors most associated with trainee satisfaction and the differences in the learning needs of trainees by level of training. Previous work has identified striking similarities in faculty/preceptor characteristics that are valued by different levels of undergraduate and graduate medical trainees, such as enthusiasm, willingness to delegate responsibility, and providing feedback.1 Preceptor interactions were identified as more highly valued by medical students than by medical residents, who, in contrast, believed that the variety of patients and evaluating patients independently were more important. Research has shown that differences in learning resource needs and perceptions of preparedness for residency also varied across the spectrum of medical students and residents.1
Trainee satisfaction has been routinely employed as an outcome measure in evaluating faculty performance and the impact on learners’ training experiences.1–3 Prior work in the area of trainee perceptions has focused on the themes of satisfaction with teaching style and role models, as well as the influence of clinical rotations on choice of specialty training. Teaching styles and practice characteristics in the ambulatory setting have also been emphasized.1,2 Whereas prior work has compared undergraduate and postgraduate trainees in common clinical settings (e.g., the ambulatory clinic), broader comparisons are lacking. The identification of other factors beyond preceptor characteristics that contribute to the satisfaction of trainees working in a variety of clinical settings will facilitate the development of effective educational experiences.
The Department of Veterans Affairs (VA) Learners’ Perceptions Survey (LPS) was developed to measure the satisfaction of all trainees at VA facilities. Since 2001, the LPS has been completed by more than 70,000 VA-based trainees in multiple disciplines, of whom more than 23,000 were residents and medical students. The LPS is a validated instrument that provides a comprehensive evaluation of the perceptions of undergraduate and graduate trainees related to their clinical training. Simultaneously, the LPS identifies key factors associated with the satisfaction of medical trainees who are working in the same training environment, that is, a VA medical facility.3
VA clinical training provides an ideal opportunity to compare factors influencing trainee satisfaction for medical students and physician residents. The education of health care professionals is one of the four congressionally mandated missions of the Veterans Health Administration, yet until the development of LPS there was no clear measurement of trainees’ satisfaction with their educational experiences in the VA. Each year, an estimated 17,000 medical students and more than 30,000 physician residents rotate through 125 VA hospitals affiliated with more than 100 medical schools.4 The VA funds approximately 8.5% of all U.S. postgraduate positions in Accreditation Council for Graduate Medical Education-accredited training programs.4 In their VA educational experiences, medical students and physician residents share common patients, faculty, and work environments.
In this study, we evaluated the data from the LPS to achieve three objectives: (1) to compare the overall level of satisfaction of medical students against that of residents during VA-based training, (2) to identify, from a broad array of domains and items, the factors most strongly associated with medical student and resident satisfaction with their training, and (3) to determine whether the factors (domains and items) associated with training satisfaction for medical students and physician residents are different.
The VA LPS was developed to examine and measure learner satisfaction for all health care trainees in the VA system. The development and initial use of the LPS has been described and previously reported.3 In brief, the survey was developed beginning in 1999 under the sponsorship and guidance of the Office of Academic Affiliations (OAA) of the VA. The use of the LPS to meet an educational performance measure was an explicit part of the rationale for the survey. An advisory committee of VA educational leaders was assembled by the OAA to develop the LPS. After a systematic review of the literature on learner satisfaction, potential domains and items were identified. The survey was pilot tested in more than 1,000 trainees from 22 geographically diverse VA medical centers. On the basis of pilot testing, the survey design was refined to provide an overall training satisfaction score and to assess levels of satisfaction in four educational domains: clinical faculty or preceptor, learning environment, working environment, and physical environment.
Each of these four domains encompasses specific items that were determined through multiple regression analyses to contribute to the overall training satisfaction score for the domain.3 The items are fully listed in Appendix 1. When implemented in April 2001 and continuing in each annual survey to the present time, the LPS included the four domains and their associated items. For each of the items, respondents rated their satisfaction with their VA training experience using a five-point Likert scale (5 = very satisfied, 4 = somewhat satisfied, 3 = neither satisfied nor dissatisfied, 2 = somewhat dissatisfied, and 1 = very dissatisfied). Trainees also rated their overall satisfaction with each of the four educational domains on a similar five-point Likert scale after evaluating each item within its respective domain.
Over time, additional questions have been added to the LPS to explore the impact of changes in the clinical education environment on trainee satisfaction. For the current analysis, we only evaluated the components of the survey that have been unchanged since its implementation in 2001.
For an overall VA training satisfaction score, trainees answered the following question: “On a scale of 0 to 100, where 100 is a perfect score and 70 is a passing score, what numerical score would you give your most recent VA clinical training experience?” The numerical response to this question is the “overall training satisfaction score” and is one of the primary outcome measures used in this analysis.
Other measures include the four educational domains: clinical faculty/preceptor, learning environment, working environment, and physical environment. The decision to include the overall training satisfaction score and four domains was made a priori to capture the dataset that was continuously present since the first implementation of the survey in 2001.
The current report presents survey results from a six-year summary analysis of the annual LPS surveys of trainees’ satisfaction with their training experiences at VA medical centers from 2001 through 2006.
We invited all medical students and physician residents involved in a course of undergraduate or graduate medical education and rotating at least once during the academic year to any VA teaching facility to complete the LPS, beginning in the spring of 2001. Medical students and physician residents who responded to the survey were included in this study if they completed the following LPS items: the overall training satisfaction score, all four domain scores, their level of medical training, and status as an undergraduate or graduate medical trainee.
Survey administration and number of eligible participants
In 2001, trainees registered to participate in the survey through a postcard registration process. Registered trainees were mailed a paper survey or could complete an online version of the survey. For all other years of the survey, there was no separate registration process; rather, we encouraged all physician trainees (medical students and residents) rotating at least once at a VA facility to participate in the survey through a combination of national and local recruitment efforts. Nationally, letters of information and invitation were sent out from the OAA to all physician trainees for whom addresses were available during the first four years of the survey, with an emphasis on completing the survey in April of each academic year. In addition, individual VA facilities were encouraged to develop complementary local processes to encourage trainee participation in the survey. Local processes for trainee recruitment varied. The survey was available in both paper and online versions in 2001 through 2003. Since 2004, the survey has been administered only online.
We can only estimate the number of medical students and physician residents eligible to take the LPS. In fiscal year 2006, the VA sponsored 8,851 residency positions4; however, because each resident trainee position may be filled by more than one trainee during the course of a year, the exact number of residents on rotation in the VA is imprecise. Furthermore, because the administration of the survey involved direct mailing and e-mail communications with trainees who have very dynamic contact information, we were unable to verify whether the request for survey participation was actually received by all potential participants. Both of these factors impair our ability to make a precise estimate of the effective pool of trainees who were available to take the survey and received an invitation.
Mixed-effects models were used to compute the association between individual items on domain scores and, in turn, the association of differences in domain scores with differences in facility-wide satisfaction scores. Models were adjusted for nesting of respondents among 125 facilities and during six years of assessments from 2001 through 2006. Adjusting for individual respondents was not possible because individual responders were anonymous. Medical students were not asked to specify the clinical rotation at the VA (e.g., internal medicine, surgery, psychiatry, etc.). Thus, an adjustment for program specialties was not available for medical students and, hence, for consistency, was not used for medical residents.
To evaluate overall training satisfaction scores, tested variables included the main effect of medical student status and mean centered domain scores for each of the four domains, and the respective medical student status by mean centered domain score interactions for each of the four domains. We computed means for each domain across both resident and medical student scores.
To assess factors associated with domain scores, main effects included medical student status and mean centered item scores for each item, and the corresponding medical student status by mean centered item score interactions. The learning environment domain encompassed 15 items, 13 items each for clinical preceptors and work environment, and 12 items for physical environment. We computed means for each item across both residents and medical student scores within each respective domain. The differences in associations for residents and medical students were calculated as mean differences.
Tests for content validity and internal consistency
The items comprising each of the LPS four domains have been assessed for both content validity3 and for internal consistency. According to these data, respondents’ answers met reliability criteria for the 15-item learning environment domain (α = 0.93, n = 15,054), the 13-item clinical faculty/preceptors domain (α = 0.96, n = 11,569), the 13-item working environment domain (α = 0.91, n = 15,707), and the 12-item physical environment domain (α = 0.89, n = 14,409).
We performed additional analyses to assess reliability for the items of other domains included in the LPS. These include internal inconsistency for the 15 items comprising clinical environment of α = 0.94 (n = 9,259), 9 items comprising clinical training experience of α = 0.87 (n = 18,046), 13 items comprising assessment of the availability and time-liness of staff and services of α = 0.93 (n = 10,190), 13 items comprising personal experience of α = 0.94 (n = 11,856), 6 items comprising the quality of staff and services of α = 0.86 (n = 15,153), and 6 items of systems and process in dealing with medical errors of α = 0.96 (n = 13,308).
All statistical procedures were performed using a standard statistical software package (Statistical Package for the Social Sciences version 13 and Hierarchical Linear Models version 5). Because of the multiple analyses, we considered findings to be statistically significant at a P < .001 level. In reporting the association of different items on the domain score, the items with a P < .001 statistically significant association and a > 0.1 impact on domain score for either medical students or residents were considered significant and were reported.
The U.S. Office of Management and Budget, which reviews and approves federal-government-sponsored surveys, approved our survey. We maintained confidentiality by keeping respondents’ information in a separate database and reviewing only aggregate data. Participation in the survey was completely voluntary. The confidential nature of the data collection and voluntary participation were fully disclosed to survey participants.
Overall training satisfaction score
During the six-year observation period, 23,110 medical students and physician residents from 125 VA facilities participated in the LPS (Table 1). The best estimate for response rate, based on the number of funded resident positions, is 31% for medical residents, with a similar response rate estimated for medical students. The mean overall training satisfaction scores of medical students and physician residents were 83.6 and 79.1, respectively (P < .001) (Table 2). After correcting for differences in the distribution of medical students and residents across 125 VA facilities and by survey year, medical students continued to have higher overall training satisfaction scores than residents, with a mean difference of 4.6 points (P < .001) (Table 2).
Respondents’ levels of training influenced their overall training satisfaction scores (Figure 1). Adjusting for facility and survey year, medical students’ satisfaction decreased from the first to the fourth year. (Δ = −0.67/year, 95% CI [−1.04, −0.30], t = 3.54, P < .001). In contrast, physician residents’ satisfaction scores were found to increase with level of training advances (Δ = 0.54/year, 95% CI [0.26, 0.83], t = 3.8, P < .001). In fact, PGY4 through PGY7 residents assigned higher overall training satisfaction scores to facilities than PGY1 through PGY3 residents (80.44 for PGY4 through PGY7 versus 78.92 for PGY1 through PGY3, Δ = 1.52, 95% CI [0.71, 2.34], t = 3.7, P < .001).
Association of domain satisfaction score with overall training satisfaction score
For medical students and residents, each domain’s score had a statistically significant association with overall training satisfaction score rating (P < .001)(Table 2). Domain score effects on overall satisfaction varied for the four different domains, but they were rank-ordered similarly by medical students and residents. For medical students and residents, the learning environment domain had the largest association with overall training satisfaction score. The learning environment domain score’s impact on the overall training satisfaction score was higher for physician residents than for medical students, although this difference did not achieve statistical significance.
Association of items of domain score
We calculated the scores for the individual items in each domain and the effects of the individual items of each domain on the overall domain score (Table 3). Results were analyzed separately for medical students and for residents; we report the difference in each item’s effect on the overall domain score by respondent. After correcting for differences in the distribution of medical students and residents by VA facilities and survey year, medical students assigned higher domain scores than residents for learning environment and working environment (P < .001). Analysis showed that the satisfaction ratings for the majority of items in each of the domains were highly correlated with the overall domain score for medical students as well as residents.
The key items contributing to the learning environment domain score were preparation for future training and quality of care. The impact of the quality-of-care item on the learning environment domain score was higher for physician residents than for medical students, although this result did not achieve statistical significance (P = .002). The quality and teaching ability of clinical faculty were key items determining the clinical faculty/preceptor domain. Work space, peer group and faculty morale, and facility maintenance upkeep and cleanliness were important items for both residents and medical students for the working environment and the physical environment domains, respectively.
Medical students reported a higher percentage of very satisfied responses to the domain scores (learning environment, clinical faculty/preceptor, working environment, and physical environment). When we evaluated the individual items within each domain, the medical students rated 51 of the 53 items higher compared with residents ’ responses. The percentage of very satisfied responses for the approachability/openness and accessibility/availability items in the clinical faculty/preceptor domain was lower for the medical students than for the physician residents (see Appendix 1).
Three key observations can be made from our findings. First, the reported level of trainee satisfaction was higher for medical students than for residents in a common training environment, with differences in satisfaction occurring as medical students and residents advanced in their training while in a common training environment. Second, the domains and items associated with the overall satisfaction with VA-based training of medical students and physician residents were similar. Third, the learning environment domain rating had the strongest association with learners’ overall training satisfaction, whereas ancillary matters associated with the physical environment were found to be less important. The associations found were very similar for both medical students and residents. In the learning environment and clinical faculty domains, the quality of care and quality of faculty were the most important items.
These findings are important because, to our knowledge, ours is the first study to simultaneously compare multiple domains of trainee satisfaction in a common clinical training environment through a serial cross-sectional national survey of medical students and residents. The LPS is a comprehensive satisfaction survey that permits evaluation and comparison of perceptions of clinical training experiences across the entire continuum of medical education. This six-year summary analysis of the LPS shows that, while varying in the degree of satisfaction, the basis for medical students’ and residents’ perceptions of their VA training was very similar overall. Although overall satisfaction was highest among medical students, overall satisfaction varied across the continuum of medical education. Satisfaction was highest early in medical school and lowest during the first postgraduate year of residency.
Much of the literature on both medical student and resident physician satisfaction has focused on individual teachers’ behaviors and styles1,5–7 as important measures of educational effectiveness. Unlike other surveys of trainee perceptions, the LPS looks beyond the role of teaching and role models in evaluating satisfaction with clinical training. Learning environment (hands-on, clinical training), working environment (support, ancillary staff and equipment), and physical environment domains are all incorporated into the LPS. Our results suggest that, at least in the VA setting, the learning environment has the greatest influence on learners’ overall training satisfaction, and the physical environment, working environment, and clinical faculty domains are less important. Although medical students’ and residents’ perceptions were similar, the effect of the learning environment domain on the overall satisfaction score was higher for residents. While this difference approached statistical significance, the contribution of the learning environment domain on medical students’ overall training satisfaction score may be limited by the “ceiling effects” of the higher medical student overall training satisfaction scores.
Previous studies have suggested that many individual items measured by the LPS’s working environment domain and clinical faculty/preceptor domain are important to medical students’ and resident physicians’ satisfaction with their clinical education.1–3,5–7 Items within the LPS’s learning environment domain such as autonomy, supervision, and spectrum of patient problems have also been asso-ciated with learners’ satisfaction.1,3,8–11 The LPS data further suggest that the satisfaction of medical students and physician residents with the learning environment is significantly influenced by the perceived quality of health care provided at the facility. VA health care has changed dramatically in the last decade, with documented improvements in the quality of care as reflected in a broad range of performance measures, such that currently, the VA mostly exceeds private-sector performance.12,13 The data we have presented here may indicate that medical students and residents recognize the VA’s improvement in quality indicators. In a broader sense, quality of care may be an underappreciated item in medical students’ and resident physicians’ satisfaction with their training. Given their direct, daily, hands-on patient contact, medical students and residents may be uniquely positioned to perceive and assess the quality of care delivered in the clinical environments where they work.
Learning theory and previous studies suggest that trainees at different levels may have different learning needs.1 Less is known about the effects of the learning environment and its relative contribution to trainee satisfaction at various levels of medical education. One study in ambulatory teaching sites demonstrated that medical students and residents from five medical schools in Ontario differed in their perceptions of what enhanced their ambulatory training experience.1 Rating the importance of “site characteristics” and “preceptor behaviors” to their learning, medical students more highly valued preceptor interactions (effective teachers, readily available preceptors, opportunity to observe preceptor), whereas residents valued patient mix and practice logistics (adequate number and variety of patients, opportunity to see patients independently). This LPS-based observation suggests that clinical aspects of the learning environment were more important to residents. In a recent study from the United Kingdom,14 medical students assigned to traditional teaching hospitals were compared with students at other types of hospitals. Students were more satisfied with the teaching at the traditional hospitals but more satisfied with the clinical and practical environment at the nontraditional hospitals. Specific practice environment items that were more valued by medical students included the spectrum of problems, opportunity for hands-on experience, and practice of clinical skills.14 Our analysis of the LPS’s national sample of trainees suggests that learners at all levels of medical education placed a higher value on the learning environment than on teaching style and behavior. In VA facilities, although learners’ satisfaction with clinical faculty was very high, the learning environment, adjusted at a facility level, contributed most to the overall training satisfaction score. Prior work1,14 has a limited and varied description of learning environment. The comprehensive development of the LPS has brought further clarity to the content of this important domain. Furthermore, items used to evaluate the perceptions of the learning environment domain, as described in this study, should be taken into consideration in assessing overall training satisfaction in other types of hospitals, settings, and training years.
The results for overall satisfaction as measured by the LPS showed significant differences in satisfaction by level of training. More advanced medical students were less satisfied than first-year students, whereas PGY1 through PGY3 residents were less satisfied than more advanced PGY4 through PGY7 residents. The overall satisfaction differences by training year may represent a reflection of what is known about medical trainee mood and attitude changes through the course of their training. Cynicism seems to increase among medical students, decline during residency, and reach its lowest point among attending physicians.15 Third-year medical students as well as interns have a loss of idealism during training.16,17 Attitude and mood changes affect medical students and physician residents, with improvements towards the end of their training.18,19 These findings lend face validity to the LPS overall satisfaction measure and demonstrate the value of serial administration of the survey across all levels of medical training. Further analysis of these trends and their impacts on the survey domains and items may lead to a better understanding of trainees’ perceptions and needs at various levels of medical education.
Our findings may provide direction to organizations that are interested in improving trainee satisfaction. The VA is an organization with a stated commitment to the education of health care professionals, and, as such, it is invested in making strides in the improvement in its trainees’ satisfaction. Improvements in the learning environment, especially quality of care, while challenging, are more likely to result in larger increases in trainee satisfaction then modifications to the physical environment (e.g., parking and food services).
This study has several limitations. The first limitation was the relatively low response rate of the LPS. No registry existed to identify all potential respondents, so the size of the target populations can only be estimated. The limited penetration raises the issue of sampling bias. Fortunately, from 2002 to 2004, questions identical to those included in the LPS were included in the fourth-year medical students’ questionnaire by the Association of American Medical Colleges (AAMC), which has a response rate of 97%. The value of the VA clinical training experience was rated similarly by fourth-year students in both the AAMC questionnaire (72%–73% rating of exce-llent or very good) and LPS (73%–78% rating of excellent or very good). Thus, on the basis of a comparison with the AAMC questionnaire, our findings indicate that the fourth-year medical students responding to the LPS were representative of fourth-year medical students in general.
A second limitation is that the survey evaluated the most recent VA clinical training experience and does not distinguish between the medical student training specialty or setting (i.e., medicine versus surgery, or outpatient versus inpatient). Although data are available for the specific disciplines of physician resident specialty, we elected to compare students against the entire pool of postgraduate trainees rather than adjust for training program. Important differences in the perceptions of medical students in different clinical settings and physician residents in different specialties may exist, but these could not be explored in our present analysis.
Additionally, the data do not permit analyzing the responses of an individual respondent over time. Because data were collected anonymously through annual surveys, the longitudinal experience of individuals and the impact of repeat respondents cannot be determined.
Finally, the LPS data are limited to the description of associations between the perceptions of respondents and their clinical training at a VA medical facility. Commonalities in the care provided in VA facilities are an advantage to our study because they allow comparisons across training sites; however, the generalizability of findings related to VA training to other training sites may be limited. Nevertheless, one could safely assume that the same factors which influence the satisfaction or dissatisfaction of medical students and residents would be the same regardless of the clinical setting.
In conclusion, medical students and resident physicians expressed high levels of satisfaction with their VA clinical training, with medical students giving higher ratings than residents. The factors contributing to overall satisfaction are remarkably similar for medical students and residents. Whereas the clinical learning environment is the domain most strongly correlated with the overall training satisfaction score, the other three domains—clinical faculty/preceptors, working environment, and physical environment—were important, suggesting that efforts to improve learners’ satisfaction should incorporate all aspects of the clinical training experience. Future work that focuses interventions on key factors associated with trainee satisfaction, as described in this study, is likely to find improvement in the satisfaction of both medical students and residents. Finally, while optimizing trainee satisfaction is important, understanding the linkage between satisfaction and competence warrants further study.
The authors wish to acknowledge the members of the Learners’ Perceptions Advisory Committee and the dedicated staff in the Office of Academic Affiliations that together created and implemented the VA’s Learners’ Perceptions Survey. Special thanks go to VA Office of Academic Affiliations staff members Dilpreet Singh, MS, MPA, Evert Melander, MBA, David Bernett, Terry Kruzan, and Ed McKay.
Other members of the Learners’ Perceptions Working Group: John M. Boyle, PhD, Schulman, Ronca & Bucuvalas, Inc., Silver Spring, Md; C. Richard Buchanan, DMD, FICD, Department of Veterans Affairs, Washington, DC, and University of the Pacific School of Dentistry, San Francisco, Calif; Christopher T. Clarke, PhD, VA Medical Center and St. Louis University, St. Louis, Mo; Stephen J. Dienstfrey, MA, MBA, Schulman, Ronca & Bucuvalas, Inc., Silver Spring, Md; Sheila C. Gelman, MD, VA Health Care System of Ohio, Department of Veterans Affairs, Cincinnati, Ohio; Stuart C. Gilman, MD, MPH, Department of Veterans Affairs and Long Beach Department of Medicine, University of California, Irvine, Calif; Mark Graber, MD, VA Medical Center, Northport, NY, and Department of Medicine, SUNY at Stony Brook, NY; Charles G. Humble, PhD, Office of Quality Improvement, Department of Veterans Affairs, Morrisville, NC; Linda D. Johnson, PhD, RN, Office of Academic Affiliations, Department of Veterans Affairs, Washington, DC; Catherine P. Kaminetzky, MD, MPH, Durham VA Medical Center and Department of Medicine, Duke University Medical Center, Durham, NC; Mark Meterko, PhD, VA HSR&D Management and Decision Research Center, Boston, Mass; Don D. Mickey, PhD, VA Medical Center, Durham, NC; Gary Nugent, VA Medical Center, Omaha, Ne; and Antonette M. Zeiss, PhD, VA Palo Alto Health Care System, Palo Alto, Calif.