Secondary Logo

Journal Logo

Research Reports

U.S. Internal Medicine Residents’ Knowledge and Practice of High-Value Care

A National Survey

Ryskina, Kira L. MD; Smith, Cynthia D. MD; Weissman, Arlene PhD; Post, Jason MD; Dine, C. Jessica MD; Bollmann, KeriLyn MD; Korenstein, Deborah MD

Author Information
doi: 10.1097/ACM.0000000000000791


Concerns about the high costs and comparatively low quality of health care in the United States1,2 have led to calls to emphasize high-value care (HVC), defined as care that balances the potential benefits, harms, and costs of tests and treatments,3 in graduate medical education (GME).4–9 In 2012, the Accreditation Council for Graduate Medical Education incorporated the practice of cost-effective care into the internal medicine (IM) milestones,10 though optimal methods for teaching and evaluating these skills were not specified.

Variations in care intensity (or the amount of care provided) across regions, institutions, and individual physicians have been well described.11–15 Although the impact of these regional and institutional differences on residents’ attitudes and behaviors is not known, the care intensity of the local environment has been associated with the use of invasive cardiac procedures,16 the use of “discretionary” care of unclear appropriateness,13 and the ability to appropriately choose conservative management options on the IM specialty certification exam.17 Practicing U.S. physicians recognize the high rate of unnecessary care provided (or care that would not improve patient outcomes, based on current evidence) and the need to reduce it.18,19 Although cost profiles are higher among less experienced practicing physicians than among those with more experience,20 no large-scale studies have evaluated residents’ views of value-based care, despite the recent emphasis on the issue in GME and the need for baseline data to inform curriculum design.

To evaluate residents’ views on this topic, we performed a national survey of U.S. IM residents, asking about their self-perceived knowledge and practice of HVC and their attitudes toward issues related to HVC. We also evaluated associations between residents’ views and the care intensity of their training hospitals as well as their exposure to formal and informal teaching about HVC. Applying social learning theory, which emphasizes the role of modeling behaviors and the importance of the social environment on learning,21,22 we hypothesized that residents training in high-care-intensity environments would report less knowledge and practice of HVC.


Study participants and data collection

The Internal Medicine In-Training Examination (IM-ITE) is a multiple-choice exam offered annually across the United States to help residents assess their knowledge of IM in preparation for taking the specialty certification exam.23 Program directors determine whether their residents, at all levels of training, will take the IM-ITE. A two-page self-administered paper survey (see Supplemental Digital Appendix 1 at was distributed to all IM-ITE examinees in October 2012. Respondents were notified in writing that the survey was voluntary and confidential, and that the results may be used for research. They could refuse to participate in any research by checking a box or leaving the survey blank. No incentives to participate were offered. Respondents completed the survey immediately after taking the IM-ITE and returned their completed surveys to the room monitors. We excluded from analysis respondents who were not residents (postgraduate year [PGY] 4 level or higher), those training in non-U.S. residency programs, those with mismatched or duplicate identification numbers, and those who did not provide consent to use their data for research. We used the American Association for Public Opinion Research RR2 response rate definition to compute the response rate.24 We were unable to analyze demographic differences between respondents and nonrespondents because no demographic characteristics were available for nonrespondents.

Survey instrument

Staff from the American College of Physicians (ACP) developed the survey in collaboration with the Alliance for Academic Internal Medicine (AAIM) to measure the constructs and dimensions of HVC as delineated in the AAIM-ACP High Value Care Curriculum25 and to evaluate existing gaps in self-perceived knowledge and practices related to HVC across subgroups of residents (i.e., by PGY level). Prior to implementation, the survey was reviewed and revised nine times by six medical education experts, including current and former IM program directors and an expert in survey design, to improve the content and face validity of the instrument.

The survey included questions related to career plans, the IM-ITE itself, and HVC (see Supplemental Digital Appendix 1 at for the full survey instrument). HVC questions covered the following areas: (1) five questions about potential causes of overuse, (2) eight questions on knowledge and practice of HVC, and (3) two questions about exposure to HVC teaching. Questions about potential causes of overuse and knowledge and practice of HVC were answered on a five-point Likert scale of agreement (strongly disagree, somewhat disagree, neither agree nor disagree, somewhat agree, or strongly agree). The questions about exposure to HVC teaching were answered on a five-point Likert scale of frequency (never, few times a year, few times a month, few times a week, or every day).

Determination of the care intensity of the training hospitals

We used the hospital care intensity (HCI) index developed and calculated by the Dartmouth Atlas group to characterize the intensity of the care environment of each respondent’s primary training hospital. The HCI index is a composite age–sex–race–illness standardized measure of hospital days and inpatient physician visits by Medicare recipients in the last two years of life, and has been used as a measure of the variation in the use of hospital and physician services.15 We used residency training program Web sites to identify each respondent’s primary training hospital and obtained each hospital’s 2010 HCI index from the Dartmouth Atlas Web site.26 Across the United States, HCI indices in 2010 ranged from 0.35 (low intensity) to 3.41 (high intensity), with a national mean of 1.0.15

Data management and analysis

Paper survey responses were scanned, and the data were merged with resident demographics and residency program information. Respondents were characterized according to their year of training, sex, medical school location, residency program type, career plans, and the quartile of their training hospital’s HCI index. Responses to questions about HVC knowledge and practice and causes of overuse were dichotomized into “somewhat agree” and “strongly agree” versus the other options.

Logistic mixed-effects models with program random effects were used to measure the associations between both levels of exposure to HVC teaching and care intensity of the training hospital with self-reported HVC knowledge and practice, accounting for correlation at the program level and adjusting for resident and program characteristics (i.e., sex, PGY level, residency program type, medical school location, degree, and career plans). Robust standard errors were estimated. Missing data resulted in observations being dropped from the model. The Bonferroni method of correction for multiple comparisons was used to adjust the significance level (specifying significance at P < .001, using two-sided testing).

All analyses were conducted using STATA version 13.0 (StataCorp, College Station, Texas). The Mayo Clinic institutional review board reviewed this study and deemed it exempt.



Of the 21,617 U.S. IM residents who received the survey, 18,102 (83.7%) completed it. Among respondents, 7,889 (43.6%) were female; 6,180 (34.1%) were PGY1, 6,264 (34.6%) were PGY2, and 5,658 (31.3%) were PGY3. Slightly more than half (10,114/17,987; 56.2%) were graduates of U.S. medical schools, and the majority (15,767/18,042; 87.4%) were in categorical residency programs. Of the 18,060 respondents who answered the question, 10,723 (59.4%) were planning to pursue subspecialty training; 4,909 (27.2%) were planning to pursue general IM careers including as a hospitalist; and 1,504 (8.3%) were undecided about their career plans. See Table 1 for complete respondent characteristics.

Table 1
Table 1:
Demographic Characteristics of U.S. Internal Medicine Residents Who Completed a Cross-Sectional Survey of Their Knowledge and Practice of High-Value Care, October 2012

Self-reported knowledge and practice of HVC

We observed wide variation in the level of agreement with statements about knowledge and practice of HVC (see Table 2). Whereas relatively few respondents agreed with “I know where to find estimated costs of tests and treatments” (4,654/17,687; 26.3%) and with “I share estimated costs of tests and treatments with patients” (4,187/17,633; 23.7%), 8,071 of 17,652 respondents (45.7%) agreed that they “incorporate the cost of tests and treatments into clinical decisions.” Over two-thirds of respondents reported adequate knowledge of the benefits and harms of tests and treatments, offering patients alternatives, considering patients’ values and concerns, and avoiding unnecessary care (see Table 2). The majority (10,356/17,654; 58.7%) agreed with “I reduce health care waste within my hospital and/or clinic.”

Table 2
Table 2:
Self-Reported Knowledge and Practice of High-Value Care According to U.S. Internal Medicine Residents, by the Hospital Care Intensity (HCI) Index of Their Training Hospital, October 2012

Perceived causes of overuse

Most respondents agreed that overuse was driven by defensive medicine (15,015/17,745; 84.6%), followed by diagnostic uncertainty (10,811/17,752; 60.9%), patient demands (9,683/17,681; 54.8%), and concerns about inadequate patient follow-up (8,346/17,703; 47.1%) (see Table 2).

Exposure to HVC teaching

Respondents reported more frequent exposure to HVC teaching by faculty during patient care than during formal conferences or rounds (see Table 3). Balancing the benefits, harms, and costs of care was the topic of discussions with faculty during patient care at least a few times a week (7,103/17,704; 40.1%) and the subject of teaching conferences or rounds at least a few times a week (4,066/17,624; 23.1%).

Table 3
Table 3:
Exposure to High-Value Care Teaching According to U.S. Internal Medicine Residents, by the Hospital Care Intensity (HCI) Index of Their Training Hospital, October 2012

Factors associated with self-reported knowledge and practice of HVC

Self-reported knowledge and practice of HVC and the perceived barriers to it varied by the quartile of the training hospital’s care intensity for a number of items (see Table 2), though the differences were small. Respondents training at higher-intensity hospitals were more likely to report knowing where to find estimated costs of tests and treatments: 1,002 of 4,124 (24.3%) in the bottom quartile versus 1,158 of 4,060 (28.5%) in the top quartile (P < .001). They also were more likely to report reducing health care waste within their hospital and/or clinic: 2,247 of 4,109 (54.7%) in the bottom quartile versus 2,468 of 4,056 (60.8%) in the top quartile (P < .001). Respondents training at lower-intensity hospitals were more likely to report that they incorporate the cost of tests and treatments into clinical decisions: 2,030 of 4,113 (49.4%) in the bottom quartile versus 1,809 of 4,046 (44.7%) in the top quartile (P < .001). They also were more likely to report that they incorporate patients’ values and concerns into clinical decisions: 3,709 of 4,106 (90.3%) in the bottom quartile versus 3,455 of 4,045 (85.4%) in the top quartile (P < .001). Respondents training at lower-intensity hospitals were more likely to agree that all presented factors were possible causes of overuse, with the largest difference in the identification of physician discomfort with uncertainty as a cause of overuse: 2,678 of 4,130 (64.8%) in the bottom quartile versus 2,314 of 4,082 (56.7%) in the top quartile (P < .001).

The findings from our multivariable regression models are presented in Table 4. Training at a higher-intensity hospital was associated with lower odds of agreement with two statements: “I incorporate the cost of tests and treatments into clinical decisions” (for the third and top quartiles compared with the bottom quartile) and “I incorporate patients’ values and concerns into clinical decisions” (for the top quartile compared with the bottom quartile). A training hospital’s care intensity was not associated with the other six reported HVC practices. Discussing HVC with faculty during patient care at least a few times a week (compared with never) was associated with all self-reported HVC practices (see Table 4). Exposure to HVC teaching during formal conferences was associated with knowing where to find cost information, sharing costs with patients, incorporating cost into clinical decisions, and reducing health care waste within the hospital/clinic, but not with the other HVC behaviors (see Table 4).

Table 4
Table 4:
Adjusted Odds of Agreement With Statements About High-Value Care Knowledge and Practice, by Quartile of Hospital Care Intensity (HCI) Index and Level of Exposure to and Modality of High-Value Care Teaching, According to U.S. Internal Medicine Residents, October 2012


In this large survey of U.S. IM residents, most reported both knowing the benefits and harms associated with common tests and treatments and practicing many components of HVC. However, only about one in four reported knowing where to find cost information, and fewer than half reported sharing costs with patients. Most residents reported at least some exposure to faculty teaching about balancing the benefits, harms, and costs of medical interventions, in spite of the findings of a recent survey of IM program directors, which found that a minority reported formal curricula in cost-conscious care.27 After adjusting for resident and program characteristics, both the care intensity of the training hospital and exposure to teaching about balancing the benefits, harms, and costs of care were associated with at least some self-reported HVC knowledge and practice. This association was more consistent for exposure to HVC teaching during patient care compared with exposure to HVC topics during formal conferences or rounds.

Our findings of residents’ lack of knowledge of where to find cost information and low rates of sharing costs with patients are not surprising. Medical educators have traditionally avoided conversations about the costs of medical interventions,5 and residents have demonstrated an inability to accurately estimate costs.28,29 A minority of residents reported taking part in discussions about cost, a critical component of HVC, though most reported knowledge and practice of other components of HVC that involve balancing the benefits and harms of medical interventions: 88% reported incorporating patients’ values into clinical decisions and 81% reported offering alternatives to patients, whereas somewhat fewer (72%) reported avoiding ordering unnecessary tests and treatments. However, all of these responses may have been inflated by the social desirability of a positive response. Alternately, other factors may have led residents to underreport some HVC practices. For example, residents may have underreported the impact of cost consideration if cost concerns have become a natural, subconscious component of clinical decision making. Further, residents who undervalue HVC teaching may tend to over- or underreport its frequency. Nevertheless, the fact that nearly 30% of residents reported not avoiding unnecessary care is notable and suggests a need for better education for residents about the potential harms of unnecessary care.

We hypothesized that self-reported practice of HVC would be less frequent in higher-intensity environments, reflecting the social norming effects of exposure to high care intensity in day-to-day interactions. We found that residents in high-intensity hospitals were somewhat less likely to report both incorporating patient values and concerns into clinical decisions (85% versus 90%) and incorporating cost into clinical decisions (45% versus 49%), although the differences were small. On the other hand, slightly more residents in higher-intensity settings reported knowing where to find costs (29% versus 24%), which may reflect increased attention to cost as administrators and educators seek to improve efficiencies, or the fact that residents in high-intensity environments overestimate their knowledge of costs. These mixed findings also may reflect the challenges in defining HVC, overuse, and unnecessary care, particularly within the constraints of a survey question. Ultimately, the small differences in attitudes based on the care intensity of the training hospital, although statistically significant, are unlikely to explain considerable practice differences.

In response to this survey, residents reported more exposure to teaching about HVC in the informal curriculum (in the context of patient care) than in the formal curriculum (in the context of lectures and conferences), and HVC teaching in informal settings appeared to be more strongly correlated with self-reported HVC practice. These findings point to a potential disconnect between educational messages from these two modes of teaching and highlight the importance of the informal curriculum in shaping residents’ future practice during training. Taken together, our findings about the roles of exposure to teaching and care intensity suggest the importance of the training environment in shaping residents’ HVC practices. Others have emphasized the importance of the practice environment and the need to change culture to improve value.7,17 Our findings speak to the importance of educational and other interventions that target the practice environment,6 though optimal strategies have not been defined.

The majority of residents agreed that malpractice concerns drive unnecessary testing. This finding is consistent with the findings from surveys of practicing physicians in high-risk specialties who reported practicing defensive medicine,30 and indeed the perception of malpractice risk has been shown to be a driver of unnecessary care in office-based practice.31 Over 60% of residents also agreed that discomfort with diagnostic uncertainty was a contributing factor to overuse. Among practicing physicians, a higher level of individual discomfort with clinical uncertainty has been correlated with increased ordering in a health maintenance organization setting32 and with lower scores on a physician cost-consciousness scale.19 Future studies should address the role of discomfort with uncertainty in inappropriate ordering and low-value care and evaluate interventions to address it. Of note, while resident-endorsed drivers of overuse in this study were consistent with findings from studies of practicing clinicians, the relevant survey questions asked about drivers for “physicians” in general, rather than for the respondent in particular. Therefore, differences between factors motivating overuse by residents and those motivating overuse by practicing physicians may have been obscured.

Our study has a number of important limitations. First, our findings are limited by the cross-sectional survey study design, and we cannot draw conclusions about causality. Second, we relied on residents’ self-reported knowledge and practice of HVC, which may not accurately reflect their actual practice, and we used a novel survey instrument that was not validated in prior studies or fully field tested. In particular, questions about knowledge of the costs of care did not specifically assess residents’ knowledge of the relative or approximate costs of tests or treatments, which may be more easily available. In addition, ambiguity of the word “unnecessary” may have impacted responses to questions about avoiding unnecessary tests and treatments. Third, the HCI index, which we used as a proxy of care intensity of the training hospital, has several limitations of its own33,34 and may be associated with other important hospital characteristics. However, it is widely used as a measure of local care intensity. Lastly, while we identified the primary training hospital of each residency program, many programs are affiliated with multiple training sites with potentially distinct care intensity characteristics. Thus, our measure of care intensity may be an incomplete depiction of the overall care intensity of the training program.

In conclusion, this large-scale investigation found that many IM residents in the United States reported practicing several components of HVC, with the notable exception of including cost information in clinical decisions. We observed variation in residents’ self-reported knowledge and practice of HVC by the level of care intensity of their training hospital, suggesting that care intensity of the environment may play a role in shaping high-value practice. Exposure to faculty discussions of HVC during patient care showed the highest correlation with reported HVC behaviors and may represent an opportunity to make needed improvements in residents’ competency in HVC. These findings may offer insights for educators, hospital administrators, and policy makers designing interventions to change the culture of medicine to increase value for patients.


1. Berwick DM, Hackbarth AD. Eliminating waste in US health care. JAMA. 2012;307:1513–1516
2. Institute of Medicine. The Healthcare Imperative: Lowering Costs and Improving Outcomes. 2010 Washington, DC National Academies Press
3. Owens DK, Qaseem A, Chou R, Shekelle PClinical Guidelines Committee of the American College of Physicians. . High-value, cost-conscious health care: Concepts for clinicians to evaluate the benefits, harms, and costs of medical interventions. Ann Intern Med. 2011;154:174–180
4. Emanuel EJ, Fuchs VR. The perfect storm of overutilization. JAMA. 2008;299:2789–2791
5. Cooke M. Cost consciousness in patient care—what is medical education’s responsibility? N Engl J Med. 2010;362:1253–1255
6. Weinberger SE. Providing high-value, cost-conscious care: A critical seventh general competency for physicians. Ann Intern Med. 2011;155:386–388
7. Korenstein D, Kale M, Levinson W. Teaching value in academic environments: Shifting the ivory tower. JAMA. 2013;310:1671–1672
8. Detsky AS, Verma AA. A new model for medical education: Celebrating restraint. JAMA. 2012;308:1329–1330
9. Hackbarth G, Boccuti C. Transforming graduate medical education to improve health care value. N Engl J Med. 2011;364:693–695
10. Caverzagie KJ, Iobst WF, Aagaard EM, et al. The internal medicine reporting milestones and the next accreditation system. Ann Intern Med. 2013;158:557–559
11. Bynum J, Song Y, Fisher E. Variation in prostate-specific antigen screening in men aged 80 and older in fee-for-service Medicare. J Am Geriatr Soc. 2010;58:674–680
12. Donohue JM, Morden NE, Gellad WF, et al. Sources of regional variation in Medicare Part D drug spending. N Engl J Med. 2012;366:530–538
13. Sirovich B, Gallagher PM, Wennberg DE, Fisher ES. Discretionary decision making by primary care physicians and the cost of U.S. health care. Health Aff (Millwood). 2008;27:813–823
14. Sirovich BE, Gottlieb DJ, Welch HG, Fisher ES. Variation in the tendency of primary care physicians to intervene. Arch Intern Med. 2005;165:2252–2256
15. Arora A, True ADartmouth Atlas of Health Care. . What kind of physician will you be? Variation in health care and its importance for residency training. A report of the Dartmouth Atlas Project. October 30, 2012. Accessed April 27, 2015
16. Wennberg DE, Kellett MA, Dickens JD, Malenka DJ, Keilson LM, Keller RB. The association between local diagnostic testing intensity and invasive cardiac procedures. JAMA. 1996;275:1161–1164
17. Sirovich BE, Lipner RS, Johnston M, Holmboe ES. The association between residency training and internists’ ability to practice conservatively. JAMA Intern Med. 2014;174:1640–1648
18. Sirovich BE, Woloshin S, Schwartz LM. Too little? Too much? Primary care physicians’ views on US health care: A brief report. Arch Intern Med. 2011;171:1582–1585
19. Tilburt JC, Wynia MK, Sheeler RD, et al. Views of US physicians about controlling health care costs. JAMA. 2013;310:380–388
20. Mehrotra A, Reid RO, Adams JL, Friedberg MW, McGlynn EA, Hussey PS. Physicians with the least experience have higher cost profiles than do physicians with the most experience. Health Aff (Millwood). 2012;31:2453–2463
21. Bandura A Social Learning Theory. 1977 Englewood Cliffs, NJ Prentice Hall
22. Bahn D. Social learning theory: Its application in the context of nurse education. Nurse Educ Today. 2001;21:110–117
23. American College of Physicians. . ACP Internal Medicine In-Training Exam. Accessed April 27, 2015
24. American Association for Public Opinion Research. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 20117th ed. Lenexa, Kan American Association for Public Opinion Research
25. Smith CD. Teaching high-value, cost-conscious care to residents: The Alliance for Academic Internal Medicine–American College of Physicians Curriculum. Ann Intern Med. 2012;157:284–286
26. . The Dartmouth Atlas of Health Care: Data by Hospital. Accessed April 27, 2015
27. Patel MS, Reed DA, Loertscher L, McDonald FS, Arora VM. Teaching residents to provide cost-conscious care: A national survey of residency program directors. JAMA Intern Med. 2014;174:470–472
28. Tek Sehgal R, Gorman P. Internal medicine physicians’ knowledge of health care charges. J Grad Med Educ. 2011;3:182–187
29. Post J, Reed D, Halvorsen AJ, Huddleston J, McDonald F. Teaching high-value, cost-conscious care: Improving residents’ knowledge and attitudes. Am J Med. 2013;126:838–842
30. Studdert DM, Mello MM, Sage WM, et al. Defensive medicine among high-risk specialist physicians in a volatile malpractice environment. JAMA. 2005;293:2609–2617
31. Carrier ER, Reschovsky JD, Katz DA, Mello MM. High physician concern about malpractice risk predicts more aggressive diagnostic testing in office-based practice. Health Aff (Millwood). 2013;32:1383–1391
32. Allison JJ, Kiefe CI, Cook EF, Gerrity MS, Orav EJ, Centor R. The association of physician attitudes about uncertainty and risk taking with resource use in a Medicare HMO. Med Decis Making. 1998;18:320–329
33. Bach PB. A map to bad policy—hospital efficiency measures in the Dartmouth Atlas. N Engl J Med. 2010;362:569–573
34. Skinner J, Staiger D, Fisher ES. Looking back, moving forward. N Engl J Med. 2010;362:569–574

Supplemental Digital Content

© 2015 by the Association of American Medical Colleges