To identify our sample, we used VA's Corporate Data Warehouse (CDW) electronic medical data, accessed through VA's Informatics and Computing Infrastructure (VINCI) platform.29,30 We identified all prescriptions (pharmacy orders) for HT (systemic and topical) between June 1, 2013, and September 30, 2015, within a cohort of 5,250 women veterans, aged 40 years or older, who had at least one encounter with PCPs, across four VA facilities participating in a women's health educational program.31 These four facilities were located in core metropolitan areas, across three states, in two geographic regions of the United States. For each Veteran with one or more prescriptions for HT, we also obtained the text of all primary care and gynecology specialty care notes in this same timeframe. As we were focusing on the guideline adherence provided by PCPs, we excluded prescriptions for patients whose notes indicated that gynecologists were managing the prescription. In addition, we excluded prescriptions if the notes indicated the Veteran was transgender. For each Veteran, we included the first prescription occurring within our timeframe and any prescriptions 12 or more months after this initially included prescription. We then selected the subset of prescriptions that were for systemic HT: oral, transdermal, or estradiol acetate vaginal ring.32
For each included prescription, we used pharmacy records to determine the name of the prescriber. For other data relevant to the seven QIs, an experienced and trained medical record abstractor reviewed the primary care notes and pharmacy orders using a structured abstraction tool we created using Research Electronic Data Capture (REDCap).33 The tool used branching logic and prompted abstractors to use the documentation in the primary care notes and pharmacy orders to specify, for each prescription, its route; whether the Veteran had received a systemic HT prescription within VA in the previous 12 months; the prescription dose; documentation of indications for a dose equivalent to CEE 0.625 mg daily or greater, for prescriptions in that dose category; whether progesterone was ordered, or already being used by the Veteran; and whether the patient had undergone a hysterectomy. For all prescriptions in which there was no indication of progesterone having been ordered and the primary care and gynecology notes in the abstraction timeframe did not document the Veteran as having had a previous hysterectomy, a physician investigator (KMC) accessed the veterans' medical records to perform a broader search of the veterans' notes before our timeframe and from other disciplines (e.g., preoperative history and physicals listing previous surgeries), to determine whether the Veteran had undergone a hysterectomy or was using progesterone provided by another source.
We classified each systemic HT prescription as being “new” if the Veteran had not received a systemic HT prescription within VA in the previous 12 months and as a “renewal” if they had. We used this classification, as well as information about dose, and whether the patient had undergone a hysterectomy, to identify, for each QI, the subcohort of applicable prescriptions. Then, for each QI, we determined the number of prescriptions in that subcohort for which the PCP's documentation indicated guideline adherence. For each prescription, we calculated a composite adherence score as the proportion of eligible indicators for which the care was adherent, then determined the mean across prescriptions to report overall guideline adherence. Furthermore, for each indicator, we assessed the proportion of prescriptions that were guideline-adherent.
Using VA administrative data (the 2013, 2014, and 2015 DWHP Assessment of Workforce Capacity databases),26 we ascertained whether the provider who ordered the systemic HT was a DWHP and assessed for differences in overall guideline adherence by whether the PCP was or was not a DWHP. We determined, for each PCP, their average monthly volume of clinical encounters with women during the study period. For prescribing providers who were trainees, we used the characteristics of their supervising provider. We compared overall guideline adherence between PCPs with less than 34 (the median), to those with 34 or more, average monthly encounters with women veterans. To assess for associations between guideline adherence and PCP DWHP status and volume of encounters with women, we used simple linear regression, adjusting for clustering of prescriptions within providers. To assess for potential independent relationships between provider characteristics and overall guideline adherence, we conducted multivariate linear regression analysis, including both provider characteristics in the same model, adjusting for clustering of prescriptions within providers. Data analysis was performed using STATA version 15 (StataCorp, 2017). All study procedures were approved by the VA Greater Los Angeles Healthcare System Institutional Review Board.
Across QIs, average adherence to prescribing guidelines for systemic HT was 57.7%. For prescriptions that were new, average adherence was 60.3%, and for renewals, adherence was 57.2%.
Among the 355 prescriptions, 305 (86%) were provided by DWHPs and 50 (14%) were provided by non-DWHPs. As shown in table 5, guideline adherence was similar for prescriptions provided by DWHPs and non-DWHPs (58% vs. 55%, p = .410). Most prescriptions, 292 (82%), were managed by PCPs with a monthly average of 34 or more encounters with women. Overall adherence was also similar for episodes managed by providers with 34 or more, versus less than 34, monthly encounters with women (58% vs. 55%, p = .291). Table 5 also shows predicted values for these provider characteristics, from a multivariate linear regression model adjusted for clustering. This adjusted analysis revealed no significant relationships between either provider characteristics and overall guideline adherence.
Our findings should be interpreted in the context of their limitations. Primary care providers vary in the extent to which they comprehensively document the elements of a clinical encounter.34-36 Therefore, providers may have considered, but did not document, HT indications and contraindications. However, lack of documentation is less likely to affect initiating systemic HT at higher doses, or failing to appropriately prescribe progesterone. A further limitation of our work is that our assessment examines prescriptions by 81 PCPs in four metropolitan VA sites between 2013 and 2015; we do not know the generalizability of our findings to other VA sites, especially those in nonmetropolitan areas, nor the effects of VA's more recent efforts to improve women's health care quality.21,37,38 In addition, our findings are limited to PCP prescriptions, and prescriptions from specialists (e.g., gynecologists) should be studied separately. However, in VA, gynecology staffing is generally limited, and PCPs are more likely to be the prescribers of systemic HT.39
This is the first measurement of the extent to which VA providers are adhering to current guidelines for prescribing systemic HT. To our knowledge, comparable studies of adherence to systemic HT prescribing guidelines in patient populations and practices outside VA have not been published. However, multiple studies have shown that, in general, clinical practice guidelines have limited effect on provider behavior, with only half of patients in the United States receiving guideline-adherent care.40,41 Our finding, that average adherence was 58%, is unfortunately consistent with previous work in other conditions and populations.
Prescribing systemic HT is an area of medical practice for which the evidence base changed, and even reversed, over a short period.17,42,43 New recommendations have recently been released for providers to further tailor HT treatment decisions based on the patient's age and number of years since menopause onset.42 These new recommendations recommend a more nuanced approach to HT prescribing, necessitating a higher level of provider knowledge and judgment than basic considerations of FDA indications and contraindications used in this study. This evolution in guidelines is reflective of a broader challenge of practitioners maintaining proficiency in the context of a continuously rapid shift in evidenced-based care.44,45 To ensure that patients are getting care guided by the current evidence, health care systems must reach beyond traditional modalities for providing continuing education to their providers, and supply “just-in-time” information and decision support, such as using computerized clinical decision support (CDS) systems, at the point-of-care and/or in support of panel management programs.46-51 Computerized CDS systems electronically match characteristics of patients with evidence-based algorithms to provide providers with recommendations in support of guideline-adherent decisions.51
The authors thank Mark Canning for project management. The authors would also acknowledge the editorial review and feedback of Chloe Bird, PhD, Senior Sociologist, RAND Corporation, Santa Monica. Her time was supported through the VA Los Angeles HSR&D Center for the Study of Healthcare Innovation, Implementation, and Policy (Project #CIN 13-417).
1. Woods NF, Mitchel ES. Symptoms during the perimenopause: Prevalence, severity, trajectory, and significance in women
's lives. Am J Med. 2005;118(12):14–24.
2. Dennerstein L, Dudley EC, Hopper JL, Guthrie JR, Burger HG. A prospective population-based study of menopausal symptoms. Obstet Gynecol. 2000;96(3):351–358.
3. Ayers B, Hunter MS. Health-related quality of life of women
with menopausal hot flushes and night sweats. Climacteric. 2013;16:235–239.
4. Katon JG, Gray KE, Gerber MR, et al. Vasomotor symptoms and quality of life among veteran and non-veteran postmenopausal women
. Gerontologist. 2016;56(S1):S40–S53.
5. Avis NE, Colvin A, Bromberger JT, et al. Change in health-related quality of life over the menopausal transition in a multiethnic cohort of middle-aged women
: Study of Women
's Health across the Nation. Menopause
6. Greendale GA, Sowers M, Han W, et al. Bone mineral density loss in relation to the final menstrual period in a multiethnic cohort: Results from the Study of Women
's Health Across the Nation (SWAN). J Bone Miner Res. 2012;27(1):111.
7. Maclennan AH, Broadbent JL, Lester S, Moore V. Oral estrogen and combined estrogen/progestogen therapy versus placebo for hot flushes. Cochrane Database Syst Rev. 2004:CD002978.
8. Barnabei VM, Cochrane BB, Aragaki AK, et al. Menopausal symptoms and treatment-related effects of estrogen and progestin in the Women
's Health Initiative. Obstet Gynecol. 2005;105(5):1063–1073.
9. National Institutes of Health. State-of-the-Science panel. National institutes of health state-of-the-science conference statement: Management of menopause
-related symptoms. Ann Intern Med. 2005;142(12):1003–1013.
10. Rossouw JE, Anderson GL, Prentice RL, et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal women
: Principal results from the women
's health initiative randomized controlled trial. JAMA. 2002;288:321–333.
11. Anderson GL, Limacher M, Assaf AR; WHI Steering Committee. Effects of conjugated equine estrogen in postmenopausal women
with hysterectomy: The women
's health initiative randomized controlled trial. JAMA. 2004;291:1701–1712.
12. The North American Menopause
Society. The 2012 hormone therapy position statement of the North American Menopause
13. The American College of Obstetricians and Gynecologists. Practice bulletin: Management of menopausal symptoms. Obstetrics Gynecol. 2014;123(1):202–216.
14. de Villiers TJ, Gass MLS, Haines CJ, et al. Global consensus statement on menopausal hormone therapy. Climacteric. 2013;16:203–204.
15. Goodman NE, Cobin RH, Ginzburg SB, Katz IA, Woode DE. American Association of Clinical Endocrinologists Medical Guidelines for clinical practice for the diagnosis and treatment of menopause
. Endocr Pract. 2011;17:949–954.
16. Moyer VA. Menopausal hormone therapy for the primary prevention of chronic conditions: U.S. preventive services task force recommendation statement. Ann Intern Med. 2013;158:47–54.
17. Shapiro S, de Villiers TJ, Pines A, et al. Risks and benefits of hormone therapy: Has medical dogma now been overturned? Climacteric. 2014;17(3):215–222.
18. Weissfeld JL, Liu W, Woods C, et al. Trends in oral and vaginally administered estrogen use among US women
50 years of age or older with commercial health insurance. Menopause
19. Sood R, Faubion SS, Kuhle CL, Thielen JM, Shuster LT. Prescribing menopausal hormone therapy: An evidence-based approach. Int J Womens Health. 2014;6:47–57.
20. Gerber MR, King MW, Pineless SL, et al. Hormone therapy use in women veterans
health administration care: A national cross-sectional study. J Gen Intern Med. 2015;30(2):169–175.
21. Schwartz EB. Capsule commentary on Gerber et al., hormone therapy use in women veterans
Health Administration Care: A National Cross-sectional Study. J Gen Intern Med. 2015;30(2):237.
22. Hayes PM. Leading the nation in women
's health: The important role of research. Womens Health Issues. 2011;21(4 suppl):S70–S72.
23. Yano EM, Haskell S, Hayes P. Delivery of gender-sensitive comprehensive primary care to women veterans
: Implications for VA patient Aligned care teams. J Gen Intern Med. 2015;29(suppl 2):703–707.
24. Frayne SM, Phibbs CS, Saechao F, et al. Sourcebook: Women Veterans
in the Veterans
Health Administration. Sociodemographics, Utilization, Costs of Care, and Health Profile. Vol 3. Washington, DC: Women
's Health Evaluation Initiative, Women
's Health Services, Veterans
Health Administration, Department of Veterans
25. Friedman SA, Phibbs CS, Schmitt SK, Hayes PM, Herrera L, Frayne SM. New women veterans
in the VHA: A longitudinal profile. Womens Health Issues. 2011;21(4 suppl):S103–S111.
26. Maisel NC, Haskell S, Hayes PM, et al. Readying the workforce: Evaluation of VHA's comprehensive women
's health primary care provider initiative. Med Care. 2015;53(1):S39–S46.
27. Bean-Mayberry B, Bastian L, Trentalange M, et al. Associations between provider designation and female-specific cancer screening in women veterans
. Med Care. 2015;53(4 suppl 1):s47–s54.
28. Bastian LA, Trentalange M, Murphy TE, et al. Association between women veterans
' experiences with VA outpatient health care and designation as a women
's health provider in primary care clinics. Womens Health Issues. 2014;24(6):605–612.
29. Gonsoulin M, Ramanathan D. VIReC Factbook: Corporate Data Warehouse (CDW) Consult 2.1. Hines, IL: U.S. Department of Veterans
Affairs, Health Services Research & Development Service, VA Information Resource Center; 2015.
31. Cordasco KM, Zuchowski J, Hamilton AB, Kirsh S, Veet L, Saavedra JO, Altman L, Knapp H, Canning M, Washington DL. Early lessons learned in implementing a women
's health educational and virtual consultation program in VA. Med Care. 2015;53(4 Suppl 1):S88–S92.
33. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–381.
34. Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc. 1997;4(5):342–355.
35. Romm FJ, Putnam SM. The validity of the medical record. Med Care. 1981;19(3):310–315.
36. Tang PC, Ralston M, Arrigotti MF, Qureshi L, Grahm J. Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: Implications for performance measures. J Am Med Inform Assoc. 2007;14(1):10–15.
38. Cordasco KM, Zuchowski JL, Hamilton AB, et al. Early lessons learned in implementing a women
's health educational and virtual consultation program in VA. Med Care. 2015;53(4 suppl 1):S88–S92.
39. Gray KE, Katon JG, Callegari LS, Cordasco KM, Zephyrin LC. Gynecologists in the VA: Do they enhance availability of sex-specific services and policies in the emergency department? Med Care. 2015;53:S76–S80.
40. Cabana MD, Rand CS, Powe NR, et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282(15):1458–1465.
41. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635–2645.
42. The North American Menopause
Society. The 2017 hormone therapy position statement of the North American Menopause
43. Gartlehner G, Patel SV, Feltner C, et al. Hormone therapy for the primary prevention of chronic conditions in postmenopausal women
: Evidence Report and Systematic Review for the US Preventive Services Task Force. JAMA. 2017;318(22):2234–2249.
44. Shaneyfelt TM. Building bridges to quality. JAMA. 2001;286(20):2600–2601.
45. Zuchowski JL, Hamilton AB, Washington DL, Gomez AG, Veet L, Cordasco KM. Drivers of continuing education learning preferences for veterans
's health primary care providers. J Contin Educ Health Prof. 2017;37(3):168–172.
46. Schwartz EB, Parisi SM, Handler SM, et al. Clinical decision support to promote safe prescribing to women
of reproductive age: A cluster-randomized trial. J Gen Intern Med. 2012;27(7):831–838.
47. Bright TJ, Wong A, Dhurjat R, et al. Effect of clinical decision-support systems: A systematic review. Ann Intern Med. 2012;157(1):29–43.
48. Shekelle P, Morton SC, Keeler EB. Costs and Benefits of Health Information Technology [internet] Rockville, MD: Agency for Healthcare Research and Quality (US); 2006. (Evidence Reports/Technology Assessments, No. 132.). https://www.ncbi.nlm.nih.gov/books/NBK37988/
. Accessed November 26th, 2018.
49. Kaminetzky CP, Nelson KM. In the office and in-between: The role of panel management in primary care. J Gen Intern Med. 2015;30(7):876–877.
50. Garg AX, Adhikari NKJ, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes. A systematic review. JAMA. 2005;293(10):1223–1238.
51. Black AD, Car J, Paglian C, et al. The impact of eHealth on the quality and safety of health care: A systematic review. PLoS Med. 2011;8(1):e1000387.
52. Filce GA, Drekonja DM, Thurn JR, et al. Use of a computer decision support system and antimicrobial therapy appropriateness. Infect Control Hosp Epidemiol. 2013;34(6):558–565.
53. Trivedi AN, Matula S, Miake-Lye I, Glassman PA, Skekelle P, Asch S. Systematic review: Comparison of the quality of medical care in veterans
affairs and non-veterans
affairs settings. Med Care. 2011;49:76–88.
54. Barnett MJ, Perry PJ, Langstaff JD, Kaboli PJ. Comparison of rates of potentially inappropriate medication use according to the Zhan criteria for VA versus private sector medicare HMOs. J Manag Care Spec Pharm. 2006;12(5):362–370.
55. Schiff GD, Amato MG, Eguale T, et al. Computerised physician order entry-related medication errors: Analysis of reported errors and vulnerability testing of current systems. BMJ Qual Saf. 2015;24:264–271.
56. Nanji KC, Slight SP, Segar DL, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(3):487–491.
57. Brunner J, Chuang E, Goldzweig C, Cain CL, Sugar C, Yano EM. User-centered design to improve clinical decision support in primary care. Int J Med Inform. 2017;10:56–64.
58. Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inf. 2005;38:75–87.
59. Rubenstein LV, Mittman BS, Yano EM, Mulrow CD. From understanding health care provider behavior to improving health care: The QUERI framework for quality improvement. Med Care. 2000;38(6):I129–I141.
60. Caminiti C, Scoditti U, Diodati F, Passalacqua R. How to promote, improve and test adherence to scientific evidence in clinical practice. BMC Health Serv Res. 2005;5:62.
61. Grol R, Grimshaw J. From best evidence to best practice: Effective implementation of change in patients' care. Lancet. 2003;362:1225–1230.
Kristina M. Cordasco, MD, MPH, MSHS, is a core investigator VA HSR&D Center for the Study of Healthcare Innovation, Implementation, and Policy; internal medicine physician with VA Greater Los Angeles Healthcare System; and associate clinical professor of Medicine at The University of California, Los Angeles (UCLA). Her research focuses on quality, care coordination, and organization of care.
Anita H. Yuan, is an investigator and Quantitative Sociologist with VA Health Services Research & Development (HSR&D) Center for the Study of Healthcare Innovation, Implementation, and Policy. She specializes in data management and analysis of surveys and electronic medical records.
Marjorie J. Danz, is a health services researcher with VA Greater Los Angeles Healthcare System and an associate scientist (Adjunct) with the RAND Corporation. Her research focuses on assessing and improving quality of care and quality of care reporting.
LaShawnta Jackson, is a research health scientist with VA Health Services Research & Development (HSR&D) Center for the Study of Healthcare Innovation, Implementation, and Policy. Her interests include social determinants of health, health equity, and implementation research.
Ellen F. Yee, is an internal medicine physician with the New Mexico VA Healthcare System, and professor of medicine at the University of New Mexico. Her interests include women's health care, cancer screening and prevention, and education.
Lueng Sophia Tcheung, is an internal medicine physician with VA Greater Los Angeles Healthcare System, and assistant clinical professor of medicine at the University of California, Los Angeles. Her interests include women's health care and quality measurement.
Donna L. Washington, is Women's Health Focused Research Area Lead at VA HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy, and professor of medicine at UCLA. Her research examines health care access, quality, and equity for women and vulnerable populations.