Secondary Logo

Psychiatric Nurse Practitioner Residents Improve Quality and Mental Health Outcomes for Veterans Through Measurement-Based Care

McGuinness, Teena M.; Richardson, Jessica W.; Nicholson, W. Chance; Carpenter, Jennifer; Cleveland, Cynthia; Rodney, Kanini Z.; Harper, Doreen C.

The Journal for Healthcare Quality (JHQ): March/April 2019 - Volume 41 - Issue 2 - p 118–124
doi: 10.1097/JHQ.0000000000000180
Original Article

Introduction: Measurement-based care (MBC) is commonly used to manage medical illness, whereas only about 20% of psychiatric care providers use MBC. One aim of the partnership between the Birmingham Veterans Affairs Medical Center (BVAMC) and the University of Alabama at Birmingham School of Nursing (UAB SON) is to provide MBC to Veterans. The goal is to describe the efficacy of MBC in the treatment of Veterans by psychiatric-mental health nurse practitioner (PMHNP) residents.

Purpose: By teaching PMHNP residents evidence-based assessment, they gain tools to shorten time to remission of depression, anxiety, and posttraumatic stress disorder.

Methods: Residents administered the Patient Stress Questionnaire (PSQ), an MBC tool that assesses depression, anxiety, trauma-related symptoms, and alcohol use, to patients of the Residency Continuity Clinic (RCC). Patient Stress Questionnaire scores from March 2016 to May 2018 were analyzed using paired t tests.

Results: Analysis revealed a downward trend in PSQ scores over time, suggesting improvement in psychiatric symptoms. Depressive and anxiety symptoms decreased significantly.

Conclusions/Implications: This quality project highlights the utility of MBC by PMHNP residents in an RCC. Measurement-based care tools can be quickly and easily administered with little impact on workflow. Use of validated screening tools can enhance care, engage patients, and improve patient outcomes.

For more information on this article, contact Teena M. McGuinness,

Supported in part by the Veterans Administration Nursing Academic Partnership between the Birmingham Veterans Affairs Medical Center and the University of Alabama at Birmingham School of Nursing. The information or content and conclusions are those of the authors and should not be construed as the official position or policy, nor should any endorsements be inferred by the Veterans Health Administration.

The authors declare no conflicts of interest.

Approval was granted at both the University of Alabama at Birmingham and the Birmingham VA Medical Center to conduct this quality improvement project.

Back to Top | Article Outline


Combat operations of the U.S. Armed Forces have been continuous since 2001 with over 1.9 million Veterans having served.1 Most return from deployment without significant psychiatric illness but others face a continuum of issues related to posttraumatic stress injury.2 Comorbidities include substance use disorders, depressive symptoms, and anxiety. If not addressed in a Veteran-centric manner, a cascade of other health problems, family stress, and vocational failure may result.3 In a partnership between the Birmingham Veterans Affairs Medical Center (BVAMC) and the University of Alabama at Birmingham School of Nursing (UAB SON), an innovative approach within the Veterans Administration Nursing Academy Partnership for Graduate Education (VANAP-GE) sought to improve mental healthcare for Veterans by teaching their residents who are post-MSN psychiatric-mental health nurse practitioners (PMHNPs) the skills of measurement-based care (MBC). The purpose of this study is to: (1) present MBC and its components (screening tools and their associated guidelines) and (2) present the results demonstrating MBC's clinical efficacy for Veterans served by the PMHNPs of the Residency Continuity Clinic (RCC).

Back to Top | Article Outline

Problem Description

Although most human illness is managed based on measurement, MBC has not been consistently used in settings where mental illness is treated.4 Furthermore, Lewis et al5 estimated that less than 20% of psychiatric care providers use symptom rating scales to evaluate symptoms and measure response to treatment. Historically, psychiatric treatment decisions have been guided by individual clinicians eliciting patients' self-reports of their symptoms. The lack of standardized measures and clinical practice guidelines leads to wide variabilities in practice, which are thought to be associated with poor outcomes for patients.4

Back to Top | Article Outline


Nearly all randomized controlled trials of the effectiveness of psychiatric care containing frequent feedback of patient-reported symptom severity to the mental health provider during clinical assessment demonstrate that recovery is quicker when compared with usual care.6 Indeed, the U.S. Department of Veterans Affairs7 has begun a nationwide effort to support the use of MBC and its associated clinical guidelines to improve care. Essentially, use of patient-reported symptom rating scales provide a rapid assessment of psychiatric symptoms such that treatment response is quantified for further decision making and recovery is faster.8 Due to the potential burden of mental illness, it is important to provide safe, timely, and effective mental health treatment. Measurement-based care provides a mechanism to accurately and efficiently assess the effectiveness of mental health treatment and improve patient outcomes.9 Furthermore, MBC improves the quality of patient care by engaging Veterans in their care and promoting shared decision making.6 Indeed, MBC is embraced as an important tool in addressing behavioral healthcare in the United States.6

The MBC processes taught in the VANAP-GE PMHNP Residency program guide practice and training experiences for graduate learners with Veteran populations receiving psychiatric-mental healthcare. These processes not only facilitate learning but also enable the graduate learner to measure outcomes. This is particularly helpful for new providers, such as psychiatric residents in medicine, nursing, and other disciplines, who are part of the team that delivers behavioral healthcare to Veterans, in that the care is standardized and the learner has a clinical guideline for an intervention that leads to higher quality care.9

Back to Top | Article Outline


The specific aims of this study are: (1) to present the intervention and its components (including relevant screening tools and their associated guidelines and (2) to present the results demonstrating MBC's clinical efficacy for Veterans served by PMHNP residents in the RCC.

Back to Top | Article Outline



In 2014, the Birmingham Veterans Affairs Medical Center (BVAMC) joined with the University of Alabama at Birmingham School of Nursing (UAB SON) to create the VANAP-GE and build a 1-year, post-MSN psychiatric nurse practitioner residency. An RCC was established for our residents to follow a panel of Veterans over the course of their 1-year residencies supervised by a VANAP-GE faculty member. The quality-improvement intervention for MBC began in 2016 and data were collected for 2.4 years; a total of nine residents from three different cohorts of PMHNP residents participated in the data collection.

Back to Top | Article Outline


A quality-improvement project using a quantitative approach (pre- and post-measures of commonly occurring psychiatric symptoms of Veterans) was planned for the PMHNP residents. These residents were introduced to the MBC measures at the beginning of their residency and were trained on how to administer, score, and interpret the measures. In turn, PMHNP residents explained the measures to Veterans and asked them if they would like to complete the questionnaires. Most patients complete the measures in 5–7 minutes. Residents asked each Veteran who attended the RCC to complete the Patient Stress Questionnaire (PSQ), a comprehensive two-page tool that addresses depression/anxiety symptoms, posttraumatic stress response, alcohol use, and one item for pain. Endorsed by the Substance Abuse and Mental Health Services Administration, the instrument is available from their website: Veterans were introduced to the PSQ with the explanation that using such an approach improves awareness of symptoms they may be experiencing to target symptoms that are most troublesome. Fortney et al10 proposed that use of rating scales during assessments helps patients validate their experiences, fosters hope, and boosts their adherence to a mutual treatment plan. Rating scales are valuable to clinicians in that the PMHNP can assess the efficacy of current treatment to consult the latest evidence-based guidelines for treatment of depression, anxiety, trauma, and substance use. Psychotropic medication guidelines are consulted as well as evidence-based psychotherapy guidelines. Unfortunately, although there is widespread use of the PSQ and its five subscales, there is no literature that cites the use of its combined subscales. Fortunately, there is much psychometric evidence for use of the individual tools, which is provided below.

Back to Top | Article Outline

Depression Measure: Patient Health Questionnaire-9

Used widely in primary care settings across the world, the Patient Health Questionnaire (PHQ)-9 has high sensitivity and specificity. In structured interviews of 580 patients, Kroenke et al11 found that the higher scores were associated with higher depressive symptoms (range for PHQ-9 scores is from 0 to 27); conversely, lower scores (5 or less) were reflective of little or no symptoms. Sensitivity and specificity were high at 88% for each.11

Back to Top | Article Outline

Anxiety Measure: Generalized Anxiety Disorder-7

Also a widely used measure, the Generalized Anxiety Disorder (GAD)-7 is a valid tool to assess for key symptoms of anxiety. Scores range from 0 to 21. The GAD-7 was found to be a valid tool based on use with a large sample (n = 985). Spitzer et al12 reported a sensitivity of 89% and specificity of 82%. Generalized Anxiety Disorder-7 scores of greater than 10 indicate a strong likelihood of GAD. Severe anxiety is indicated by a score of 15 or greater.

Back to Top | Article Outline

Posttraumatic Stress Disorder Measure: Primary Care Posttraumatic Stress Disorder Screen

Combat Veterans were the population used to validate the Primary Care Posttraumatic Stress Disorder (PC-PTSD), a four-item self-report measure assessing key target symptoms of PTSD. Bliese et al13 found that three positive responses indicated a sensitivity of 78% and specificity of 87%. Although a positive response to any of the four questions does not necessarily indicate that a patient may have PTSD, it does signal to the mental health clinician that the patient may be experiencing trauma-related symptoms warranting further investigation.14

Back to Top | Article Outline

Alcohol Use Measure: Alcohol Use Disorder Identification

Developed by the World Health Organization, the Alcohol Use Disorder Identification (AUDIT) is a self-report questionnaire to identify hazardous drinking behaviors.15 Both sensitive and specific, the AUDIT has been validated across genders and in a variety of ages and racial/ethnic groups.15 The 10-item self-report screening tool covers 3 symptoms areas: hazardous use, dependence symptoms, and harmful use.16 Research consistently confirms the validity of the AUDIT, with sensitivities and specificities comparable to or exceeding those of other alcohol screening tools.17 A cutoff score of eight has been widely accepted as an indicator of hazardous alcohol use.16

Back to Top | Article Outline

Pain Question

The PSQ has a final component, which is a one-item question that asks about the presence or absence of pain. Patients report only a yes or no.

Back to Top | Article Outline

Study of the Intervention (Approach)

Both PMHNP residents and VANAP-GE faculty associated with the RCC learned how to administer the PSQ and each of its scales. Interpretation of the scores was shared with each Veteran along with a discussion of pertinent evidence-based guidelines for additional treatment options. Patient Stress Questionnaire scores were recorded in the Veteran's electronic medical record at each visit. Some Veterans who have been attending the RCC clinic for 2 years have numerous data points, whereas others have scores from the first and second visits. Only the first and last scores of Veterans were included in the data set (Excel). Anonymized patient data included first and last visit PSQ scores, the number of psychotropic medications, and the class of psychotropic medications prescribed. Demographic information was also included along with whether the Veteran was participating in psychotherapy. Participation in psychotherapy was determined by reviewing the patient's record for notes from a psychologist or social worker who provides various evidence-based psychotherapies (whether trauma-focused, cognitive-behavioral, or group therapies). SPSS was used for data analysis.

Back to Top | Article Outline

Data Analysis

One hundred patients (N = 100) had at least two encounters with PSQ data in the chart. Only patients with at least two or more encounters were included in data analysis. The patient's first and last scores on the MBC tools were compared. The first score is the first documented PSQ score in the patient's chart. The last score is the patient's last, or most recent, documented PSQ score available in the chart. This could include the patient's second visit in the RCC or the patient's sixth, for example. From March 1, 2016, to May 1, 2018, the highest number of appointments for seen patients in the RCC with documented PSQ scores was 12. Descriptive statistics were performed to characterize the data. Paired t tests were used to determine differences between the first and last scores for each individual PSQ component.

Back to Top | Article Outline

Ethical Considerations

This project was deemed quality improvement by the BVAMC, and was both approved from review as a quality improvement project by the UAB Institutional Review Board and the BVAMC Quality Improvement Review Committee.

Back to Top | Article Outline


Demographics and Treatment Characteristics

A total of 9 PMHNP residents collected data on a total of 100 patients over the course of 2.4 years. This sample of RCC Veterans comprised 80% males. Ages of patients ranged from 22 to 86 years, with the average age being 52.4 years. African Americans comprised the majority of patients seen in the RCC (63%) and the remaining patients were Caucasians (36%) and Latinos (1%).

The most common diagnoses were depression (38%), anxiety (21%), and PTSD (19%). The majority of patients (64%) had more than one diagnosis. Depression and anxiety were the most commonly occurring diagnoses.

The first time PHQ-9 scores were recorded, 69% of Veterans scored 10 or higher, reflective of moderate to severe depression (higher scores reflecting more severe depressive symptoms). Similarly, according to the first GAD-7 scores recorded, 62% of Veterans scored a 10 or higher indicating moderate to severe anxiety (the higher the score, the more severe the anxiety). More than half of all Veterans (54%) had positive responses to three or more questions on the PC-PTSD measure, indicating that the Veteran was experiencing trauma-related symptoms.

According to the AUDIT scores (scored eight or higher), 13% of Veterans likely had an alcohol use disorder. The most frequently used substances were alcohol and cannabis. Veterans who reported using illegal substances (13%) named cannabis most frequently (8%).

Characterizations of patient treatment included four categories: (1) medication only (no psychotherapy), (2) psychotherapy only (no medication), (3) both medication and psychotherapy, and (4) or neither psychotherapy nor medication. The majority of patients (61%) in the RCC received both medication and psychotherapy and 37% of patients received medication only (no psychotherapy). Of those patients receiving medication (n = 98), 61% received a combination of one or more psychotropics (such as sertraline and trazodone, for example). The most commonly prescribed psychotropics were selective serotonin reuptake inhibitors.

Back to Top | Article Outline

Comparison of Patient Stress Questionnaire Scores

Comparison of PSQ scores between first and last scores for the RCC patients (N = 100) demonstrated a downward trend in all PSQ scores, indicating clinical improvement in all symptom measures (Table 1). Significant differences were found for the PHQ-9 and GAD-7 (p ≤ .001 for both tools). Significance was also found for the PC-PTSD (p = .022). Although not statistically significant, the AUDIT demonstrated a downward trend in scores over time (in the expected direction of improvement).

Table 1

Table 1

Back to Top | Article Outline


This project had limitations, which included the lack of a comparison group, because the RCC clinic is limited in size with a relatively small sample size. Furthermore, the project used a convenience sample of patients seen in the RCC. In addition, the defined time frame between the “first” and “last” PSQ scores is wide ranging and may not have captured all patients' response to treatment. For example, some patients may have two encounters, whereas others have multiple encounters. For those patients with fewer encounters, there might not have been long enough time between the first and last scores to capture a complete response to treatment. Furthermore, time between visits is not standardized because of Veteran work schedules or transportation challenges. As a result, some patients return for a second visit after 1 month, whereas others may return for a second visit after 3 months. Finally, the sample indiscriminately included those receiving no treatment, only psychopharmacology, only psychotherapy, and those receiving combined approaches. Evidence suggests that persons engaging in combined psychotherapy and psychopharmacology treatment have better treatment outcomes. Given this, future research studies in the RCC will focus on examining whether or not these treatment distinctions inform differences in PSQ scores.

Back to Top | Article Outline


The outcomes of this quality improvement project highlight the importance of MBC in improving the quality of psychiatric care and also as a training strategy for PMHNP residents in behavioral health. In examining initial to last PSQ scores, statistically and clinically significant decreases in the means were noted on the PHQ-9, GAD-7, and PC-PTSD scores, indicating an improvement in psychiatric symptoms and progress toward recovery.

Veterans served in the RCC resembled national estimates of alcohol use among Veterans with approximately 13% endorsing alcohol use. According to national estimates of Veterans presenting for first time care within the VA Health System, approximately 10.5% of men and 4.8% of women had an alcohol use disorder and 4.8% of men and 2.4% of women had a drug use disorder.18

Although the AUDIT scores decreased over time, the results were not statistically significant. One possible influence for this finding is that most patients did not report alcohol use. When looking at the entire RCC sample, the mean on the AUDIT first score was 3.17 and the mean on the AUDIT last score was 2.79. A score of eight or more on the AUDIT is indicative of hazardous drinking behaviors.14

Measurement-based care may help to elicit unreported symptoms and inform care. For example, if the patient did not report symptoms of PTSD during the interview, but endorsed symptoms of PTSD on the PC-PTSD, then the PMHNP resident could use this as an opportunity to explore PTSD symptoms and treat accordingly. Furthermore, the PMHNP residents evaluate scores over time to assess patient response to treatment. Lower scores indicate an improvement in patient-reported symptomology and an increase in scores can signal the provider to reevaluate the current treatment plan. Measurement-based care, in conjunction with clinical practice guidelines, provides a method to objectively monitor patient response to treatment and guides a plan of care.9 However, it is important to note that although MBC provides a systematic method to track patient progress and inform treatment decisions, it does not replace the provider's clinical judgment.

Back to Top | Article Outline


The use of MBC in the RCC enhanced the continuity of care among providers. For example, in one PMHNP resident's absence, another resident could assume care for the Veteran in a structured and systematic manner. The use of the PSQ to gauge symptom response to treatment reduced variability and standardized resident's assessment. Furthermore, implementation of MBC in the RCC improves the quality of care by ensuring that each Veteran is screened for common psychiatric illnesses.

Back to Top | Article Outline


This project demonstrates the clinical utility of MBC in the psychiatric setting. By educating our PMHNP residents in the use of MBC, they carry the message to other clinicians who may not be using MBC on each patient at each encounter. In interprofessional trainings, PMHNP residents share the importance of MBC and how it has become a crucial clinical skill, thereby informing other trainees regarding its utility. For innovations in clinical practice across healthcare disciplines to be adopted, the changes “must be effective, applicable to a large population, cost neutral, positively correlated with patient satisfaction, and relatively straightforward to implement.”19 Indeed, that is the essence of MBC.

The outcomes achieved by the RCC have potential for replicability at other VA facilities. The MBC tool required minimal disruption to workflow and involved little to no risk to Veterans. The use of MBC in the RCC enhanced the continuity of care among providers and improved the quality of care provided to Veterans. Through the standardization of care, MBC engaged Veterans in their care, quantified their symptoms, and provided a structured approach to treatment of symptoms. This project demonstrates that MBC increases the likelihood that patients will experience remission and recovery, giving hope to Veterans who experience these chronic disorders.

Back to Top | Article Outline


1. Institute of Medicine, Board on the Health of Select Populations; Committee on the Initial Assessment of Readjustment Needs of Military Personnel, Veterans, and Their Families. Returning Home From Iraq and Afghanistan: Preliminary Assessment of Readjustment Needs of Veterans, Service Members, and Their Families. Washington, DC: The National Academies Press; 2010. Accessed May 15, 2018.
2. Jaffee MS, Meyer KS. A brief overview of traumatic brain injury (TBI) and post-traumatic stress disorder (PTSD) within the department of defense. Clin Neuopsychol. 2009;23(8):1291–1298.
3. Tanielian T, Batka C, Meredith L. Bridging Gaps in Mental Health Care: Lessons Learned from the Welcome Back Veterans Initiative. Santa Monica, CA: RAND Corporation; 2017. Accessed May 15, 2018.
4. Harding KJ, Rush J, Arbuckle M, Trivedi MH, Pincus HA. Measurement-based care in psychiatric practice: A policy framework for implementation. J Clin Psychiatry. 2011;72(8):1136–1143.
5. Lewis CC, Scott K, Marti CN, et al. Implementing measurement-based care (iMBC) for depression in community mental health: A dynamic cluster randomized trial study protocol. Implementation Sci. 2017;10(1):127–141.
6. Kennedy Forum. Fixing Behavioral Healthcare in America: A National Call for Measurement-based Care in the Delivery of Behavioral Health Services. The Kennedy Forum; 2015. Accessed May 15, 2018.
7. Affairs USDoV. Measurement-based Care in Mental Health Initiative; 2016. Accessed May 21, 2018.
8. Scott K, Lewis CC. Using measurement-based care to enhance any treatment. Cogn Behav Pract. 2015;22(1):49–59.
9. Rush AJ. Isn't it about time to employ measurement-based care in practice? Am J Psychiatry. 2015;172(10):934–936.
10. Fortney JC, Unutzer J, Wrenn G, et al. A tipping point for measurement-based care. Psychiatr Serv. 2017;68(2):179–188.
11. Kroenke K, Spitzer R, Williams JB. The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–613.
12. Spitzer RL, Kroenke K, Williams JW, Löwe B. A brief measure for assessing generalized anxiety disorder: The gad-7. Arch Intern Med. 2006;166(10):1092–1097.
13. Bliese PD, Wright KM, Adler AB, Cabrera O, Castro CA, Hoge CW. Validating the primary care posttraumatic stress disorder screen and the posttraumatic stress disorder checklist with soldiers returning from combat. J Consult Clin Psychol. 2008;76(2):272–281.
14. Administration SAMHS. Primary Care PTSD Screen (PC-PTSD); 2018. Accessed May 15, 2018.
15. Babor TF, Higgins-Biddle J, Saunders JB, Monteiro MB, World Health Organization. AUDIT: The Alcohol Use Disorders Identification Test: Guidelines for Use in Primary Health Care. Dependence DoMHaS, ed. 2nd ed. Geneva, Switzerland: World Health Organization; 2001. Accessed May 15, 2018.
16. Lundin A, Hallgren M, Balliu B, Forsell Y. The use of alcohol use disorders identification test (AUDIT) in detecting alcohol use disorder and risk drinking in the general population: Validation of AUDIT using schedules for clinical assessment in neuropsychiatry. Alcohol Clin Exp Res. 2015;39(1):158–165.
17. Reinert DF, Allen JP. The alcohol use disorders identification test: An update of research findings. Alcohol Clin Exp Res. 2007;31(2):185–199.
18. Teeters JB, Lancaster CL, Brown DG, Back SE. Substance use disorders in military Veterans: Prevalence and treatment challenges. Subst Abuse Rehabil. 2017;8:69–77.
19. Black WE, Nagarkatti-Gude DR, Jetmalani A, Keepers G. Harnessing technology to implement measurement-based care. Acad Psychiatry. 2018;42(5):711–716.
Back to Top | Article Outline

Authors' Biographies

Teena M. McGuinness, PhD, CRNP, FAANP, FAAN, is a professor and codirector of the VANAP-GE Mental Health Nurse Practitioner Residency in Birmingham, AL.

Jessica W. Richardson, DNP, CRNP, PMHNP-BC, is an instructor and codirector of the VANAP-GE Mental Health Nurse Practitioner Residency in Birmingham, AL.

W. Chance Nicholson, PhD(c) CRNP, PMHNP-BC, is an instructor with the VANAP-GE Joint Mental Health Nurse Practitioner Residency in Birmingham, AL.

Jennifer Carpenter, MSN, CRNP, PMHNP-BC, is an instructor with the VANAP-GE Mental Health Nurse Practitioner Residency in Birmingham, AL.

Cynthia Cleveland, DNP, RN, NE-BC, is the associate director for Patient Care Services at the Birmingham VA Medical Center.

Kanini Z. Rodney, MPH, MBA, MD, FACP, is the Women's Health Medical Director and Acting Chief of Mental Health at the Birmingham VA Medical Center and the VISN 7 Women's Health Clinical Consultant.

Doreen C. Harper, PhD, RN, FAAN, is the dean and Fay B. Ireland Endowed Chair in Nursing at the University of Alabama at Birmingham School of Nursing.


measurement-based care, psychiatric care; Veterans

© 2019 National Association for Healthcare Quality