Secondary Logo

Journal Logo

Articles

Evaluating Clinician Attitudes After Local Implementation of the Veterans Affairs Predictive Analytic Model for Suicide Prevention

Piccirillo, Marilyn L. PhD; Pruitt, Larry D. PhD; Reger, Mark A. PhD

Author Information
Journal of Psychiatric Practice: January 2022 - Volume 28 - Issue 1 - p 14-23
doi: 10.1097/PRA.0000000000000595
  • Free

Abstract

Current evidence suggests that Veterans’ suicide rates are 1.5 times those of non-Veterans.1 Accordingly, suicide prevention has become a critical priority for the Veterans Health Administration (VHA), and VHA has implemented a comprehensive suicide prevention strategy.2 One of the initiatives of this strategy is called the Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment (REACH VET) program, which was launched in 2017. REACH VET is designed to improve identification of risk and access to care for Veterans who may be at elevated risk for suicide or other adverse outcomes.3 The REACH VET program uses a predictive analytic model constructed from >60 variables present in the Veteran’s electronic medical record to calculate suicide risk on a monthly basis for each Veteran who received care in the last 2 years at a given facility. Veterans in the top 0.1% of the risk tier are identified by the program. The REACH VET coordinator for each local facility performs a chart review of each identified Veteran’s electronic health record to systematically identify a provider familiar to the Veteran. The coordinator asks this provider to conduct a comprehensive evaluation of the Veteran’s care via chart review and reach out to the identified Veteran to discuss their risk, the REACH VET program, and any recommended treatment changes or additions (eg, appointment frequency, additional safety planning, enrollment in the Caring Letters program).4 These efforts to enhance care for the Veteran are then documented in the Veteran’s electronic health record. Inclusion in the REACH VET program is associated with having more outpatient encounters, more safety plans, and fewer inpatient mental health admissions, emergency department visits, and documented suicide attempts.5

Since the national roll-out of the REACH VET program,6 concerns have been expressed regarding the utility of predictive analytic tools for suicide prevention7,8 and for the social sciences more broadly.9 Researchers have long noted clinician-specific barriers that may interfere with the successful implementation and continuation of predictive analytic programs like REACH VET. For example, clinicians may lack familiarity with the purpose of the program or their responsibilities. Negative attitudes about predictive analytic models may result in the lack of adherence to the clinical guidelines of REACH VET. Furthermore, as employees of a government agency, clinicians may experience unique challenges associated with implementing new policies in addition to other administrative tasks. Finally, decreased staffing, insufficient time allotments for new tasks, or perceived increases in liability may pose structural barriers to successful implementation. When considering the logistical and emotional effort involved in caring for high-risk patients, these structural factors should not be underestimated.10

Previous research has evaluated clinician attitudes toward algorithmic-based care more broadly. Researchers found that physicians did not report much behavior change in response to an algorithm used to determine medication treatment for individuals with schizophrenia and other serious mental illnesses.11 Results from a qualitative study suggested that clinicians felt that algorithms would limit their agency in prescribing medications and would interfere with providing individualized, patient-centered care; responses also cited environmental barriers, such as limited time and resources.12 These studies underscore the importance of evaluating clinician-identified barriers to sustained implementation.

OBJECTIVE

The REACH VET program was implemented under the direction of the Under Secretary of Veterans Affairs for Health. Although thoughtful efforts were taken to consider clinician needs in the initial implementation at the local level,6 clinician input and feedback have not been evaluated since. To evaluate the early implementation of the REACH VET program within our facility and to identify areas of improvement to facilitate program sustainment efforts, we used a mixed-methods approach to evaluate clinician attitudes towards the REACH VET program following initial implementation. We aimed to identify clinician-specific barriers to the sustainment of REACH VET and provide recommendations for how identified barriers could be resolved at the facility level.

METHODS

Participants

A national web-based dashboard application used to manage REACH VET cases was reviewed, and all clinicians in a large Veterans Affairs (VA) health care system who were identified as a REACH VET provider for at least 1 Veteran in the preceding year were emailed an invitation to participate in this evaluation. To preserve the confidentiality of survey responses, demographic characteristics of those who completed the survey were not assessed. However, the recruitment sample included clinicians and trainees across medical (n=21), psychology (n=20), social work (n=28), and other affiliated disciplines (eg, nursing, n=5). A majority of the recruitment sample worked in outpatient mental health clinics (n=39), although some worked in community-based care (eg, VA Supported Housing Program, n=6), telemental health (n=14), or short-term care settings (n=10). Of the 74 providers who were contacted, 35 (47%) completed the anonymous self-report survey, and 12 completed the interview. (There was 1 clinician who participated in the qualitative interview who was not invited to participate in the anonymous survey; this was likely due to an oversight in the recruitment process for the anonymous survey.) Most clinicians who completed the interview were psychologists (n=5) and/or worked in outpatient mental health clinics (n=8), and 1 served in a REACH VET coordinator position. This REACH VET coordinator served on the local facility suicide prevention team and shared responsibility for local suicide prevention efforts and coordinating duties, including providing REACH VET trainings, assigning REACH VET providers, and working with individual clinicians to complete REACH VET duties.

Measures

Self-report Survey

The anonymous self-report survey was a 17-item measure designed by the first and third authors to assess clinician attitudes about the REACH VET program. All 17 items used a 5-point Likert scale ranging from 1 (Strongly disagree) to 5 (Strongly agree) and were administered via an online survey. Items addressed themes including knowledge about the REACH VET program and beliefs regarding the utility of the program (Table 1).

TABLE 1 - Descriptive Statistics for Survey Data (N=35)
n (%)
Mean (SD) Strongly Disagree Disagree Neutral Agree Strongly Agree
REACH VET
 Significantly increases my workload 3.26 (1.04) 1 (2.9) 10 (28.6) 5 (14.3) 17 (48.6) 2 (5.7)
 Validates my preexisting concerns regarding my patient 3.20 (1.11) 3 (8.6) 6 (17.1) 10 (28.6) 13 (37.1) 3 (8.6)
 Significantly increases my ethical responsibility 2.71 (1.05) 5 (14.3) 9 (25.7) 13 (37.1) 7 (20.0) 1 (2.9)
 Significantly increases my liability as a provider 3.34 (1.19) 3 (8.6) 5 (14.3) 10 (28.6) 11 (31.4) 6 (17.1)
 Provides me with practical recommendations regarding my patient 1.83 (0.71) 12 (34.3) 17 (48.6) 6 (17.1) 0 (0.0) 0 (0.0)
 Identifies those who are “silent sufferers” at risk for suicide 2.14 (0.85) 7 (20.0) 19 (54.3) 6 (17.1) 3 (8.6) 0 (0.0)
 My responsibilities as a REACH VET provider are clear 2.71 (1.05) 3 (8.6) 15 (42.9) 7 (20.0) 9 (25.7) 1 (2.9)
 The risk factors identified for my Veteran on the REACH VET dashboard are clear 2.89 (1.13) 4 (11.4) 10 (28.6) 9 (25.7) 10 (28.6) 2 (5.7)
 My responsibilities as a REACH VET provider are congruent with my values as a health care provider 3.31 (1.13) 2 (5.7) 9 (25.7) 3 (8.6) 18 (51.4) 3 (8.6)
 REACH VET has the potential to make a difference in the quality of care a Veteran receives 2.94 (1.03) 4 (11.4) 6 (17.1) 14 (40.0) 10 (28.6) 1 (2.9)
 The REACH VET program fills a need within the broader VA system 2.74 (1.01) 4 (11.4) 10 (28.6) 13 (37.1) 7 (20.0) 1 (2.9)
 The REACH VET algorithm identifies those in need of additional services 2.74 (1.04) 4 (11.4) 11 (31.4) 11 (31.4) 8 (22.9) 1 (2.9)
 Having another person review my clinical documentation as part of the REACH VET program would be helpful 2.89 (0.93) 2 (5.7) 10 (28.6) 14 (40.0) 8 (22.9) 1 (2.9)
 The REACH VET program identifies the appropriate providers to assist REACH VET patients 2.77 (1.09) 5 (14.3) 10 (28.6) 8 (22.9) 12 (34.3) 0 (0.0)
 I would appreciate additional training on REACH VET duties* 2.85 (1.08) 3 (8.8) 12 (35.3) 7 (20.6) 11 (32.4) 1 (2.9)
 The time I invest in REACH VET duties is well spent 2.09 (0.85) 9 (25.7) 16 (45.7) 8 (22.9) 2 (5.7) 0 (0.0)
 I am aware of resources (eg, local information, national websites) available for REACH VET providers 2.37 (0.97) 6 (17.1) 16 (45.7) 7 (20.0) 6 (17.1) 0 (0.0)
*Only 34 respondents answered this item.
REACH VET indicates Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment; VA, Veterans Affairs.

Qualitative Interview

The qualitative interview was an 11-item semistructured interview designed to assess clinician attitudes towards the REACH VET program. The interview was developed with consultation from an implementation scientist, Sara Landes, PhD. Table 2 displays the coding framework for the qualitative interviews.

TABLE 2 - Coding Framework Used to Analyze Qualitative Interviews (N=12)
Final Coding Framework Initial Coding Framework
1. Number of Veterans served by REACH VET provider (ie, clinician) Few Veterans (<5) Some Veterans (5-10) Many Veterans (>10)
2. Nature of relationship between clinician and Veteran identified by REACH VET Established relationship and regular communication with REACH VET Veteran Established relationship and infrequent, but clinically indicated, contact with REACH VET Veteran Minimal established relationship and minimal communication with REACH VET Veteran No established relationship with REACH VET Veteran
3. Awareness of the REACH VET program and web-based platform Clinician not aware of web-based platform Clinician aware of web-based platform, but hasn’t accessed it Clinician not interested in more information on REACH VET algorithm Clinician interested in more information on REACH VET algorithm Clinician wants more education/training about REACH VET program, outcomes
4. Clinician experience of notifying Veterans of REACH VET status Clinician did not notify Veteran Clinician notified some Veterans Clinician notified all Veterans Clinician was not able to notify Veteran (eg, Veteran left treatment) Clinician did not see relevance or utility in notifying Veteran Clinician anticipated difficulty with notifying Veteran (eg, might be perceived as intrusive) Clinician did not feel comfortable notifying Veteran due to lack of knowledge of program Veteran responded with neutral or positive response
5. Change in care as a result of the REACH VET program No change in care provided Increased time and effort for clinician (eg, chart review) Increased worry for clinician Increased outreach from clinician if Veteran no-shows appointments
6. Clinician attitudes towards REACH VET responsibilities Clinician understood how to provide and document outreach Clinician wanted more direction on how frequently to provide outreach Clinician did not feel that they were the appropriate provider identified for the Veteran
Final Coding Framework Initial Coding Framework
7. Clinician attitudes towards REACH VET documentation REACH VET documentation did not increase burden for clinician Clinician wanted feedback on their documentation Clinician wanted to know when Veteran was identified to document changes in risk Clinician saw need to modify documentation to decrease liability
8. Clinician attitudes towards REACH VET program Clinician did not think REACH VET will improve Veteran health Clinician did think REACH VET will improve Veteran health Clinician desired more validation for efforts Clinician connected with values of suicide prevention, extending access to health care Clinician believed that REACH VET does not identify Veterans in need Clinician did not see value in algorithmic identification of high-risk Veterans Clinician desired information about Veteran preferences Clinician desired data on REACH VET outcomes
9. Clinician attitudes towards implementation of REACH VET program Clinician concerned that REACH VET is not capturing risk for homeless Veterans or those receiving care outside VA Clinician perceived that Veteran is already engaged in care Clinician perceived support from their team/clinic for assessing and treating high-risk Veterans Clinician believed that REACH VET does not address psychosocial needs Clinician desired information about length of time Veteran was followed by REACH VET program
REACH VET indicates Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment; VA, Veterans Affairs.

Procedure

Eligible clinicians were emailed 2 separate invitations to participate in the online survey and qualitative interviews. Interviews took place either in person or via telephone according to the clinician’s availability and lasted ∼25 minutes. Although informed consent was not required, as this study did not constitute human subjects research, clinicians were provided details about the purpose of the study and informed that they could end their participation in the study at any time. Clinicians were not compensated for their participation in either portion of the evaluation. This quality improvement project was reviewed jointly by the local Human Research Protection Program (HRPP) and Quality, Safety, & Value service line and was determined not to constitute human subjects research. [Projects with the primary goal of program evaluation are classified as quality improvement within the VA system. These projects are separate from (human subjects) research, which is conducted for the purpose of contributing to generalizable knowledge.]

Data Analysis

Descriptive analyses were used to examine trends in the clinicians’ responses to the anonymous survey using R software.13 Qualitative analysis codes were developed using themes from the quantitative survey, as well as by identifying novel themes that emerged from the qualitative interviews themselves. Codes were reviewed, collated, analyzed, and interpreted in keeping with established practices for qualitative analysis.14

SURVEY RESULTS

Overall, results from the anonymous survey demonstrated largely neutral to negative reactions to the REACH VET program (Table 1). Just over half of the respondents (51.5%) reported that they did not feel their responsibilities as a REACH VET clinician were clear, and 40% did not find that the risk factors identified on the national web-based platform were clear. Most reported that they were not aware of national resources for REACH VET (62.8%), but a substantial minority did not want additional training (44.1%). Over half of the clinicians (54.3%) reported that they perceived REACH VET as having significantly increased their workload, and almost half of the clinicians (48.5%) reported that REACH VET increased their perceived liability as a clinician. Forty percent of the clinicians reported that REACH VET did not significantly increase their ethical responsibility in regard to respect for Veteran’s confidentiality and autonomy, although a nearly equal number (37.1%) provided a neutral response to this question.

Overwhelmingly, clinicians did not appear to agree with the guidelines or purpose of the REACH VET program. For example, although almost half of the clinicians (45.7%) reported that the REACH VET program validated their preexisting concerns about the identified Veteran’s risk, three quarters (74.3%) of the respondents also reported that the REACH VET program did not identify Veterans who were previously unknown to the health care system (ie, silent sufferers), and 42.8% felt that the program did not identify Veterans in need of additional resources. Accordingly, most clinicians (71.4%) reported that their time completing REACH VET duties did not feel well-spent, and 42.9% felt that the REACH VET coordinators did not identify the right clinicians for a given Veteran. The providers also felt that they were not given practical recommendations regarding outreach (82.9%).

However, clinicians perceived potential for the REACH VET program to engage high-risk Veterans in care. Approximately 60% of clinicians surveyed indicated neutral to strong agreement regarding whether REACH VET filled a health care need (37.1% were neutral on this item, and 22.9 agreed or strongly agreed). Clinicians were more divided on whether REACH VET had the potential to improve the quality of care Veterans received: most were neutral (40.0%), although some (31.5%) reported that this program could improve quality of care. Finally, clinicians agreed that their responsibilities as a REACH VET provider were consistent with their values as a health care clinician (60.0%).

CLINICIAN INTERVIEW RESULTS

Results from the qualitative interview largely echoed the results of the survey (Table 2). Half (n=6) of the 12 clinicians interviewed had limited experience with REACH VET, serving as the clinician for <5 Veterans. In contrast to the survey results, a majority (n=9) had established relationships with the identified Veterans and were meeting at a frequency that was clinically indicated for their role. There were 7 clinicians who were not aware of the web-based REACH VET dashboard application used for the management of REACH VET cases, while 4 of the clinicians were aware of the dashboard but had not accessed it. Similarly, a majority of the clinicians (n=8), including the REACH VET coordinator, were interested in learning more about the REACH VET algorithm (eg, variables included in the algorithm, weighting of these variables, predictive validity of the algorithm).

Notifying Veterans of REACH VET Status

Six clinicians who notified the Veteran that they had been identified by the REACH VET program reported receiving mostly neutral responses. Other clinicians who were interviewed either reported that they had not been able to contact the Veteran or had chosen not to disclose the Veteran’s REACH VET status due to either a lack of knowledge about the REACH VET program or because discussing the Veteran’s REACH VET status “could have felt shaming.” Only 1 clinician reported discomfort specific to the predictive model approach.

Changes in Veteran Care and Workload

Of the 12 clinicians, 10 reported that they had not significantly changed their care as a result of the REACH VET program, although 1 noted that, because of the Veteran’s risk status, “I will now go an extra step to contact [the Veteran identified by REACH VET] if they’re a no-show.” Eight clinicians reported that they did not perceive the amount of documentation associated with REACH VET outreach as significantly increasing their workload. However, 1 clinician working as an intake coordinator in a short-term care setting reported feeling that the documentation might increase liability because of not being able to adequately demonstrate long-term enhanced care when working in a short-term care role. This clinician reported a high emotional burden associated with the additional time spent modifying documentation to adequately describe the limitations of that clinical position. The REACH VET coordinator did not report receiving any concerns from clinicians regarding notifying Veterans of REACH VET status or with documentation.

Attitudes Towards REACH VET Provider Role

A majority of clinicians (n=10) reported that they perceived themselves as competent in carrying out REACH VET duties (eg, performing chart review and outreach). This finding was echoed by the REACH VET coordinator who reported that the “providers want to do well.” However, 1 clinician affiliated with the VA Supported Housing Program reported feeling additional pressure to increase outreach, given the ability to meet with Veterans in the community. Others expressed confusion with being identified as a REACH VET provider for a Veteran when they did not have an established relationship stating, “I’m not meeting frequently or long enough … to carry out these roles” and, “it seemed like there were other clinicians who had longer and more established relationships to leverage.”

Perceived Potential of REACH VET Program

Overall, most clinicians emphasized that they valued suicide prevention efforts and working with the REACH VET coordinators, despite expressing mixed attitudes towards the REACH VET program. Some clinicians emphasized concerns with the meta-message of the REACH VET program, “[I] don’t like the assumption that an algorithm can do my job better than me” and, “this [represents] a disconnect between research and clinical work.” A few clinicians requested more information on REACH VET outcomes, “I want to see the data [on outcomes of REACH VET]” explaining, “I could see how this program feels like checking a box … if there are no obvious red flags.” Another clinician reported, “People identified seem high risk on paper, but face-to-face it’s different because they deny suicidality or have strong protective factors.” In contrast, the REACH VET coordinator acknowledged similar barriers to implementation while also sharing a clinical anecdote and stating, “this [program] adds more work [for us] … but [REACH VET] can really make a difference.”

DISCUSSION

The goal of this quality improvement study was to assess clinician attitudes toward the REACH VET program in our local health care system. Overall, results from the anonymous survey demonstrated largely neutral to negative attitudes toward the local implementation of REACH VET. For example, providers reported that most Veterans identified by REACH VET were already perceived as being engaged in care, and most clinicians did not feel that their time invested in REACH VET duties was well-spent. Clinicians interviewed cited a lack of added value (“we already know this person is high risk”) and pessimism with additional outreach (“the factors that led to them being identified are unchangeable”). However, clinicians also perceived potential for REACH VET to improve the quality of care for high-risk Veterans.

The results demonstrated a need for improved communication between local REACH VET coordinators and clinicians. REACH VET coordinators serve a critical and multifaceted role within the REACH VET program. Primarily, they are responsible for disseminating information regarding the national initiative to individual clinicians, typically via training sessions at the local facility. They are also responsible for identifying, selecting, and assigning a REACH VET provider once a Veteran has been identified and working with the clinician to identify potential outreach efforts, answer questions about the program, and facilitate completion of REACH VET duties (ie, documentation in the Veteran’s chart). Therefore, REACH VET coordinators serve a critical role in the implementation process, communicating leadership directives to individual clinicians.

Improved communication efforts between local REACH VET coordinators and clinicians may include more clearly outlined responsibilities and expectations, as well as providing information about existing VA resources to address psychosocial factors, such as homelessness. Clinicians also raised concerns about whether the program was identifying the clinician with the best likelihood of engaging the Veteran. To address this concern, it may be helpful for local REACH VET coordinators to spend more time explaining the process of identifying and selecting a REACH VET provider during regular trainings. For example, clinicians may not be aware that REACH VET coordinators are instructed to prioritize mental health versus medical providers when they identify and select a REACH VET provider. Finally, providers requested more validation from REACH VET coordinators and local leadership for their efforts in providing care to the highest risk Veterans. Building in more structured validation of clinician efforts during clinic-wide trainings and individual interactions with REACH VET coordinators would likely address this need and improve the local implementation of the REACH VET program.

Evaluating these findings within the framework of existing implementation models can help to illustrate areas of successful local implementation, as well as strategic next steps in the implementation process. For example, after the national roll-out, the local facility conducted careful planning, training, and coordination among all affected staff in keeping with suggestions for early-stage implementation. Since then, the facility’s REACH VET program coordinators have been reassigned to a team of staff involved with facility-wide suicide prevention efforts. Therefore, this evaluation represents a critical effort to further ongoing implementation efforts, specifically to observe the implementation process and to provide feedback to stakeholders involved in the program at the ground level (eg, REACH VET coordinators, individual clinicians). Thus, results of this evaluation have been shared with this local suicide prevention team and clinicians to further feedback loops regarding implementation.

During this feedback consultation with the REACH VET coordinating team, there was agreement that more frequent training and education with clinicians on the REACH VET program would improve messaging about the REACH VET program and communication between REACH VET coordinators and clinicians. Information about the algorithm and national program resources will be shared during regular trainings as this information may help to address misunderstandings regarding predictive models, such as why some Veterans did not appear to have “… obvious red flags” for suicidality. This would also be an opportunity to educate local clinicians about a recent national REACH VET evaluation that showed that REACH VET implementation was associated with fewer Veteran suicide attempts, inpatient mental health admissions, and emergency department visits.5 In addition, members of the suicide prevention team (ie, REACH VET coordinators) acknowledged clinicians’ need for increased validation for their efforts to care for the most at-risk Veterans. To minimize clinician burden, these trainings will be held during existing clinic staff meetings and facilitated by the local REACH VET coordinators. This will increase the visibility of the coordinators and provide opportunities for the coordinators to review the purpose, intended value, and outcomes of the REACH VET program. In addition to consulting with the suicide prevention team, 2 information sessions have been held to present these results to clinicians and local leadership.

Overall, educational interventions represent an important first step toward addressing the concerns raised by clinicians in this evaluation and sustaining local implementation; however, more targeted interventions may also be necessary in later stages of intervention. For example, a review of the clinical characteristics of local Veterans identified by the REACH VET algorithm would be useful in determining whether REACH VET identifies Veterans who are already engaged in clinically indicated levels of care. A separate local quality improvement study is already underway to evaluate these characteristics. It may also be useful to review the local standard operating procedure that is used to identify REACH VET providers to determine whether modifications are needed to prioritize the clinician with the most established relationship with the Veteran. For example, some Veterans who have just been discharged from an inpatient unit may not yet be fully established with an ongoing provider. It may also be useful to consider revising the electronic medical records documentation template to address differences in clinician roles (eg, acute care vs. longer term care) as this impacts the scope of practice. Finally, employing a dedicated REACH VET coordinator (rather than a team) may decrease communication challenges and facilitate opportunities for improved consultation with clinicians. Likewise, a mandatory or standardized training developed for REACH VET may be useful as an annual requirement for VA clinicians.

The results presented here should be considered in the context of the project’s limitations. First, we recruited a fairly small sample, and it is likely that selection biases were present among the clinicians who responded to the evaluation invitations. For example, those who completed the surveys may have held strong opinions about the program that led them to use their unscheduled time to complete the survey or interview, or they may have had more time available to participate compared with other providers. Furthermore, many of the providers interviewed had worked with <10 Veterans in the program. Therefore, these results may not reflect the attitudes of more experienced REACH VET providers. Second, both of the assessment tools used were created specifically for this quality improvement project, and therefore, their measurement characteristics are unknown. Psychometric testing of these measures is necessary to increase the reliability and validity for future evaluations. Furthermore, these results were obtained at a single time point, about 2 years after the initial implementation of the program. As implementation is an evolving process, additional process evaluations across multiple time points, multiple sites, and with larger sample sizes are needed.15 Providers’ perceptions and attitudes are subject to change as their familiarity with a program increases and sustainment improvements are implemented. These results were also obtained from a single facility, and the study was conducted as part of a quality improvement project. Therefore, attitudes of providers in other locations may differ, and the results are not intended to generalize to other facilities or the national REACH VET program. In addition, the providers’ opinions may not relate to program effectiveness; recent national data supports the effectiveness of the program.5

CONCLUSIONS

It is notable that the clinicians’ concerns rarely focused on fears about the algorithmic approach replacing or limiting their role as providers, but rather concerns about what this approach was adding to their role. We interpret this finding with cautious optimism. Our evaluation here suggests that local clinicians may be more open to interventions based on predictive analytic models than has previously been suggested; however, additional work is needed locally to harness the strength of predictive analytic models for suicide prevention.

ACKNOWLEDGMENTS

The authors acknowledge and appreciate the efforts of Sara Landes, PhD, who provided consultation regarding the creation of the items used in the qualitative interview, and Caitlin Manchester, MPH, who provided consultation regarding the analysis of qualitative data.

REFERENCES

1. Office of Mental Health and Suicide Prevention, US Department of Veterans Affairs. 2019 National Veteran Suicide Prevention Annual Report. Washington, DC: US Department of Veterans Affairs; 2019. Available at: https://www.mentalhealth.va.gov/docs/data-sheets/2019/2019_National_Veteran_Suicide_Prevention_Annual_Report_508.pdf. Accessed August 7, 2020.
2. Office of Mental Health and Suicide Prevention, US Department of Veterans Affairs. National Strategy for Preventing Veteran Suicide 2018–2028. Washington, DC: US Department of Veterans Affairs; 2018. Available at: https://www.mentalhealth.va.gov/suicide_prevention/docs/Office-of-Mental-Health-and-Suicide-Prevention-National-Strategy-for-Preventing-Veterans-Suicide.pdf. Accessed August 7, 2020.
3. Office of Public Affairs, US Department of Veterans Affairs. News Release: VA REACH VET Initiative Helps Save Veterans Lives: Program Signals When More Help Is Needed for At-risk Veterans. Washington, DC: US Department of Veterans Affairs; 2017. Available at: https://www.va.gov/opa/pressrel/includes/viewPDF.cfm?id=2878. Accessed August 7, 2020.
4. McCarthy JF, Bossarte RM, Katz IR, et al. Predictive modeling and concentration of the risk of suicide: implications for preventive interventions in the US Department of Veterans Affairs. Am J Public Health. 2015;105:1935–1942.
5. McCarthy JF, Cooper SA, Dent KR, et al. Evaluation of the Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment suicide risk modeling clinical program in the Veterans Health Administration. JAMA Netw Open. 2021;4:e2129900.
6. Reger GM, McClure ML, Ruskin D, et al. Integrating predictive modeling into mental health care: an example in suicide prevention. Psychiatr Serv. 2019;70:71–74.
7. Belsher BE, Smolenski DJ, Pruitt LD, et al. Prediction models for suicide attempts and deaths: a systematic review and simulation. JAMA Psychiatry. 2019;76:642–651.
8. Belsher BE, Smolenski DJ, Pruitt LD. Positive predictive values and potential success of suicide prevention models—a reply. JAMA Psychiatry. 2019;76:870–871.
9. Salganik MJ, Lundberg I, Kindel AT, et al. Measuring the predictability of life outcomes with a scientific mass collaboration. Proc Natl Acad Sci USA. 2020;117:8398–8403.
10. Warren MB, Smithkors LA. Suicide prevention in the US Department of Veterans Affairs: using the evidence without losing the narrative. Psychiatr Serv. 2020;71:398–400.
11. Dewan NA, Conley D, Svendsen D, et al. A quality improvement process for implementing the Texas Algorithm for schizophrenia in Ohio. Psychiatr Serv. 2003;54:1646–1649.
12. Wisdom JP, Bielavitz S, Mcfarland B, et al. Preparing to implement medication algorithms: staff perspectives and system infrastructure. J Psychiatr Pract. 2008;14:209–215.
13. R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: Foundation for Statistical Computing; 2017. Available at: http://www.r-project.org. Accessed August 7, 2020.
14. Bryman A. Social Research Methods, 4th edition. Oxford, UK: Oxford University Press; 2012.
15. Meyers DC, Durlak JA, Wandersman A. The quality implementation framework: a synthesis of critical steps in the implementation process. Am J Community Psychol. 2012;50:462–480.
Keywords:

predictive analytics; clinician attitudes; suicide prevention; Veterans; program implementation

Copyright © 2022 Wolters Kluwer Health, Inc. All rights reserved.