Secondary Logo

Journal Logo

Implementation Science

A Longitudinal Mixed-Methods Examination of Positive Health Check: Implementation Results From a Type 1 Effectiveness-Implementation Hybrid Trial

Garner, Bryan R. PhDa; Burrus, Olivia MPHa; Ortiz, Alexa MSNa; Tueller, Stephen J. PhDa; Peinado, Susana PhDa; Hedrick, Haley BAa; Harshbarger, Camilla PhDb; Galindo, Carla MPHb; Courtenay-Quirk, Cari PhDb; Lewis, Megan A. PhDa

Author Information
JAIDS Journal of Acquired Immune Deficiency Syndromes: September 1, 2022 - Volume 91 - Issue 1 - p 47-57
doi: 10.1097/QAI.0000000000003018
  • Open



After the results of several studies supporting treatment as prevention,1–4 HIV treatment has become a key HIV prevention priority. Nonetheless, the number of new HIV diagnoses in the United States has remained relatively stable at about 38,000 new diagnoses annually. Contributing to the number of new HIV infections each year is the number of people with HIV (PWH) who are not virally suppressed. Viral suppression rates have improved over time, yet about 420,000 PWH aged 13 years or older are not virally suppressed.5 Suboptimal adherence to the prescribed HIV treatment regimen is a key factor in why viral suppression rates are not higher.6 Research has focused on developing effective interventions to improve HIV medication adherence, including interventions that are computer based.7,8 Web-based HIV adherence promotion interventions may be more cost-effective and sustainable than person-delivered interventions.

In 2015, Claborn et al conducted a systematic review of computer-based HIV adherence promotion interventions.9 Of the 10 studies identified, only 3 were fully powered randomized controlled trials (RCTs).10–12 Concluding there was not yet sufficient evidence to support the efficacy of computer-delivered HIV adherence interventions, the authors noted more RCTs were needed and “Future studies should be designed with regard to evaluation of implementation and sustainability of the intervention within the clinic setting.” In 2018, Kemp and Velloza reviewed research conducted since 2015 that focused on implementing electronic health interventions designed to improve outcomes along the HIV care continuum.13 Of the 17 studies identified, more than half (n = 9; 53%) included a focus on improving HIV medication adherence,14–22 only 2 were RCTs, and the most of the studies (n = 15; 88%) focused on acceptability and/or feasibility.14,23–26 Consequently, in addition to recommending future research to examine other implementation outcomes, Kemp and Velloza highlighted the need for more research to help identify predictors of implementation.

The utility of implementation research for maximizing the prevention and treatment of HIV has been highlighted several times during the past decade.27–34 Indeed, in their discussion of challenges in the optimal implementation of HIV prevention, diagnosis, treatment, and care, Eisinger et al35 concluded that “knowledge gained through implementation science will be critical…to bring HIV prevention and treatment interventions to scale, and thus, achieve the goal of ending the HIV epidemic domestically and globally.”

The present study sought to build on the extant research on implementation effectiveness36–41 defined as the consistency and quality of implementation over time. We examine several constructs hypothesized by the theory of implementation effectiveness36–39 to be determinants of implementation effectiveness as part of an effectiveness-implementation hybrid trial focused on Positive Health Check (PHC). PHC is a web-based video doctor intervention designed to be delivered to PWH while in the clinic. PHC provides tailored content about medication initiation and adherence, sexual risk reduction, and other behaviors to decrease HIV transmission risk. The purpose of the effectiveness-implementation hybrid trial was to test if PHC supports viral suppression and retains PWH in care. Analyses showed that PHC was particularly effective for a priori defined subgroups. Males were more likely to achieve viral suppression, and the youngest and oldest participants were more likely to be retained in care.42

The design of the PHC effectiveness-implementation hybrid trial has been previously described.43 As prompted by Kemp and Velloza, one aim of the PHC pragmatic trial was to explore beyond feasibility and acceptability to what extent other important implementation constructs could be identified as predictors of implementation effectiveness.13 The theory of implementation effectiveness36–39 posits that innovation-values fit, organizational readiness for change, and implementation climate are 3 key determinants of implementation effectiveness. These implementation constructs are defined as follows: (1) innovation-values fit (ie, the extent to which staff perceive that innovation use will foster the fulfillment of their values)36–39; (2) organizational readiness for change (ie, the extent to which organizational members are psychologically and behaviorally prepared to implement organizational change); and (3) implementation climate (ie, the extent to which organizational members perceive the use of a specific innovation to be expected, supported, and rewarded within their organization).

Given the limited empirical research examining these determinant constructs39–41 and the need for psychometrically sound measures to assess these constructs,44–46 we chose to use a mixed-methods longitudinal analysis for this study to use qualitative data from interviews with PHC implementers to help confirm and explain quantitative findings regarding the stability or change over time around these 3 constructs. Our general null hypothesis was there would not be any significant changes over time in these 3 contrasts. Consistent with the philosophy that change is the only constant in life our general alternative hypothesis was that there would be significant changes over time for each of these posited determinants of implementation effectiveness. We then used novel joint display tables to present the statistical outcomes and supporting qualitative data side-by-side. This approach to integrating qualitative and quantitative data allowed us to comprehensively describe our findings from the implementation component of the PHC hybrid trial.


Study Design

For this hybrid trial's implementation aim, a longitudinal mixed-methods design was used in which quantitative (surveys) and qualitative (interviews) assessments were conducted regarding PHC implementation.

Surveys and interviews overlapped with the PHC implementation period across all 4 clinics and were conducted at the beginning of the study and then at every 3 months over a 23-month period, providing 8 data collection time points. These assessments were conducted from February 2018 to December 2019. The design uses data from the qualitative assessments to provide context for the interpretation of the quantitative findings.


PHC is an interactive, highly tailored intervention informed by multiple health behavior theories, including motivational interviewing,47 the Information-Behavioral-Motivation Model,48,49 and the Transtheoretical Model.49 The PHC intervention consists of 7 core components: (1) participant-reported tailoring questions. This included 4 demographic questions, delivered by a video nurse used to tailor and route a participant through the intervention, and 17 questions delivered by a video doctor, interspersed throughout the intervention to provide tailored information in 6 domains, including treatment readiness, medication adherence, RIC, sexual risk reduction, mother-to-child transmission, and injection drug use; (2) tailored content delivered in the 6 domains; (3) behavior change “tips” provided across the 6 domains; (4) 4 video doctor options [varying by race (Black, White) and sex (female and male)]; (5) library that autogenerated a list of tailored questions based on participant preferences that could be used during their clinical encounter; (6) patient handout, which could be printed on site, showing the behavior change tips and questions for the provider; and (7) an “Extra Info” microsite at the end of the intervention with additional resources and information, such as sexually transmitted infections, condom use, mental health, and transgender health. A clinic web application also helped clinic staff keep track of user progress, assisted with personal identification creation, and produced summary descriptive reports about overall use of PHC.

Implementation Strategy

The implementation strategy for the trial was informed by a previous 1-month pilot implementation. The strategy involved preimplementation preparation, such as clinic workflow assessments, technology assessments, staff training, and understanding the patient population. Implementation strategies supporting the PHC launch involved physical walkthroughs of the clinic environment and workflow integration, Wi-Fi testing, educational sessions for clinic staff, study staff training, development of an implementation binder describing all aspects required for successful implementation, and a soft launch. During implementation, each clinical site received tailored support, lessons learned were shared across sites, FAQs were revised in real time, implementation technical assistance was provided as needed, PHC implementation was monitored via the clinic web application, and the Centers for Disease Control and Prevention maintained a Help Line users could call with technical questions.50


PHC implementation and data collection occurred simultaneously across 4 demographically diverse US-based HIV care clinics, 1 in the south-central region, 2 in the southeast region, and 1 in the northeast region. The clinics are described as follows:

  • Clinic A Southeast Region Clinic is a primary care practice and medical specialty practice with an average of 20 HIV patients per day.
  • Clinic B South-Central Region Clinic is an ambulatory clinic, a community health center, a primary care practice, and a nonprofit clinic with an average of 15 HIV patient visits per day.
  • Clinic C Northeast Region Clinic is an ambulatory care and academic medical center with an average of 40 HIV patient visits per day.
  • Clinic D Southeast Region Clinic is a primary care practice and specialty care practice with an average of 110 HIV patient visits per day.


Guided by our teams' research focused on the rates and impacts of staff turnover,51,52 our current research focused on clinic roles/positions (eg, project coordinator, data manger, outreach coordinator, research coordinator, or clinic champion). This is important given that clinics generally have much less control on the extent to which individuals are retained than they do on the extent to which their clinic roles/positions have 1 or more actively employed staff.

Data Sources and Data Collection Procedures

At each of the 8 time points (T1–T8), starting immediately before implementation of PHC with patients, and every 3 months thereafter, participating staff members across all 4 clinics completed a 15-minute online survey assessing 3 key implementation constructs: perceived fit, organizational readiness, and implementation climate. At the start of each survey, staff agreed to participate through an online consent form. After the online survey, we conducted individual interviews with staff who completed the survey at that time point. At the start of each qualitative interview, we obtained verbal consent from the clinic staff member. Each clinic's Institutional Review Board and the RTI International Institutional Review Board approved the study protocol.

Quantitative Measures

Innovation-values fit was assessed via 25 items developed for this project. More specifically, using a 5-point scale (0 = not at all to 4 = highest extent possible). Each staff member answered 5 fit questions related to implementing PHC (ie, fit your clinic workflow; fit your clinic values; fit your clinic treatment philosophy; was accepted by staff within your clinic; was well-matched to your clinic environment). These fit questions were asked in regard to 5 PHC study components: (1) PHC Patient Onboarding (ie, staff assisting users with logging-in), (2) PHC Delivery through iPads or android tablets, (3) PHC Handout Printing and Delivery, (4) Clinic Web Application for tracking patients' PHC use, and (5) PHC Patient Outreach. The average coefficient alpha (α) across the 8 time points was 0.95 (SD = 0.02). Organizational readiness was assessed with 12 items adapted from a measure developed by Shea et al.53 Reponses were given on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Across the 8 time points, α was 0.92 (SD = 0.03). Example items include “Positive Health Check staff implementing PHC want to implement this intervention” or “Positive Health Check staff implementing PHC are motivated to implement this intervention.” Finally, implementation climate was assessed using 6 items adapted from a measure developed by Jacobs et al.54 Reponses were recorded on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Although not as high as for the other 2 measures, the α was acceptable at 0.76 (SD = 0.11). Example items include “Positive Health Check staff implementing PHC were expected to help the clinic meet its goals for implementing Positive Health Check” or “Positive Health Check staff implementing PHC got the support from clinic management they need to use Positive Health Check with eligible patients.” Scale scores for innovation-values fit, organizational readiness, and implementation climate were constructed as the mean of the individual items corresponding to each construct computed at the respondent level.

Qualitative Measures

Using a semistructured interview guide, we sought to further understand the stakeholders' perceptions of their respective clinic's implementation climate, organizational readiness, and perceived fit of PHC as assessed by the online survey. After reviewing the clinic site's survey responses, individual staff were asked a series of open-ended questions, for example, “Thinking about the process overall, what are some activities related to clinic workflow that are facilitating the implementation of Positive Health Check?” and “How, if at all, are project staff supported to make sure PHC is implemented as it needs to be?” Interviews typically lasted 30–45 minutes. Two study staff conducted each interview by phone, one to lead the interview and one to take notes. All interviews were digitally recorded and transcribed for analysis.

Quantitative Data Analysis

Examination of the implementation context over time often entails having a relatively few analytic units,55 in this case HIV primary care clinics. We managed the constraints of the small sample size by using longitudinal mixed-effects regression models of respondent level scale scores. Clinic differences were modeled using random intercept terms for clinic, and repeated measures within individual respondent were modeled using random slopes. The fixed effects for linear and quadratic slopes assess change over time, and their associated t-tests were used to test whether there was significant change over time within the implementation context for PHC. Each outcome was modeled separately with time and time squared as predictors, and the data collection time points were scaled from 0 to 7 so that the mean of the random intercepts can be interpreted as the predicted value of the outcome at the first time point. These predictors, respectively, tested whether there was change in the outcome overall and whether change was curvilinear (eg, the outcome improved, then deteriorated). Model fitting was implemented using the PersonAlytics56 R package,57 which uses the GAMLSS framework and R package for model fitting.58

Qualitative Data Analysis

We used a framework analysis method59 for the qualitative coding and analysis approach. Once interviews were transcribed and entered into NVivo, a multifunctional software system for coding and analysis, 3 project team members independently coded a subset of transcripts. Discrepancies in initial coding were resolved by discussion between project team members. The coders then independently conducted a final coding of the remaining data, with 1 staff person coding each transcript. Intercoder agreement was quantified for each code using Cohen Kappa.60 The range of Kappa across codes was 0.75–1.00. Finally, thematic analysis was conducted.

Mixed-Methods Integration

Upon completion of the quantitative and qualitative data analyses, we integrated the data using joint display tables to examine the changes in implementation context over time (Tables 1–4). These tables present the 2 forms of data to display how the commonalities and differences across dimensions of implementation context vary over time and also account for stakeholder type (eg, PHC outreach coordinator) and characteristics of each clinical site (eg, academic medical center).

TABLE 1. - Overall Trajectories for Innovation-Values Fit, Organizational Readiness for Change, and Implementation Climate
Cross-clinic Trajectories Beta PHC Staff Experiences
Innovation-values fit Intercept: 4.18*
Time: −0.25
Time2: 0.03
Three of the 4 clinics reported few major ongoing
barriers with integrating PHC into the clinic workflow.
Communication and collaboration with front office staff and the PHC staff's physical location within the clinic emerged as key factors for integrating PHC into the clinic workflow.
Addressing internet connectivity issues and printer access at 2 clinics and space concerns at 1 clinic align with the slight increase in perceived fit at T4 ().
Organizational readiness for change Intercept: 4.42*
Time: 0.11
Time2: −0.02
Three of the 4 clinics noted a slight decrease in
organizational readiness at T7 because of clinic providers misunderstanding PHC requirements (clinic B), a nondirect chain of communication clinic C, and staff turnover clinic D.
Implementation climate Intercept: 3.95*
Time: 0.55*
Time2: −0.09*
Clinics had not started implementing the intervention
at T1. The initial trend upward may be because of ramp-up time.
The most common leadership method used to support PHC was advocating for the study in clinical site meetings.
All 4 clinics noted receiving support from site health care providers.
Implementation climate dipped in the end for many reasons, including decreased engagement from champions and difference in priorities between research and clinic staff.
Time refers to measurement time point, T1 to T8. The Time variable indicates how fast the outcome is increasing over time. The Time2 variable indicates how fast the outcome starts moving back to values from earlier in the study. The combination of these 2 effects describes the upside-down “u” shape of the trajectory over time.
*P < 0.05 indicates a significant change over time.

TABLE 2. - Innovation-Values Fit Trajectories, by Clinic
Trajectories PHC Staff Experiences Illustrative Quotes
Clinic A
Intercept: 3.89*
Time: 0.00
Time2: −0.01
There were very few clinical workflow barriers to integrating PHC.
Consistent communication and collaboration between PHC staff and clinic staff facilitated intervention delivery.
The clinic's physical space enabled easy intervention and handout delivery within the clinic's existing workflow.
The clinic web application was useful for monitoring patient progress, but it would time out too quickly.
There were small but ongoing concerns around PHC delaying clinic schedules, particularly when patients arrive late.
“No (workflow barriers to report), our clinic is
awesome and their workflow, they've adjusted so well to us and want to accommodate us. So that's not a problem ever.”
(T3, Study Coordinator)
“There's a designated spot for the patients who utilize the device. We have a phlebotomy room where we're able to see the patients privately deliver the intervention privately. Having those features of the clinic makes the implementation or the delivery of the intervention much easier.” (T5, Outreach Coordinator)
Clinic B
Intercept: 3.20*
Time: −0.16
Time2: 0.03
Finding space for PHC delivery was a significant
challenge until the clinic moved into a new building at T4.
New clinical care hires at T3 exacerbated space issues.
Early frustration with the printer location was alleviated when the clinic was relocated to a new building with technology that enabled staff to print handouts at any printer within the clinic.
Patient availability for PHC was influenced by the number of appointments with other departments (eg, case managers, behavioral health).
Toward the end of the study, decreased staffing on PHC became a minor capacity issue.
“As it’s been, it's been fine except for when we had to
move the printer around.” (T2, Project/Outreach Coordinator)
“We've recently had an uptake in hires, especially at the marine building. The pods that we used to have open all the time, now they're starting to be taken […]”
(T3, Study Coordinator)
“We're in a newer building. […] The clinic is designed where there's lots of private rooms where they can be very close to either the lab or the doctor's office. So, I don't see any barriers.”
(T7, Clinic Champion)
Clinic C
Intercept: 3.94*
Time: −0.25
Time2: 0.03
The physical location of the PHC staff within the
clinic supported the integration of PHC into the workflow.
Ongoing issues included limited clinic space, patient no-shows, and late arrival times that prevented patients from completing the tool—especially at T5 and T6.
At T2, there were issues with the clinic's Wi-Fi and with printing patient handouts (which worsened with poor Wi-Fi). At T4, staff began using a clinic network printer, easing the handout printing process.
At T5, staff noted that research was secondary to patient care and that patients occasionally were not interested in PHC because it did not provide new information.
“We did have a problem with printing the handouts
for a while because the hotspot or whatever that we were given didn't really work. […] We started a new process […] where we get instead of having to print the handout on that specific printer and everything, we get it in our email. […] This is really an amazing change because now we get all the handouts.” (T4, Research Supervisor)
“It depends on, patients don't always come when they're supposed to, and I think that has been the challenge. […] If they come right on time and the physician is ready to take the patient, they're not going to wait because then they will get backed up, and that impacts the clinic load.”
T4, Clinic Champion
Clinic D
Intercept: 3.90*
Time: 0.04
Time2: −0.02
Overall, there was ample clinic space to implement
PHC staff consistently noted challenges around communicating with and engaging clinic staff, particularly front desk staff. This was attributed in part to high clinic staff turnover.
The location of the PHC research team outside of the physical clinic space posed a challenge. The relocation of the research office near T3 further contributed to this challenge.
At T2, staff discussed concerns around Wi-Fi connectivity, which was addressed at T3 with the purchase of a Wi-Fi hotspot.
With the exception of the printer occasionally being slow, there was no issue with handout delivery.
“Communication between research and the front desk,
or providers. Although we've improved, there have been barriers where communication maybe hasn't been processed all the way through. Like we'll tell the provider, and the provider maybe forgets to tell the patient, or the front desk forgets to tell the patient.” (T5, Outreach Coordinator)
“Although we are in the same building, we're not located in the same exact room. We're a little bit further away, and out of the natural flow of things, from like check-in to nursing station, to labs and everything. We're completely removed from that. So that's been a huge challenge.” (T6, Outreach Coordinator)
Time refers to measurement time point, T1 to T8. The Time variable indicates how fast the outcome is increasing over time. The Time2 variable indicates how fast the outcome starts moving back to values from earlier in the study. The combination of these 2 effects describes the upside-down “u” shape of the trajectory over time.
*P < 0.05 indicates a significant change over time.

TABLE 3. - Organizational Readiness for Change Trajectories, by Clinic
Trajectories PHC Staff Experiences Illustrative Quotes
Clinic A
Intercept: 4.43*
Time: 0.04
Time2: −0.01
Early engagement from the clinical champion set the
stage for organization buy-in at the start and throughout the PHC implementation process.
Continual communication between PHC staff and clinic staff enabled the organization to smoothly implement PHC.
“Over the past several months our clinic staff
meeting, which is not a research meeting but just a meeting of all the clinic providers and staff, I've been getting people excited and enthusiastic and trying to head off any concerns we may have as we roll into this.” (T1, Clinical Champion)
“I think I speak for the entire project staff that we feel very supported by our leadership, by our clinic team […] and the clinic staff is as supportive as they can be without causing too much additional work for them because they're overworked and understaffed. […] But they're incredibly supportive.”
(T5, Outreach Coordinator)
Clinic B
Intercept: 4.57*
Time: −0.04
Time2: 0.00
“Road bumps” were noted at T2 because PHC
research staff were grouped within the IT department, whose workflow is not integrated into that of the clinic or research department.
At T7, staff noted a lack of confidence navigating some PHC implementation challenges, specifically those related to co-enrolment in other studies in the clinic, and therefore contacted the clinical champion for help.
“I think for the position we're in, it's being
implemented really smoothly, but I think a lot of the road bumps, or the bumps along the way, there's questions we've had to ask that would have been easier answered if we were in a different department.” (T2, Project/Outreach Coordinator)
“We've been withdrawing some folks because they were getting to co-enrolled in another research program and we definitely had to go to (…) our PI, and some leadership and ask them to kind of step in and take over on that front because we didn't think we could do it.”
(T7, Study Coordinator)
Clinic C
Intercept: 4.40*
Time: 0.23
Time2: −0.03
Organizational readiness was high except at T1 and
T7. This may have been because of frustrations around communication.
At T1, PHC staff noted that physicians had not yet been fully introduced to PHC, which might have been a barrier to their fully supporting the intervention.
At T7, PHC staff noted that the chain of communication for PHC staff is not direct. However, meetings helped clarify roles and expectations within the organization.
“I think we've mentioned it at clinic meetings, and
it'll be on our agenda for our next physician meeting, […] We'll be able to introduce it in a little bit more detail to the physicians, to let them know the study is going to be opening and that the types of patients that are going to be eligible for it.” (T7, Clinical Champion)
“We'll set time at the end of the week to huddle, particularly when all the other changes, all of our staffing changes were happening. We were meeting a bit more frequently to go over what the expectations are, to understand what people's roles were, and to make sure everything was really clear.” (T7, Research Supervisor)
Clinic D
Intercept: 4.48*
Time: 0.13
Time2: −0.02
Overall, staff felt that leadership displayed open
communication and problem-solving abilities.
At T7, staff felt less confident in their ability to successfully execute PHC because of organizational challenges, such as high clinical staff turnover.
“PHC is definitely one of the main topics [discussed
during the monthly clinic meeting] most of the time because it's the one study that requires the most coordination between providers and research. I think it's always being discussed, better ways of maintaining the study.”
(T2, Study Coordinator)
“You know, we end up doing a lot of heavy lifting by the PHC staff rather than involve the clinic staff, because we have a lot of staff turnover and is so true. So, (Data Manager) eg, and (Out reach Coordinator) now has been more involved to get the patient to come back for follow up.” (T6, Outreach Coordinator)
Time refers to measurement time point, T1 to T8. The Time variable indicates how fast the outcome is increasing over time. The Time2 variable indicates how fast the outcome starts moving back to values from earlier in the study. The combination of these 2 effects describes the upside-down “u” shape of the trajectory over time.
*P < 0.05 indicates a significant change over time.

TABLE 4. - Implementation Climate, by Clinic
Trajectories PHC Staff Experiences Illustrative Quotes
Clinic A
Intercept: 4.24*
Time: 0.38*
Time2: −0.06
PHC research staff felt supported by and had strong
communication with clinic staff.
Clinic leadership was highly engaged throughout and regularly sent email updates to clinic staff and discussed PHC during meetings.
Early on and toward the end of the study, some clinical staff got upset when PHC caused delays in the clinic workflow. No clear theme emerged explaining a decline in climate at the end.
“Oh yeah. We definitely are all updates are sent to us
during staff meetings in the clinic. We are kept in the loop.” (T3, Project Coordinator)
“Well, anytime you make a change to something, folks who are established in a particular clinic can feel like this is one more thing we got to do and we try not to make that a burden for them but for us to handle it.” (T5, Project Coordinator)
Clinic B
Intercept: 3.92*
Time: 0.38*
Time2: −0.06*
PHC was supported by the clinic and staff
expectations were consistently communicated.
The most common leadership method used to support PHC was advocating for the study in clinical site staff meetings.
There were changes at T4, including hiring a new clinical champion and data manager and moving to a new building, which temporarily decreased champion engagement.
Toward the end, troubleshooting and problem solving was expected to be done independently by the research team with only ad hoc support from the clinical champion.
“I think it's been pretty unique at this clinic because
we had so many changes in the past 3 months with […], our previous PI leaving and […], our data person, being absent for a while and the clinic moving, that it’s definitely leadership has been a little bit more absent than I think we'd like.”
(T4, Outreach Coordinator)
“On our monthly clinician meetings or provider meetings we always do some sort of update or reminder to the folks that, whether they're working on re-engagement or recruiting, we let the providers know, you know, that it’s happening.” (T6, Clinical Champion)
Clinic C
Intercept: 4.11*
Time: 0.42*
Time2: −0.05*
There was a lot of support within the clinic for PHC.
The clinic had 2 clinical champions but not until T2 did PHC staff note feeling supported.
Leadership engagement and support were mainly provided through routine or ad hoc meetings.
Clinical staff showed their support by being flexible in their schedules and helping to identify patients.
“Leadership is very involved and that I can call
either one of the PIs on this study and ask them a question right now and they would help me. And if they couldn't help me they would find a way to help me. (…) They're always really available for us.”
(T2, Data Manager)
“We support each other […] we always talk to each other, the case managers, the outreach people, the social worker, nutritionist. All is integrated here in the clinic, and mostly we work together.”
(T7, Project Coordinator)
Clinic D
Intercept: 4.13*
Time: 0.38*
Time2: −0.05*
The research team felt supported by clinic
leadership. Although some were more involved than others, the clinical champion was engaged throughout.
Early on, and at T6 and T7, staff noted a disconnect between PHC staff and clinic staff because of poor communication and resistance from clinical staff to adjust the clinic's workflow to support the project.
Implementation climate improved when the clinical champion helped troubleshoot problems for PHC staff.
“We were still figuring out who should be doing
what and at what time or at what stage of the patients' visit. But I think now everything is working pretty well and everyone more or less knows what they need to be doing.” (T2, Project Coordinator)
“Yeah, I help to troubleshoot all the problems in terms of implementation, recruitment, I work closely with my team to do that.” (T5, Clinical Champion)
Time refers to measurement time point, T1 to T8. The Time variable indicates how fast the outcome is increasing over time. The Time2 variable indicates how fast the outcome starts moving back to values from earlier in the study. The combination of these two effects describes the upside-down “u” shape of the trajectory over time.
*P < 0.05 indicates a significant change over time.



A range of staff across the 4 clinics participated in each round of surveys and interviews. Staff across clinics also shared roles. Consequently, a clear delineation of which staff member was taking on which PHC role in each round was not feasible. We interviewed site clinic champions, and staff responsible for implementing and monitoring the intervention, including site coordinators responsible for onboarding and introducing PHC to participants, data managers, and outreach workers. We had participation from 3 to 5 staff at each site for each time point, resulting in a total of 126 completed interviews and surveys. Staff did not receive individual incentives for participation. Clinics received funding for their overall involvement in the PHC study which covered the salary for involved staff. Although some clinic staff were assigned to work on the study full time, others spilt time between the study and other clinic-based tasks.

Outcome Data

Results are summarized in Tables 1–4. In Table 1, quantitative results are presented in the “Trajectories” column and results of the mixed-effects models are shown in the “Beta” column; they are also superimposed in the “Trajectories” column of Tables 2–4, with the labels “Time” and “Time2” (time squared). Qualitative results are summarized in the “PHC Staff Experiences” columns of Tables 1–4 and the “Illustrative Quotes” columns of Tables 2–4.

Results of the mixed-effects models are summarized in statistical tests for each predictor labeled as “beta” in Tables 2–4, which show the clinic average of the scale scores (see Quantitative Measures) for each outcome. The beta value for the intercept gives the predicted average baseline value for each outcome, and they are all significantly different from zero. The beta values for time and time squared are the rates of change over time and over time squared, respectively. When the beta values for time were not significantly different from 0, there is no change over time on average. A significant positive beta value indicates improvement over time on average, and a significant negative beta indicates deterioration over time. When the beta values for time squared were significantly different from 0 and were negative, this indicates that improvements in the outcome improved and then attenuated over time. All beta values for time squared that were significantly different from zero were negative (if positive values had occurred, this would indicate deterioration, then improvement).

Innovation-Values Fit

On average, innovation-values fit was 3.77 (SD = 0.79) at T1 and 3.62 (SD = 0.63) at the final time point, with no significant change over time (Table 1). As reported in Table 2, only one of the 4 clinics reported having a noteworthy issue regarding innovation-values fit. More specifically, staff from clinic B reported barriers to finding space for delivering PHC, which were exacerbated at T3 because of several new clinical care hires. By T4, space issues had been resolved as a result of moving into a new building with more available space. For clinic C, innovation-values fit was found to be the lowest at T5 and T6, with staff reporting that some patients were not interested in PHC because it did not provide new information.

Organizational Readiness for Implementing Change

On average, organizational readiness was 4.39 (SD = 0.69) at T1 and 4.67 (SD = 0.41) at T8, the final time point, with no significant change over time across all clinics (Table 1). As also reported in Table 3, clinic C staff noted at T1 that PHC had not fully been introduced to physicians, which may explain the relatively low organizational readiness rating at T1. However, organizational readiness significantly increased at T2 and remained relatively stable for the remainder of the project.

Organizational readiness was very high and very stable in clinic A, where early engagement from the clinical champion was reported as having set the stage for organizational buy-in and maintained buy-in throughout the project. Although not significant, both clinic C and clinic D reported decreased organizational readiness at T7. At this time point, clinic C staff noted that the chain of communication for PHC staff was not direct, whereas clinic D staff noted feeling less confident in their readiness to implement PHC because of recent staff turnover.

Implementation Climate

There was significant curvilinear change over time regarding implementation climate. On average, implementation climate was 3.82 (SD = 0.88) at T1 and 4.08 (SD = 0.87) at T8, the final time point, but peaked at 4.77 (SD = 0.32) at T5 and decreased at T6 through T8 (Table 1). Table 4 shows that in clinic A, no clear themes emerged that helped explain the declines in implementation climate reported at T7 and T8, last 2 time points. In clinic B, however, decreased implementation climate seems to have been due in part to staffing changes and decreased support from the clinical champion toward the end of implementation, who was expecting staff to be able to address issues with only ad hoc support.

The implementation climate in clinic C remained relatively high over time. In clinic D, the decrease in implementation climate rebounded after the clinical champion helped troubleshoot problems.


As part of a type 1 effectiveness-implementation hybrid trial, we used a mixed-methods approach to examine the context of implementing PHC over a 23-month period within 4 HIV primary care clinics. Overall, we found innovation-values fit and organizational readiness for change to be quite high throughout the project. There were some logistical challenges, however, in implementing the intervention, and we found a significant curvilinear pattern of change over time in implementation climate.

The analysis of the quantitative survey data show that, overall, innovation-values fit remained stable. Staff interviews suggested that most of the barriers were around WIFI glitches and finding space. Any barriers to fitting PHC into clinic workflow were well managed by clinic staff. This is encouraging for the successful implementation of PHC in other settings and contrasts with previous research reporting that clinic workflow and other aspects of the clinic environment can be barriers to implementing STD/HIV interventions in busy clinic settings.61,62

With regard to organizational readiness for change, the quantitative data also show relatively stable levels over time across the 4 clinic environments. The qualitative data reveal many facilitators for organizational readiness, including the clear communication of staff roles and responsibilities for implementing PHC, and many barriers such as in one clinic, neglecting to confirm that all physicians were introduced to the PHC intervention.

Although adequate levels of innovation-values fit and organizational readiness may be prerequisites for effective implementation, they are not likely to be sufficient. Indeed, implementation effectiveness theory posits implementation climate—the extent to which implementation by the intended users is expected, supported, and rewarded—as the most proximal determinant of implementation effectiveness.36,38 In alignment with the theory of implementation effectiveness, we found that both our quantitative and qualitative data support significant changes over time in implementation climate.

Importantly, we observed significant variation in implementation climate for each of the 4 HIV primary care clinics. The qualitative data provided more contexts to facilitate a partial explanation for this pattern of results. For example, during interviews, stakeholders reported that key issues between research staff and healthcare providers included communication, coordination, and staffing. Thus, we hypothesize that a key factor that may have helped improve implementation climate, especially the support dimension of implementation climate, was the increasing involvement of a strong clinic champion, which has been noted in previous literature as an important implementation strategy.63 In implementation climate declines observed, we hypothesize that these declines may be common to research projects that have a predefined end and may not be observed if the implementation of the intervention was something the organization was planning to sustain or even scale-up over time.

As previously mentioned, much of the implementation research on HIV interventions has focused on feasibility and acceptability.23,32–35 The current study moves beyond those constructs to explore other important implementation constructs, including innovation-values fit, organizational readiness for change, and implementation climate.36–38 However, the extent to which research has empirically examined these constructs has been limited because of the lack of psychometrically sound measures.44,64 In addition to being one of the first studies to use the organizational readiness measure developed by Shea et al,53 the current study is the first known study to have collected this measure at multiple time points throughout the implementation process. The current study is also the first we are aware of to have longitudinally collected measures of innovation-values fit at so many time points during the implementation process. In the absence of the qualitative interviews, the limited change over time found may suggest that these 2 measures lacked sensitivity to change. Fortunately, however, the qualitative interviews substantiated the relatively high and stable levels of innovation-values fit and organizational readiness for implementing change.


A potential limitation to this assessment of PHC implementation is the inability to adjust measures by clinic-specific respondent. Staff roles across each clinic shifted and the number surveys/interviews for each round ranged as a result of limited staff availability. Consequently, it was not feasible to make clear delineations for which staff member assumed which PHC role. Future studies examining clinic-level climate and culture issues should consider adjusting for respondent composition in the regression models.

Another possible limitation is that the data collection ended (December 2019) before the PHC implementation process ultimately ended (March 2020), so we cannot describe how these constructs varied near the end of implementation. In addition, because we had only 4 clinics in the study, we did not have sufficient power to examine the relationship between these implementation constructs and implementation effectiveness (ie, the consistency and quality of implementation over time). Future studies engaging larger numbers of clinics should include analyses that link these constructs with implementation effectiveness.


Technology-based interventions may be one of the most promising innovations to increase HIV viral suppression rates and decrease the number of new HIV infections.65 However, given the consistent lag between research and practice, there is increasing recognition of the need to integrate effectiveness research and implementation research.66,67 Our innovative type 1 hybrid trial design demonstrated that within the context of a research project, PHC could be successfully implemented within 4 HIV clinics. We believe that the combined factors of perceived fit, organizational readiness, and implementation climate facilitated implementation of PHC over time. Indeed, the qualitative data show that as participating HIV primary care clinics intermittently addressed space, workflow, and technical issues, they also consistently reported PHC as being a good fit within their clinic. Additionally, each HIV primary care clinic was able to achieve and then sustain adequate levels of readiness to implement the PHC intervention. In contrast, the extent to which PHC implementation was expected, supported, and rewarded was found to vary significantly over time in each of the HIV primary care clinics. Examining these factors as determinants of implementation effectiveness is planned as part of our future work.

Finally, as this study demonstrates, it may be important for implementation strategies to increase the extent and consistency with which an intervention is expected, supported, and rewarded to result in improved implementation outcomes.


The authors would like to thank members of the Positive Health Check Study Team, including: Mable Chow, Surya Nagesh, Bomi Choi, Kelli Bedoya, Paige Ricketts, Gabriela Plazarte, and Nadine E. Coner at the Florida Department of Health, Hillsborough County; Eva Dunder, MPH, Christian Telleria, BS, Dustin Baker, MPH, and Jade Zeng, MPH from NO/AIDS Task Force at Crescent Care; Theron Clark Stuart, Elena Gonzalez, Kathryn Meagley, MPH, Nana Addo Padi-Adjirackor, and Rincy Varughese at the Atlanta VA Medical Center; and Marta Paez Quinde, Valorie Cadorett, Eric Asencio, and Robenson Saint-Jean at the Rutgers, New Jersey Medical School Infectious Disease Practice; and Arin Freeman, Ann O'Leary, and Gary Marks from the Centers for Disease Control and Prevention.


1. Cohen MS, Chen YQ, McCauley M, et al. Antiretroviral therapy for the prevention of HIV-1 transmission. N Engl J Med. 2016;375:830–839.
2. Rodger AJ, Cambiano V, Bruun T, et al. Sexual activity without condoms and risk of HIV transmission in serodifferent couples when the HIV-positive partner is using suppressive antiretroviral therapy. JAMA. 2016;316:171–181.
3. Bavinton BR, Pinto AN, Phanuphak N, et al. Viral suppression and HIV transmission in serodiscordant male couples: an international, prospective, observational, cohort study. Lancet HIV. 2018;5:e438–e447.
4. Rodger AJ, Cambiano V, Bruun T, et al. Risk of HIV transmission through condomless sex in serodifferent gay couples with the HIV-positive partner taking suppressive antiretroviral therapy (PARTNER): final results of a multicentre, prospective, observational study. Lancet. 2019;393:2428–2438.
5. Centers for Disease Control and Prevention. Monitoring selected national HIV prevention and care objectives by using HIV surveillance data-United States and 6 dependent areas, 2018. HIV Surveill Supplemental Rep. 2020;25:1–104.
6. Glass TR, De Geest S, Hirschel B, et al. Self-reported non-adherence to antiretroviral therapy repeatedly assessed by two questions predicts treatment failure in virologically suppressed patients. Antivir Ther. 2008;13:77–85.
7. Chaiyachati KH, Ogbuoji O, Price M, et al. Interventions to improve adherence to antiretroviral therapy: a rapid systematic review. AIDS. 2014;28(suppl 2):S187–S204.
8. Kanters S, Park JJ, Chan K, et al. Interventions to improve adherence to antiretroviral therapy: a systematic review and network meta-analysis. Lancet HIV. 2017;4:e31–e40.
9. Claborn KR, Fernandez A, Wray T, et al. Computer-based HIV adherence promotion interventions: a systematic review. Transl Behav Med. 2015;5:294–306.
10. Fisher JD, Amico KR, Fisher WA, et al. Computer-based intervention in HIV clinical care setting improves antiretroviral adherence: the LifeWindows Project. AIDS Behav. 2011;15:1635–1646.
11. Cote J, Godin G, Gueheneuc YG, et al. Evaluation of a real-time virtual intervention to empower persons living with HIV to use therapy self-management: study protocol for an online randomized controlled trial. Trials. 2012;13:187.
12. Hersch RK, Cook RF, Billings DW, et al. Test of a web-based program to improve adherence to HIV medications. AIDS Behav. 2013;17:2963–2976.
13. Kemp CG, Velloza J. Implementation of eHealth interventions across the HIV care cascade: a review of recent research. Curr HIV/AIDS Rep. 2018;15:403–413.
14. Murray MC, O'Shaughnessy S, Smillie K, et al. Health care providers' perspectives on a weekly text-messaging intervention to engage HIV-positive persons in care (WelTel BC1). AIDS Behav. 2015;19:1875–1887.
15. Georgette N, Siedner MJ, Zanoni B, et al. The acceptability and perceived usefulness of a weekly clinical SMS program to promote HIV antiretroviral medication adherence in KwaZulu-Natal, South Africa. AIDS Behav. 2016;20:2629–2638.
16. Henwood R, Patten G, Barnett W, et al. Acceptability and use of a virtual support group for HIV- positive youth in Khayelitsha, Cape Town using the MXit social networking platform. AIDS Care. 2016;28:898–903.
17. Kurth AE, Chhun N, Cleland CM, et al. Linguistic and cultural adaptation of a computer-based counseling program (CARE+ Spanish) to support HIV treatment adherence and Risk reduction for people living with HIV/AIDS: a randomized controlled trial. J Med Internet Res. 2016;18:e195.
18. Bardosh KL, Murray M, Khaemba AM, et al. Operationalizing mHealth to improve patient care: a qualitative implementation science evaluation of the WelTel texting intervention in Canada and Kenya. Glob Health. 2017;13:87.
19. de Bruin M, Oberje EJM, Viechtbauer W, et al. Effectiveness and cost-effectiveness of a nurse- delivered intervention to improve adherence to treatment for HIV: a pragmatic, multicentre, open- label, randomised clinical trial. Lancet Infect Dis. 2017;17:595–604.
20. Hirsch-Moverman Y, Daftary A, Yuengling KA, et al. Using mHealth for HIV/TB treatment support in Lesotho: enhancing patient-provider communication in the START study. J Acquir Immune Defic Syndr. 2017;74(suppl 1):S37–S43.
21. Patel AR, Kessler J, Braithwaite RS, et al. Economic evaluation of mobile phone text message interventions to improve adherence to HIV therapy in Kenya. Medicine (Baltimore). 2017;96:e6078.
22. Ronen K, Unger JA, Drake AL, et al. SMS messaging to improve ART adherence: perspectives of pregnant HIV-infected women in Kenya on HIV-related message content. AIDS Care. 2018;30:500–505.
23. Bauermeister JA, Pingel ES, Jadwin-Cakmak L, et al. Acceptability and preliminary efficacy of a tailored online HIV/STI testing intervention for young men who have sex with men: the Get Connected! program. AIDS Behav. 2015;19:1860–1874.
24. Dryden-Peterson S, Bennett K, Hughes MD, et al. An augmented SMS intervention to improve access to antenatal CD4 testing and ART initiation in HIV-infected pregnant women: a cluster randomized trial. PLoS One. 2015;10:e0117181.
25. Wray T, Chan PA, Simpanen E, et al. eTEST: developing a smart home HIV testing kit that enables active, real-time follow-up and referral after testing. JMIR MHealth UHealth. 2017;5:e62.
26. Bull S, Thomas DS, Nyanza EC, et al. Tanzania Health Information Technology (T-HIT) system: pilot test of a tablet-based system to improve prevention of mother-to-child transmission of HIV. JMIR Mhealth Uhealth. 2018;6:e16.
27. Schackman BR. Implementation science for the prevention and treatment of HIV/AIDS. J Acquir Immune Defic Syndr. 2010;55(suppl 1):S27–S31.
28. Herbst JH, Glassman M, Carey JW, et al. Operational research to improve HIV prevention in the United States. J Acquir Immune Defic Syndr. 2012;59:530.
29. Glasgow RE, Eckstein ET, Elzarrad MK. Implementation science perspectives and opportunities for HIV/AIDS research: integrating science, practice, and policy. J Acquir Immune Defic Syndr. 2013;63(suppl 1):S26–S31.
30. Odeny TA, Padian N, Doherty MC, et al. Definitions of implementation science in HIV/AIDS. Lancet HIV. 2015;2:e178–e180.
31. Ware NC. Qualitative contributions to implementation research on HIV prevention and treatment. J Acquir Immune Defic Syndr. 2019;82(suppl 3):S217–S221.
32. Hargreaves JR, Hassan S, Schellenberg J, et al. Five challenges in the design and conduct of IS trials for HIV prevention and treatment. J Acquir Immune Defic Syndr. 2019;82(suppl 3):S261–S270.
33. Hargreaves J, Dalal S, Rice B, et al. Repositioning implementation science in the HIV response: looking ahead from AIDS 2018. J Acquir Immune Defic Syndr. 2019;82(suppl 3):S299–S304.
34. Packel L, Fahey C, Njau P, et al. Implementation science using proctor's framework and an adaptation of the multiphase optimization strategy: optimizing a financial incentive intervention for HIV treatment adherence in Tanzania. J Acquir Immune Defic Syndr. 2019;82(suppl 3):S332–S338.
35. Eisinger RW, Dieffenbach CW, Fauci AS. Role of implementation science: linking fundamental discovery science and innovation science to ending the HIV epidemic at the community level. J Acquir Immune Defic Syndr. 2019;82(suppl 3):S171–S172.
36. Klein KJ, Sorra JS. The challenge of innovation implementation. Acad Manage Rev. 1996;21:1055–1080.
37. Klein KJ, Conn AB, Sorra JS. Implementing computerized technology: an organizational analysis. J Appl Psychol. 2001;86:811–824.
38. Helfrich CD, Weiner BJ, McKinney MM, et al. Determinants of implementation effectiveness: adapting a framework for complex innovations. Med Care Res Rev. 2007;64:279–303.
39. Weiner BJ, Lewis MA, Linnan LA. Using organization theory to understand the determinants of effective implementation of worksite health promotion programs. Health Educ Res. 2009;24:292–305.
40. Jacobs SR, Weiner BJ, Reeve BB, et al. Determining the predictors of innovation implementation in healthcare: a quantitative analysis of implementation effectiveness. BMC Health Serv Res. 2015;15:6.
41. Turner K, Trogdon JG, Weinberger M, et al. Testing the organizational theory of innovation implementation effectiveness in a community pharmacy medication management program: a hurdle regression analysis. Implement Sci. 2018;13:105.
42. Lewis MA, Harshbarger C, Bann C, et al. Effectiveness of an interactive, highly tailored “video doctor” intervention to suppress viral load and retain patients with HIV in clinical care: a randomized clinical trial. Under Review. [epub ahead of print].
43. Lewis MA, Harshbarger C, Bann C, et al. Positive Health Check evaluation: a type 1 hybrid design randomized trial to decrease HIV viral loads in patients seen in HIV primary care. Contemp Clin Trials. 2020;96:106097.
44. Emmons KM, Weiner B, Fernandez ME, et al. Systems antecedents for dissemination and implementation: a review and analysis of measures. Health Educ Behav. 2012;39:87–105.
45. Martinez RG, Lewis CC, Weiner BJ. Instrumentation issues in implementation science. Implement Sci. 2014;9:118.
46. Lewis CC, Fischer S, Weiner BJ, et al. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015;10:155.
47. Miller ER, Rollnick S. Motivational Interviewing. 3rd ed. New York, NY: Guildford Press; 2013.
48. Fisher JD, Fisher WA. Changing AIDS-risk behavior. Psychol Bull. 1992;111:455–474.
49. Prochaska JO, Velicer WF. The transtheoretical model of health behavior change. Am J Health Promot. 1997;12:38–48.
50. Zulkiewicz BA, Burrus O, Harshbarger C, et al. Identifying implementation strategies that address barriers and facilitate implementation of digital interventions in HIV primary care settings: results from the pilot implementation of positive health check. AIDS Behav. 2021;25:154–166.
51. Garner BR, Hunter BD. Examining the temporal relationship between psychological climate, work attitude, and staff turnover. J Subst Abuse Treat. 2013;44:193–200.
52. Garner BR, Hunter BD, Modisette KC, et al. Treatment staff turnover in organizations implementing evidence-based practices: turnover rates and their association with client outcomes. J Subst Abuse Treat. 2012;42:134–142.
53. Shea CM, Jacobs SR, Esserman DA, et al. Organizational readiness for implementing change: a psychometric assessment of a new measure. Implement Sci. 2014;9:7.
54. Jacobs SR, Weiner BJ, Bunger AC. Context matters: measuring implementation climate among individuals and groups. Implement Sci. 2014;9:46.
55. Proctor EK, Landsverk J, Aarons G, et al. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Pol Ment Health. 2009;36:24–34.
56. PersonAlytics. State-Of-The-Art Small Sample Analytics. RTI International. 2020. Available at: Accessed June 27, 2021.
57. R: A Language and Environment for Statistical Computing. 2020. Available at: Accessed June 27, 2021.
58. Stasinopoulos MD, Rigby RA, Heller GZ, et al. Flexible Regression and Smoothing: Using GAMLSS in R. Boca Raton, FL: CRC Press; 2017.
59. Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In: Bryman A, Burgess R, eds. Analysing Qualitative Data. London, UK: Routledge; 1994:173–194.
60. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37–46.
61. Warner L, Klausner JD, Rietmeijer CA, et al. Effect of a brief video intervention on incident infection among patients attending sexually transmitted disease clinics. PLoS Med. 2008;5:e135.
62. Glasgow RE, Emmons KM. How can we increase translation of research into practice? Types of evidence needed. Annu Rev Public Health. 2006;28:413–433.
63. Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:1–14.
64. Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22.
65. Horvath KJ, Walker T, Mireles L, et al. A systematic review of technology-assisted HIV testing interventions. Curr HIV/AIDS Rep. 2020;17:269–280.
66. Curran GM, Bauer M, Mittman B, et al. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50:217–226.
67. Landes SJ, McBain SA, Curran GM. Reprint of: an introduction to effectiveness-implementation hybrid designs. Psychiatry Res. 2020;283:112630.

HIV; implementation effectiveness; web-based interventions; computer-based interventions; video interventions; medication adherence

Copyright © 2022 The Author(s). Published by Wolters Kluwer Health, Inc.