LEARNING OUTCOMES
In this study, we aim to
Examine the feasibility and acceptability of a digital psychosocial intervention, Wysa for Return to Work, for individuals with a work-related injury, by measuring study qualification rate and app usage parameters; and
Evaluate the demographics of these individuals, alongside their app usage, in order to assess their engagement and retention on the program.
A work-related injury or illness is broadly defined as any psychological or physical harm sustained in the course of one’s work duties.1,2 3 4
Approximately 80% percent of WC compensation claim costs in the United States are attributed to the 5% to 10% of injured workers who suffer long-term disability and are on long-term claims.5,6 5 7–10 8,11 12,13 14,15
Effective psychosocial resources have been shown to promote recovery from illness and injury.16–18 19–21
Digital psychosocial interventions for recovery and resilience can be delivered through AI-based conversational agents (CAs).22,23 24 25
An analysis of user feedback on Wysa, an AI-based CA, has indicated high levels of acceptability, with users highlighting its nonjudgmental approach and easy conversational style,26 27 28,29
In this study, we examine the feasibility and acceptability of a digital psychosocial intervention, Wysa for Return to Work, for individuals with a work-related injury. This study evaluates the demographics of these individuals, alongside their app usage, in order to understand engagement and retention on the program. The findings of this study could help identify individuals who would be more inclined to use digital interventions and who may benefit from them.
METHODOLOGY
Ethical Considerations
The Wysa app is publicly available as an app on the Android and iOS app stores. It has been designed to prioritize safety, privacy, and security by design. There is no user registration required, and no personally identifiable information is asked at any time during app use.
The study involved analyzing app usage data of injured workers on WC claims administered by Travelers, a national insurance carrier. This study only utilizes deidentified data after receiving explicit consent from study participants.
The participants gave explicit consent on the app to the collection of data and its aggregation, and to the app’s terms of service and privacy policy. The study participants were also informed about how they could exercise their rights to restrict the processing of their data for research purposes.
For ethical and privacy reasons, neither the authors nor Travelers had access to user messages or conversations. Only anonymized app usage data were extracted for this research, for 15 months from March 2021 to June 2022. The study data set was deidentified using one-way cryptographic functions. User data were adequately secured according to the organization’s privacy, security, and safety policies.
Study Design
This was a single-armed, prospective cohort pilot study that offered a resilience and recovery-based intervention to the claimants as part of their claim. Participants were enrolled throughout the study period, from March 2021 to June 2022.
Participants
Eligibility Criteria and Recruitment
Participants were recruited from a cohort of WC claims after an absence from work due to an injury. The injury could be of various types, from low severity with minimal lost time from work, to complex claims with longer durations of lost time and potentially long recovery timeframes. Individuals with catastrophic injuries were excluded from this study.
In the course of a regular claims journey, demographic information is collected from the claimants, and their assigned nurse case manager asks them five psychosocial screening questions. Claimants who received a positive score (as defined by the criteria given below in Fig. 1 ) on any one of the five questions qualified for the study, and were recommended the use of the Wysa RTW program by the nurse case manager. Based on the number of questions they scored positively on, they each had a total score ranging from 1 to 5.
FIGURE 1: Psychosocial risk-factor screening questions and required response flow to qualify for the study.
Qualified participants were randomly assigned to either the control group or the test group (see Fig. 2 below).
FIGURE 2: Participant flow, from screening to onboarding on the Wysa app.
Intervention
All study participants received a complimentary, 1-year subscription to Wysa for Return to Work, a novel, customized version of Wysa to facilitate recovery management and RTW. This version is not available commercially. After the 12-week RTW program, the remaining period provided the users with continued support in the form chosen by the user.
Wysa is a multicomponent intervention that uses AI-based technology to deliver therapeutic content such as cognitive behavioral therapy, dialectical behavioral therapy, motivational interviewing, mindfulness training, deep-breathing techniques, and sleep meditations through a CA.30,31
Using behavioral activation and resilience principles, this RTW version of Wysa supported the user with building a recovery mindset. The AI-enabled free-text CA offered users a 12-week program where the CA acted as a companion in the participant’s journey of recovering and returning to work, and the goal of the program was to improve recovery outcomes, help in recovery management, and reduce time taken to RTW.
Data
Data were collected from March 23, 2021, to June 29, 2022, and were obtained from two sources: participants’ self-reported information during screening and nurse case management, collected by Travelers, which included details about their demographics (gender, age), injury (part of body/nature of injury), and psychosocial screening responses; and app usage reports from Wysa, which included user onboarding dates and assessment data, energy scores reported, and sessions/tools completed and their dates.
No personally identifiable data were shared with Wysa by the Travelers study team and vice versa. Screened participants with missing data on gender, and those with “undetermined” (unclassified) injury severity levels, were excluded from the analysis.
Analysis
For the analysis, participants were stratified on the basis of the following demographic criteria: gender (male, female), age range (17–24, 25–40, 41–56, and 57–75 years), and injury severity level (low, medium, and high). Participants were also stratified on the basis of the number of psychosocial risk-factor indicators they scored positively on.
The primary study objectives were related to the feasibility and acceptability of introducing a digital psychosocial intervention (Wysa) for WC claimants. This analysis aims to understand the individuals who would be inclined to use a digital intervention like Wysa, and infer the feasibility of such a support mechanism. Significance was set a priori at P < 0.05. Statistical analyses were performed using R (v4.2.0).
Objective 1: Feasibility
The first outcome of this study assessed the need for a psychosocial intervention like Wysa in the target population through the study recruitment rate, evaluated as the proportion of screened participants who qualified for the study by scoring positively on at least one of the psychosocial risk-factor indicator questions presented to them at screening. The likelihood of injury types in the qualified group was also examined.
To determine whether any significant differences existed in recruitment rates across demographic strata, the rate was calculated on the basis of gender, age-range, and severity. We conducted a chi-squared test of proportions for the significance test across genders, with multiple pairwise tests between different age ranges and severity levels, and adjusted the P values for the results found positive using the Bonferroni correction.
Objective 2: Acceptability
To assess the acceptability and likelihood of usage of a psychosocial intervention like Wysa, four key outcomes were measured: (1) onboarding rate, (2) retention, (3) engagement, and (4) change in user-reported mood.
The onboarding rate was measured as the percentage of qualified participants who onboarded on the Wysa RTW program. To determine whether any significant differences existed in onboarding rates across demographic strata, the onboarding rate was calculated on the basis of gender, age range, and severity also. A chi-squared test of proportions was utilized for the significance test across genders and psychosocial risk-factor indicators, and multiple pairwise tests between different age ranges and severity levels were conducted, and the P values were adjusted for the results found positive using the Bonferroni correction.
During onboarding, participants also reported what they were looking forward to doing after recovery, the challenges that were affecting their journey, the activities that gave them joy, and how much time they could spare for their daily interactions with the app, and these responses were then used to customize their app experience. The frequencies of these responses were analyzed.
User rate of engagement was defined as the proportion of participants who engaged in at least one Wysa intervention after onboarding to the program. The rate of engagement was calculated across genders, age ranges, severity levels, and psychosocial risk factor indicators.
Descriptive statistics (total sessions, average sessions) of the different types of interventions undertaken by users helped us determine the most commonly used tools. The difference between average sessions across genders, age ranges, severity levels, and total psychosocial risk factor indicators was also tested. The Wilcoxon signed rank test was used for the significance test across genders, age ranges, severity levels, and psychosocial risk factor indicators, since the data were not normally distributed (as per the Shapiro-Wilk normality test).
Retention was defined as the number of days spent on the app before either disengagement or the end of the study period (June 29). Participants who stopped using the app at least a month before June 29 were considered to have disengaged. The participants who onboarded after April 29, 2022, 60 days before the end of the study period were not included. The mean number of retention days was calculated for those who disengaged. The mean difference between retention days across genders, age ranges, severity levels, and total psychosocial risk factor indicators was tested. The Wilcoxon signed rank test was used for these significance tests, since the data sets across genders, age ranges, severity levels, and psychosocial risk factor indicators were not normally distributed (as per the Shapiro-Wilk test for normality). Ages 17 to 24 years and low severity levels were excluded from this analysis because of the low number of users in these cohorts (12 and 2, respectively).
When the participants accessed Wysa through the “talk” button on the home screen, or they started a morning check-in, they were asked about their energy levels and were provided with a visual sliding scale where they could rate their energy levels from 0 to 100 points. Energy scores are rated “high” if they are above 65 points, “neutral” if they are between 35 and 65 points, and “low” if they are below 35 points. We calculated the average energy score rating and (after filtering out users with only one reported energy score, n = 51) the average rate of change of participant’s energy scores over time.
RESULTS
Objective 1: Feasibility
Forty-one percent (5150/12663) of the screened population qualified for the study, having scored positively on at least one of the psychosocial risk-factor indicator questions presented to them at screening (Table 1 ).
TABLE 1 -
Numbers of participants who were screened, qualified, and onboarded across demographic stratification
Demographic stratification
Screened
Qualified (Test) for Wysa RTW
Onboarded
Gender
Men
8166
1425
96
Women
4497
816
94
Age range
57–75 years
2979
505
37
41–56 years
4585
767
72
25–40 years
3887
745
67
17–24 years
1157
216
14
Severity levels
High
1673
376
46
Medium
10,870
1845
142
Low
120
20
2
When examined by demographic variables, 42.4% of women (1910/4497) qualified for the study, and 39.6% of men (3241/8166) qualified, and this difference was statistically significant (P = 0.002423; α = 0.05); that is, women were found to be more in need of psychosocial interventions like Wysa (see Table, Supplemental Digital Content 1, https://links.lww.com/JOM/B235
A total of 42.7% (495/1157) of people aged 17 to 24 years qualified for the study, as compared with 40.6% of those aged 25 to 40 years (1579/3887), 39.9% of those aged 41 to 56 years (1830/4585), and 38% (1220/2979) of those aged 57 to 75 years.
A total of 48.5% (813/1673) of the screened population with high severity injuries qualified for the study, compared with 39.5% (4300/10870) of participants with medium severity injuries and 31.6% (38/120) of participants with low severity injuries. Participants with high severity injuries were found to be statistically more in need of psychosocial interventions than those with medium (P = 3.042e-12; α = 0.05) and low (P = 0.0004783; α = 0.05) severity injuries (see Table, Supplemental Digital Content 2, https://links.lww.com/JOM/B236
Objective 2: Acceptability
From those who downloaded the intervention, 190 users (93.5%) successfully onboarded onto Wysa, and 88.4% of all onboarded participants (168/190) engaged with one or more interventions, with an average of 10.8 sessions utilized across participants. Mood check-ins, mindfulness-based tools, behavioral activation interventions, sleep tools, and thought recording were the most commonly used tools across the population (Fig. 3 ).
FIGURE 3: The total number of sessions completed by all users across tools on the app.
Participants whose last day of app usage was 1 month before the end of the study period (n = 165) were retained for an average of 35 days. Participants with continued app usage (n = 10) were retained on the app for an average of 153.9 days.
Women engaged in a greater number of average sessions as compared with men, and this difference was statistically significant (P = 0.02239, α = 0.05) (see Table, Supplemental Digital Content 3, https://links.lww.com/JOM/B237
Participants aged 57 to 75 years were retained for an average of 55.3 days on the program, with the difference between those aged 57 to 75 years and those aged 25 to 40 years found to be statistically significant (P = 0.025977; α = 0.05; see Table, Supplemental Digital Content 4, https://links.lww.com/JOM/B238
Participants with high severity injuries used the RTW program for 53.1 days on average, and the difference in retention compared with those with medium severity injuries was found to be statistically significant (P = 0.006074; α = 0.05; see Table, Supplemental Digital Content 5, https://links.lww.com/JOM/B239
When stratified by demographic criteria (n = 190), men formed 50.5% and women were 49.4% of those who onboarded. Women onboarded at a higher rate than men, and this difference was statistically significant (P = 0.000127; α = 0.05; see Table, Supplemental Digital Content 6, https://links.lww.com/JOM/B240
Individuals with more than three psychosocial risk factors retained for a greater number of days than those with fewer risk factors (68.2 days, P = 4.749e-07) and engaged in a greater number of sessions per user (18.9 sessions; P = 0.002153).
Participants aged 41 to 56 years formed 37.8% (72/190) of the onboarded population, followed by age ranges 25 to 40 years at 35.2% (67/190), 57 to 75 years at 19.4% (37/190), and 17 to 24 years at 7.3% (14/190).
Participants with medium severity injuries formed 74.7% (142/190) of the onboarded population, followed by those with high severity injuries at 24.2% (46/190), and low severity injuries at 1% (2/190). Participants with high severity injuries were found to be more inclined than those with medium severity injuries toward using digital interventions like Wysa, because of their statistically higher onboarding rates (P = 0.005447; α = 0.05; see Table, Supplemental Digital Content 7, https://links.lww.com/JOM/B241
A total of 90.4% women engaged in an intervention after onboarding onto the app, as compared with 86.4% men. Participants in the 57 to 75 age group had the highest engagement rate across age groups (94.5%), with participants in age groups 41 to 56 years and 25 to 40 years close behind at 90.2% and 88%, respectively.
In the course of onboarding onto the app, 38% of the respondents stated that their recovery was taking longer than they had hoped, 76% stated that they were looking forward to getting back to work after their recovery, 78% wanted to feel like themselves again, and 46% wanted to feel independent. The key challenges mentioned by the respondents were pain (81%), sleep (42%), fatigue (39%), frustration (61%), fear (33%), and sadness (26%). They also stated that a lack of motivation (23%) and confidence (18%) was affecting their recovery.
Out of a total of 357 energy score ratings given by 99 users, 54.8 points was the average energy score per user. On average, participants’ energy scores increased by 3.58 points over time (SD = 13.14 points).
DISCUSSION
In this pilot study, we examined the feasibility and acceptability of a digital psychosocial intervention, Wysa for Return to Work, which was intended to facilitate and expedite recovery for individuals with a work-related injury and receiving WC benefits.
In the study, 41% of the WC claimants qualified by screening positively on at least one psychosocial risk-factor question, demonstrating a high need for psychosocial interventions among injured workers. This finding is supported by research that documents the large indirect toll on the lives of claimants, caused by disruptions to sleep and daily activities.13 32 33 34
Individuals who screened positively on three or more psychosocial risk factor indicators onboarded at a higher rate than those with fewer risk factors, retained on the RTW program for much longer, and also engaged in a higher number of sessions per user, indicating that people with a greater need for psychosocial support were also more inclined to seek the support they needed through the program. Furthermore, participants with high severity injuries, which are associated with greater psychological morbidity,35 26
For users who completed usage within the study period, the mean engagement in the study was 10.8 sessions and the mean retention period was 35 days, and over 153.9 days of app usage for those who continued utilization. Evidence suggests an average retention period between 4 and 16 days in most digital health applications.36,37 38 user engagement .39
Greater engagement with CAs has also been shown to lead to greater reductions in psychological distress,31,40–43
According to a systematic review, a common aspect of successful RTW interventions was behavioral activation, psychoeducation, and a focus on the RTW context.44 45,46 47 48–50
These findings suggest that a digital mental health intervention is feasible and acceptable for an RTW population and that a fully powered study can help in understanding the extent of the impact that a digital mental health intervention could have on RTW, recovery, and lost time.
Strengths and Limitations
The primary strength of this feasibility study is that, to our knowledge, it is the first investigation to incorporate a digital psychosocial intervention into a WC program for injured workers. The primary limitations are that there was no true control arm/comparison group, which reduced our ability to establish causality for the potential factors responsible for retention and engagement, and delineate variables around type and length of injury as a comparative. The intervention was also only available in English during the study.
Receiving direct feedback from the users of the program, and being able to follow up with them, would help understand their experience better and customize psychosocial support services to fit their needs, and having larger sample sizes across demographic stratifications would help establish a greater degree of confidence in the findings.
Furthermore, information about the app was provided to the qualified population at the start of their claim journey, and there might have been individuals who needed the intervention but did not onboard because of the information overload that is common at this stage.
CONCLUSIONS
In summary, this pilot study explored the feasibility and acceptability of an AI-led digital psychosocial intervention, Wysa for Return to Work, which was intended to facilitate and expedite recovery for individuals with a work-related injury and receiving a WC claim in an accessible and affordable way. This study highlights the ability to attain high retention and engagement among injured workers with an app that uses a digital CA, and offers the potential to improve recovery outcomes for injured workers in a manner that is feasible and scalable to deliver in a WC program.
REFERENCES
1. Lilley R, Davie G, Langley J, Ameratunga S, Derrett S. Do outcomes differ between work and non-work-related injury in a universal injury compensation system? Findings from the New Zealand Prospective Outcomes of Injury Study.
BMC Public Health 2013;13:995.
2. 1904.5 - determination of work-relatedness. Available at:
https://www.osha.gov/laws-regs/regulations/standardnumber/1904/1904.5 . Accessed May 17, 2022.
3. Workers’ Compensation Benefits, Costs, and Coverage.
National Academy of Social Insurance . 2021. Available at:
https://www.nasi.org/research/workers-compensation/workers-compensation-benefits-costs-and-coverage/ . Accessed August 15, 2022.
5. MacEachen E, Kosny A, Ferrier S, Chambers L. The ‘toxic dose’ of system problems: why some injured workers don’t return to work as expected.
J Occup Rehabil 2010;20:349–366.
6. Wickizer TM, Franklin G, Plaeger-Brockway R, Mootz RD. Improving the quality of workers’ compensation health care delivery: the Washington State Occupational Health Services Project.
Milbank Q 2001;79:5–33.
7. Anderson JT, Haas AR, Percy R, Woods ST, Ahn UM, Ahn NU. Clinical depression is a strong predictor of poor lumbar fusion outcomes among workers’ compensation subjects.
Spine 2015;40:748–756.
8. Cancelliere C, Donovan J, Stochkendahl M, et al. Factors affecting return to work after injury or illness: best evidence synthesis of systematic reviews.
Chiropr Man Therap 2016;24:32.
9. Blank L, Peters J, Pickvance S, Wilford J, Macdonald E. A systematic review of the factors which predict return to work for people suffering episodes of poor mental health.
J Occup Rehabil 2008;18:27–34.
10. Caruso G, Kertay L. Part I: psychological factors in delayed and failed recovery and unnecessary disability.
Guides Newsletter 2019;24:3–9.
11. Franche RL, Carnide N, Hogg-Johnson S, et al. Course, diagnosis, and treatment of depressive symptomatology in workers following a workplace injury: a prospective cohort study.
Can J Psychiatry 2009;54:534–546.
12. Dembe AE. The social consequences of occupational injuries and illnesses.
Am J Ind Med 2001;40:403–417.
13. Keogh JP, Nuwayhid I, Gordon JL, Gucer PW. The impact of occupational injury on injured worker and family: outcomes of upper extremity cumulative trauma disorders in Maryland workers.
Am J Ind Med 2000;38:498–506.
14. Boden LI, Galizzi M. Economic consequences of workplace injuries and illnesses: lost earnings and benefit adequacy.
Am J Ind Med 1999;36:487–503.
15. Boden LI, Galizzi M. Income losses of women and men injured at work.
J Hum Resour 2003;38:722–757.
16. Edwards RR, Dworkin RH, Sullivan MD, Turk DC, Wasan AD. The role of psychosocial processes in the development and maintenance of chronic pain.
J Pain 2016;17(9 Suppl):T70–T92.
17. Noorain S, Paola Scaparra M, Kotiadis K. Mind the gap: a review of optimisation in mental healthcare service delivery.
Health Systems 2022;1–34.
18. Singh OP. Closing treatment gap of mental disorders in India: opportunity in new competency-based Medical Council of India curriculum.
Indian J Psychiatry 2018;60:375–376.
19. MacIsaac A, Mushquash AR, Mohammed S, Grassia E, Smith S, Wekerle C. Adverse childhood experiences and building resilience with the JoyPop App: evaluation study.
JMIR Mhealth Uhealth 2021;9:e25087.
20. Wind TR, Rijkeboer M, Andersson G, Riper H. The COVID-19 pandemic: the ‘black swan’ for mental health care and a turning point for e-health.
Internet Interv 2020;20:100317.
21. B. Inkster and Digital Mental Health Data Insights Group (DMHDIG). Early warning signs of a mental health tsunami: a coordinated response to gather initial data insights from multiple digital services providers.
Front Digit Health 2020;2:578902.
22. Cluver L. Solving the global challenge of adolescent mental ill-health.
Lancet Child Adolescent Health 2020;4:556–557.
23. Dalton L, Rapa E, Stein A. Protecting the psychological health of children through effective communication about COVID-19.
Lancet Child Adolesc Health 2020;4:346–347.
24. Shamekhi A, Bickmore T, Lestoquoy A, Gardiner P. Augmenting group medical visits with conversational agents for stress management behavior change. In:
Persuasive Technology: Development and Implementation of Personalized Technologies to Change Attitudes and Behaviors . 2017:55–67. Available at:
https://link.springer.com/chapter/10.1007/978-3-319-55134-0_5 . Accessed December 14, 2022.
25. Hauser-Ulrich S, Künzli H, Meier-Peterhans D, Kowatsch T. A smartphone-based health care chatbot to promote self-management of chronic pain (SELMA): pilot randomized controlled trial.
JMIR Mhealth Uhealth 2020;8:e15806.
26. Malik T, Ambrose AJ, Sinha C. Evaluating user feedback for an artificial intelligence-enabled, cognitive behavioral therapy-based mental health app (Wysa): qualitative thematic analysis.
JMIR Hum Factors 2022;9:e35668.
27. Beatty C, Malik T, Meheli S, Sinha C. Evaluating the therapeutic alliance with a free-text CBT conversational agent (Wysa): a mixed-methods study.
Front Digit Health 2022;4:847991.
28. Leo AJ, Schuelke MJ, Hunt DM, et al. A digital mental health intervention in an orthopedic setting for patients with symptoms of depression and/or anxiety: feasibility prospective cohort study.
JMIR Form Res 2022;6:e34889.
29. Leo AJ, Schuelke MJ, Hunt DM, Miller JP, Areán PA, Cheng AL. Digital mental health intervention plus usual care compared with usual care only and usual care plus in-person psychological counseling for orthopedic patients with symptoms of depression or anxiety: cohort study.
JMIR Form Res 2022;6:e36203.
30. Wysa - Everyday Mental Health. In:
Wysa - Everyday Mental Health . Available at:
https://www.wysa.io/ . Accessed June 27, 2022.
31. Inkster B, Sarda S, Subramanian V. An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study.
JMIR Mhealth Uhealth 2018;6:e12106.
32. Capogna BM, Mahure SA, Mollon B, Duenes ML, Rokito AS. Young age, female gender, Caucasian race, and workers’ compensation claim are risk factors for reoperation following arthroscopic ACL reconstruction.
Knee Surg Sports Traumatol Arthrosc 2020;28:2213–2223.
33. Breslin C, Koehoorn M, Smith P, Manno M. Age related differences in work injuries and permanent impairment: a comparison of workers’ compensation claims among adolescents, young adults, and adults.
Occup. Environ. Med . 2003;60:E10.
34. Mikkelsen MB, Rosholm M. Systematic review and meta-analysis of interventions aimed at enhancing return to work for sick-listed workers with common mental disorders, stress-related disorders.
Occup Environ Med 2018; [Online]. Available at:
https://oem.bmj.com/content/75/9/675.abstract . Accessed July 15, 2022.
35. Sareen J, Erickson J, Medved MI, et al. Risk factors for post-injury mental health problems.
Depress Anxiety 2013;30:321–327.
36. Adjust Global App Trends 2019 - Get free report. Available at:
https://www.adjust.com/resources/ebooks/adjust-global-app-trends-report-2019/ . Accessed August 16, 2022.
37. Pratap A, Neto EC, Snyder P, et al. Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants.
NPJ Digit Med 2020;3:21.
38. Eysenbach G. The law of attrition.
J Med Internet Res 2005;7:e11.
39. Sinha C, Cheng AL, Kadaba M. Adherence and engagement with a cognitive behavioral therapy-based conversational agent (Wysa for chronic pain) among adults with chronic pain: survival analysis.
JMIR Form Res 2022;6:e37302.
40. Ring L, Shi L, Totzke K, Bickmore T. Social support agents for older adults: longitudinal affective computing in the home.
J Multimodal User Interfaces 2015;9:79–88.
41. Fulmer R, Joerin A, Gentile B, Lakerink L, Rauws M. Using psychological artificial intelligence (Tess) to relieve symptoms of depression and anxiety: randomized controlled trial.
JMIR Ment Health 2018;5:e64.
42. Burton C, Szentagotai Tatar A, McKinstry B, et al. Pilot randomised controlled trial of Help4Mood, an embodied virtual agent-based system to support treatment of depression.
J Telemed Telecare 2016;22:348–355.
43. Ly KH, Ly A-M, Andersson G. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods.
Internet Interv 2017;10:39–46.
44. Venning A, Oswald TK, Stevenson J, et al. Determining what constitutes an effective psychosocial ‘return to work’ intervention: a systematic review and narrative synthesis.
BMC Public Health 2021;21:2164.
45. Lyzwinski LN, Caffery LJ, Bambling M, Edirippulige S. Consumer perspectives on mHealth for weight loss: a review of qualitative studies.
J Telemed Telecare 2018;24:290–302.
46. Crane D, Garnett C, Brown J, West R, Michie S. Behavior change techniques in popular alcohol reduction apps: content analysis.
J. Med. Internet Res 2015;17:e118.
47. Baumel A, Muench F, Edan S, Kane JM. Objective
user engagement with mental health apps: systematic search and panel-based usage analysis.
J Med Internet Res 2019;21:e14567.
48. Szinay D, Perski O, Jones A, Chadborn T, Brown J, Naughton F. Perceptions of factors influencing engagement with health and well-being apps in the United Kingdom: qualitative interview study.
JMIR Mhealth Uhealth . 2021;9:e29098.
49. Szinay D, Jones A, Chadborn T, Brown J, Naughton F. Influences on the uptake of and engagement with health and well-being smartphone apps: systematic review.
J Med Internet Res 2020;22:e17572.
50. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis.
Transl Behav Med 2017;7:254–267.
