WOLPIN, SETH PhD, MPN, RN; NGUYEN, H. Q. PhD, RN; DONESKY-CUENCO, DORANNE DNSc, RN, FAAN; CARRIERI-KOHLMAN, VIRGINIA DNSc, RN, FAAN; DOORENBOS, ARDITH RN, PhD, FAAN
Eliciting patient-reported outcomes (PROs) such as symptom and quality-of-life information is an important component of medical and nursing care processes. Traditional approaches, which have been largely paper based, present inherent inefficiencies. Most notable is the lack of real-time information delivery where interventions can be tailored and quickly provided in response to worsening health. Another disadvantage with paper approaches is the inability to deliver a prompt such as an audible alarm to the patient as a reminder to record PROs. In addition, previous research has demonstrated that the use of traditional paper-based diaries often results in diary hoarding, where health information is entered immediately prior to meeting with health practitioners/researchers.1
Early technology-based efforts have equipped patients with electronic mobile devices for reporting PROs. These devices may be programmed with electronic prompts such as audible alarms, vibration alerts, text messages, or multimedia-based messages reminding patients to enter their data. In a comparison of compliance with logging pain levels using paper and electronic diaries, Stone et al1 found that electronic mobile devices (Palm computers) with audible prompts for missed entries produced a 94% compliance rate. In contrast, compliance for recording pain at prespecified times using a paper diary was only 11%. Despite the benefits of early electronic devices, one limitation was a lack of wireless capabilities and an inability to report real-time results. Instead, these devices frequently had to be "docked" into a phone cradle or desktop computer cradle with results transmitted in one batch at the end of the day, decreasing the ability of the healthcare team to respond in a timely fashion to symptom exacerbations.
Emerging information and communication technologies are providing new ways to collect PROs. For example, Web-enabled phones and other mobile devices can be used by patients to enter PROs, which are then transmitted in real time to a Web server or other information stores. This information could be analyzed by providers or even autonomously by expert systems. With synchronous connections to a server, such devices have the potential for delivering real-time electronic prompts and tailored feedback.2,3 Furthermore, these devices may decrease diary hoarding by prompting patients to log their information and moving assessment, intervention, and evaluation beyond traditional care settings. This proliferation of wireless networks and ubiquitous sensing may allow for more ecologically valid assessments.4
We recently reported the primary results of a study where patients with chronic obstructive pulmonary disease (COPD) were given mobile devices to log their exercise and symptom information on a daily basis as part of a 6-month Internet-based dyspnea self-management intervention.5 We expected that electronic prompts for data entry on mobile devices would result in timely entry of symptom and exercise data, however, an early analysis showed that patients hoarded their diary entries, entering their exercise and symptom data sometimes days after the prompt was delivered.6
Our earlier analysis was descriptive in nature and did not explore factors associated with delays in response time. A better understanding of how patient characteristics, prompt types, and timing influence reporting behavior could inform the development of more effective protocols for collection of PROs using mobile devices. The overall objective of this analysis was to examine the relationship between response to automated prompts on a mobile device for symptom and exercise data with patient characteristics, prompt types, and other temporal factors. The primary research question was: Did the effectiveness of prompts decrease over 6 months? Second, were sociodemographic characteristics (age, sex, education, living situation, computer skills), prompt type (exercise vs symptom), and temporal characteristics (weekend vs weekdays) associated with response time?
The data used for this analysis were from a randomized clinical trial to compare the effects of an Internet-based dyspnea self-management program to a face-to-face program in patients with COPD. This research study was approved by the institutional review boards at both sites and was registered with ClinicalTrials.gov (NCT00102401). The methods for the primary study have been reported in detail elsewhere.5 Methods relevant to the interpretation of mobile device usage patterns are presented below in the Procedures.
Participants were recruited from two research sites, one at the University of California, San Francisco, and the other at the University of Washington in Seattle using a combination of Web-based and off-line sources as detailed in prior work.5 The inclusion criteria were (1) a diagnosis of COPD and clinically stable for at least 1 month; (2) spirometry results showing at least mild obstructive disease defined as ratio of postbronchodilator forced expiratory volume in 1 second (FEV1) to forced vital capacity (FVC) of less than 0.70 with FEV1 of less than 80% predicted or FEV1/FVC ratio of less than 0.60 with FEV1 of greater than 80% predicted; (3) activities of daily living limited by dyspnea; (4) actively using a computer and Internet with a Windows operating system (Microsoft, Redmond, WA); (5) oxygen saturation of greater than 85% on room air on less than 6 L/min of nasal oxygen at the end of a 6-minute walk test. Participants were excluded if they (1) had any active symptomatic illness (eg, cancer, heart failure, ischemic heart disease with known coronary artery or valvular heart disease, psychiatric illness, and neuromuscular disease); (2) participated in a pulmonary rehabilitation program in the last 12 months; or (3) were currently participating in more than 2 days of supervised maintenance exercise.
Participants were provided with a Blackberry 680 mobile device (Research in Motion, Waterloo, Ontario, Canada) and oriented to the device in person by a research nurse. They were encouraged to play an electronic game to increase their comfort with the navigation widgets. The orientation included an assessment of participants' exercise plans as that dictated the number of exercise prompts that would be sent. Prompts were sent to the mobile device at a time that was agreeable to participants. Participants received up to three daily prompts (symptom, strengthening exercise, endurance exercise), with each prompt being accompanied by an audible alarm and an e-mail message (described below). All participants received the daily symptom prompt, but the number of exercise prompts varied based on frequency of planned exercise. This frequency also changed over the course of the study, depending on whether patients had illnesses that interfered with exercise or were ramping up their exercise program. Regular communication between the nurse and participant allowed adjustments to scheduling of prompts. Prompts were "pushed" to the devices in the form of an e-mail message with subject lines that indicated whether it was for an exercise or symptom survey; the e-mail message contained a hyperlink that directed patients to secure Web pages where they entered data about their exercise or symptoms. Depending on question branching within the data forms, participants performed as many as 30 unique actions using the scroll wheel to enter their symptom and exercise data.
Data were cleaned before transformation and analysis. Prompts that were sent to the mobile devices during the initial teaching session between the research nurse and the patient were excluded. A total of 26 patients were issued mobile devices. However, only those who completed the 6-month study were included in the analysis (n = 19). Because some patients were scheduled late for their final 6-month visit and continued to use the mobile device, prompts that were generated after 180 days from initial enrollment were removed from the data set, thus making our analytical cutoff 180 days. Descriptive summaries of the response times were generated. Outliers were included in this summary since they are informative about usage patterns.
Responses submitted within a 24-hour time frame were determined to have the greatest clinical value since research nurses could intervene in a timely manner, and recall was less likely to be affected. Thus, we focused the analysis for our primary research question on this subset of data. A Pearson correlation test was used to determine the relationship between length of time in the study and response delay.
For our secondary research question, parametric and nonparametric tests were used to explore the association between demographic characteristics, prompt type, and other temporal factors with response delay. Independent t tests or χ2 tests were used to analyze continuous and categorical variables, respectively. For these analyses, outliers with respect to response delays were removed from the data set. To accomplish this, all response delay scores were converted to z scores. A total of 89 responses with z scores greater than 3.0 were removed from the data set; these scores belonged to five patients and ranged from 218 to 911 hours, with an interquartile range of 344 hours. Participants were dichotomized into slow and fast responders based on the median response time of 7.56 hours.
A total of 19 patients with moderate to severe COPD were included in the analyses. Fewer than half (42%) were female, and the mean age was 68 (SD, 8) years. A majority of participants (79%) self-reported their computer skills as intermediate or advanced, with four participants describing themselves as beginners. Seven participants (37%) reported living alone, and 12 reported living with a spouse or with others (63%).
Summary of Prompts and Responses
As represented in Table 1, a total of 7474 prompts were sent to 19 participants, resulting in a mean of 394 prompts per participant. Averaged over the 180 days, participants received 2.1 prompts per day, with participants responding to 78% of prompts. With respect to time, the average delay was 25.2 (SD, 64) hours, with a range of 6 minutes to 38 days. The distribution of responses was found to be positively skewed (8.2) with a peaked distribution as evidenced by a kurtosis of 85.2. Results from the Kolmogorov-Smirnov test found that the distribution differed significantly from a normal distribution (0.35, df = 5832, P < .01). The median response delay was 7.76 hours, with an interquartile range of 19.93 hours.
Primary Question: Did the Effectiveness of Prompts Decrease Over 6 Months?
A positive, albeit marginal, correlation was found between length of time in the study and delay in responding to prompts, r = 0.15, n = 4421, P < .01. The scatter plot (Figure 1) shows that as participant days in the study increased, the response delay became longer and more variable. This finding provides support for our primary research question.
Secondary Question: Were Sociodemographic Characteristics (Age, Sex, Education, Living Situation, Computer Skills), Prompt Type (Exercise vs Symptom) and Temporal Characteristics (Weekend vs Weekdays) Associated With Response Time?
Our analysis found that slow responders were more likely to be older-in their early 70s (P < .01). As indicated in Table 2, we also found that slow responders were more likely to be female (P < .01). A higher percentage of slow responders reported living with other people compared with fast responders (68% vs 48%, P < .01). Computer experience also influenced response time, with participants reporting intermediate or advanced computer skills responding faster than beginners (91% vs 83%, P < .01) (Table 2).
However, prompt type (exercise vs symptom) was not associated with response times (P = .567). In reporting what type of exercise was performed, slowest responders were most likely to report not exercising, whereas fast responders were more likely to report exercising (P < .01). Slow responders were marginally more likely to respond on a weekend (P = .044).
We found only a modest relationship between duration of monitoring and delay in submission of exercise and symptom data in response to electronic prompts from a mobile device in patients with COPD. Participants who were younger, were male, lived alone, had better computer skills, and reported that they exercised were more likely to have a shorter response delay.
There are several possible reasons for the modest correlation between timeliness of response and length of participation in the study. Participants may have been more likely to respond sooner early in the study because of novelty effects and an eagerness to report their information. However, as the study progressed, the novelty may have worn off along with participant commitment to exercise. Drop-off in usage over the course of any technology-mediated study is common.6-8 The complex interactions between usability, novelty effects, and dynamics of behavior change and reporting of those behaviors may also have resulted in the heterogeneous response to the electronic prompts.
The mobile device (Blackberry PDA) used in this study presented notable usability challenges that could have contributed to response delays. Participants had to navigate as many as 30 steps to log their symptom or exercise data.9 We received anecdotal reports of participants learning skip patterns and biasing their self-report to avoid having to respond to queries about six symptoms. It is also possible that the prompts may have caused confusion in the manner in which they queued on the mobile device. For example, if participants did not open and/or delete a message prompt, additional prompts accumulated in their inbox, thus creating confusion as to which prompt for which period they were answering, this could be a possible explanation for some of the outliers we found. Other studies have also found that usability can negatively affect compliance with reporting PROs in electronic format.10 The static nature of the prompting messages as well as the lack of positive reinforcement or messaging with each submission may erode a participant's responsiveness to electronic prompts.
Because of the small sample size, we were unable to examine multivariate models to fully understand the independent effects of demographic factors, health status, and prompt characteristics on response delay. Nevertheless, these preliminary results suggest that several baseline characteristics and process variables should be considered in future studies. Individuals who are older and female and have limited computer skills may require more training. That patients who lived with others were slower to respond to prompts than those living alone was an unexpected finding; it is possible that they had less free time or less need for interaction or that other family members took possession of the electronic devices for gaming or other purposes. Consideration should also be given so that patients who are unable to perform a behavior that they had committed to doing, for example, exercise, are still motivated to log this information. Similarly, the addition of tailored, automated, motivational messages could help to keep patients engaged in both exercise activities and also responding to prompts.11 Another area needing exploration is providing greater flexibility and control to users in terms of being able to suspend response to prompts1 and the choice of audio or video reminders since the use of audio alarms has been positively associated with compliance.12
There are a number of current and emerging technologies that overcome some of the challenges we identified during the study. Notable are reliable tools (accelerometers and other sensors) that automatically track physical activity and exercise behavior.13,14 These tools work to reduce the need to manually self-report symptoms and activities and potential negative usability issues that could have plagued reporting behavior in our study. A complementary set of technologies include context-based sensors that allow a richer understanding of multiple factors that influence behavioral patterns. In addition to more sophisticated sensors, we expect to see advances in more usable interfaces for mobile devices and attention to expert assessment systems. Such systems would use algorithms and theory-based messaging to deliver tailored assessments including motivational messaging and appropriate alerts to patients and/or health team when individualized threshold values are reached.
We have learned important lessons from this early implementation of using mobile devices to prompt patients to self-report symptom and exercise information collection. In future investigations, attention to usability issues of mobile devices is paramount as it can impact adoption and diffusion rates. We also recommend that investigators research and test strategies to keep people with chronic illness stay motivated to self-report symptoms and exercise behavior. This is particularly relevant for older adults who are living with others, are female, exercise less, and have limited computer skills. We are hopeful that emerging technologies will address some of these gaps; however, it is important that balance is kept with respect to these technologies and their potential value for patients.
1. Stone AA, Shiffman S, Schwartz JE, Broderick JE, Hufford MR. Patient compliance with paper and electronic diaries. Control Clin Trials
2. Lukasiewicz M, Fareng M, Benyamina A, Blecha L, Reynaud M, Falissard B. Ecological momentary assessment in addiction. Expert Rev Neurother
3. Richardson CR, Mehari KS, McIntyre LG, et al. A randomized trial comparing structured and lifestyle goals in an Internet-mediated walking program for people with type 2 diabetes. Int J Behav Nutr Phys Act
4. Smyth JM, Stone AA. Ecological momentary assessment research in behavioral medicine. J Happiness Stud
5. Nguyen HQ, Donesky-Cuenco D, Wolpin S, et al. Randomized controlled trial of an internet-based versus face-to-face dyspnea self-management program for patients with chronic obstructive pulmonary disease: a pilot study. J Med Internet Res
6. Litcher-Kelly L, Kellerman Q, Hanauer SB, Stone AA. Feasibility and utility of an electronic diary to assess self-report symptoms in patients with inflammatory bowel disease. Ann Behav Med
7. Wantland DJ, Portillo CJ, Holzemer WL, Slaughter R, McGhee EM. The effectiveness of Web-based vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes. J Med Internet Res
8. Eysenbach G. The law of attrition. J Med Internet Res
9. Nguyen HQ, Wolpin S, Chiang KC, Cuenco D, Carrieri-Kohlman V. Exercise and symptom monitoring with a mobile device. AMIA Annu Symp Proc
10. Hardwick ME, Pulido PA, Adelson WS. The use of handheld technology in nursing research and practice. Orthop Nurs
11. Rogers EM. Diffusion of Innovations
. 4th ed. New York: The Free Press; 1995.
12. Morren M, Dulmen SV, Ouwerkerk J, Bensing J. Compliance with momentary pain measurement using electronic diaries: a systematic review. Eur J Pain
13. He J, Li H, Tan J. Real-time daily activity classification with wireless sensor networks using Hidden Markov Model. Conf Proc IEEE Eng Med Biol Soc
14. Tanaka S, Motoi K, Nogawa M, Yamakoshi K. A new portable device for ambulatory monitoring of human posture and walking velocity using miniature accelerometers and gyroscope. Conf Proc IEEE Eng Med Biol Soc
© 2011 Lippincott Williams & Wilkins, Inc.