RYAN, POLLY PhD, RN, CNS-BC; PUMILIA, NICHOLAS J. MA, BS; HENAK, BRANDON BS; CHANG, TINA BBA
Healthcare consumers are assuming increasing responsibility for engaging in healthy behaviors and self-managing chronic health conditions. For decades, small group meetings were one of the primary delivery methods used to impart health information. However, small groups are no longer as well attended as in the past.1,2 People are turning to different sources for information-evidenced by the popularity of the Internet as a source of health information. Healthcare professionals need to explore alternative methods of delivering health information. Technology offers innovative and promising methods of delivering health messages to provide knowledge and potentially facilitate health behavior change.
Nurses assume a primary role in educating persons to engage in health-promotion activities and self-management of diseases. As newer delivery methods are used, nurses continue to play a major role in ensuring that patients are able to use and understand health information. While other disciplines have become increasingly involved in the design and creation of e-delivery products, nurses need to maintain their responsibility for developing educational content and evaluating people's use of e-delivery interventions to ensure achievement of positive health outcomes.
Results of numerous studies support the use of computers for educational purposes3-6; however, use of computers is not without significant problems. "User impatience" and "resistance to learning" are significant obstacles to e-learning.7-10 When using computers, novices prefer to "get something done" rather than "learning how to get something done," and experts presume prior knowledge to the use of the new e-delivery method even when prior knowledge does not apply or work. If use of the e-delivery method is not intuitively apparent, people discontinue its use quickly. Hence, it is critical to determine people's ability to use electronic interventions correctly, in a timely manner, and with high levels of satisfaction11 prior to introducing their use into the clinical arena.12-15
The complexity of electronic interventions is often underreported. Failure to describe the sequencing, timing, and dimensions of electronic interventions hampers research related to intervention efficacy. Without knowledge of the components and processes of an intervention, replication is seriously threatened. Likewise, the processes used during the development of an intervention are often transparent.16,17
Usability testing is one component in the preparation of an electronic intervention. Usability testing is an iterative, systematic process designed to obtain user feedback to improve the speed, accuracy, and satisfaction of e-delivery methods.18 This article describes two types of usability testing used to evaluate a newly developed intervention, Tailored Interactive Computerized Intervention (TICI). The two types of performance usability, "Complete a Task" and "Talk Aloud," were used. This evaluation study was used to determine performance usability of an electronically delivered intervention. The intervention was designed to attenuate osteoporotic bone loss in women during menopausal transition. The intervention was designed to be integrated into women's busy lifestyles. One component of the intervention is presented as an exemplar for the purposes of this discussion, realizing, however, that usability testing is recommended for all dimensions and components of any site or program.
The TICI is a newly developed program designed to attenuate bone loss in women during menopausal transition, with the long-term health goal of preventing or delaying osteoporosis. The intervention, TICI, focuses on increasing women's intake of calcium and vitamin D, increasing weight-bearing exercise, promoting engagement in fall-prevention activities that improve balance, and supporting recommendations for bone health monitoring and medical treatment. (Select dimensions related to calcium and vitamin D intake are used to provide examples for this article.) The TICI is based on the Integrated Theory of Health Behavior Change (ITHBC).19 According to this model, people are more likely to engage in health behaviors when they have the knowledge and beliefs about recommended behavior changes, possess the skills and abilities to self-regulate their health behaviors, and experience social facilitation (influence and support) to change their health behaviors. It is proposed that the combination of knowledge and beliefs, self-regulation abilities, and social facilitation enables persons to engage in health-promoting behaviors. One of the assumptions of the ITHBC is based on the importance of using patient-centered interventions to foster health behavior change.20,21 The TICI is delivered electronically using a combination of technologies to tailor information, provide activities facilitating skill development needed to self-regulate, and offer opportunities for social facilitation.
Both a Web site and handheld computer are used to deliver TICI. The Web site is tailored and designed to increase knowledge and foster health beliefs. Content is tailored to match a woman's recommended calcium intake, actual intake, behavior-specific self-efficacy, outcome expectancy, and goal congruence. The site is sequentially tailored for repeated use. As women change their health behaviors, the Web site is programmed to change, providing different information that matches their newly changed perceptions and behaviors. The handheld computer is used to assist women to develop self-regulation skills (eg, goal setting, self-monitoring, feedback, and decision making) to facilitate health behavior change. The handheld computer is intended for regular use during initiation and early maintenance of health behavior change. Over a period of several months, the handheld computer fosters development of skills and activities needed to learn self-regulating health behaviors. A handheld computer was selected over a desktop computer to foster self-regulation because of its immediate accessibility, enabling women to incorporate use of the handheld computer into their daily lives.22-25
Tailored Web Site
The goal of the Web site is to provide factual information and facilitate changes in health beliefs. Although initially developed prior to the publication of Research-Based Web Design & Usability Guidelines,26 the Web site is consistent with the research-based findings identified in this report. The site opens to a splash page (Figure 1), clearly and simply identifying the purpose of the Web site, inviting women to participate, and suggesting how to use the site. The site respects personal preference as users have access to all major components of the site at all times. Colors, photographs, and content project the image of a site "for women." Throughout the Web site, the major areas of content (calcium and vitamin D, etc) are listed across the top, and dimensions are listed on the left (Figure 2).11 Dimensions listed on the left consist of "General Information" (standardized information applicable to everyone), "All About You" (tailored information beginning with a brief clinical assessment and seamlessly progressing to content matched to each woman), "Helpful Information" (resource information beginning with detailed indexing linked with content; eg, list of calcium supplement or lists of food fortified with calcium), and "Need More Info" (section containing how and where people can obtain additional information and links to other Web sites).
The Web site is tailored, matching informational content to a person's key characteristics. The concepts of ITHBC provide the basis for selection of key characteristics used to tailor the intervention. The site includes content related to current behaviors (ie, current calcium and vitamin D intake) and knowledge and health beliefs (ie, knowledge about osteoporosis, self-efficacy to increase one's intake of calcium and vitamin D, outcome expectancy, and goal congruence).
Tailored interventions consist of five essential components: assessment of a person's key characteristics, message library containing small segments of information, decisional algorithm or logic matching message segments to key characteristics, a delivery method, and feedback.27 Persons using this Web site are assessed for their current calcium intake, outcome expectancy, behavior-specific self-efficacy, and their goal congruence. The information obtained via this assessment is categorized into levels or quantity of the characteristic. A message library contains small segments of information prepared to match the levels for all of the key characteristic. The message library contains numerous permutations and combinations of content. The decisional algorithm is the logic used to match the segments of information to level of the key characteristics (see examples in Table 1 and 2). The logic consists of a number of "if-then" statements. Women with low or very low levels of calcium receive feedback about their current calcium and vitamin D intake, the calcium intake specifically recommended for them, and suggestions on how to increase their intake. For example, "if" a woman has low levels of calcium, "then" she is encouraged to increase her calcium intake by eating foods rich in calcium and specific food choices are suggested, whereas "if" a woman has a very low level of calcium intake, "then" she is encouraged to increase her dietary intake as well as to regularly use supplements28 and she receives specific information for both calcium-rich foods and supplements. The Web site is the method used to deliver this component of the intervention. Feedback is provided in writing and visually through charts, graphs, and pictures. Women receive feedback specific to the levels of their key characteristics. For example, women with high levels of self-efficacy are encouraged to be role models for friends, family, and coworkers. Women with low levels of self-efficacy receive encouraging messages detailing how to make small changes knowing that small changes are more likely to result in success and increase self-efficacy.
The Web site is designed to facilitate social influence and support. The site contains two mechanisms of communication. One method, e-mail, links the woman with a nurse researcher and is intended to serve as a type of social influence (personal recommendations and comments from someone with recognized expertise). The second method of communication is a chat room. This asynchronous chat room is intended to provide social contact and support among participants actively engaging in the behavior change process.
Content experts developed the assessment tools, informational message segments, decisional algorithm, and feedback for use on the site. Professional programmers were integral to the development of the Web site. The Web site is based on recognized principles such as using simple pages, using multiple frames with scrollable windows, reusing graphic images, having easy and intuitive linkages throughout the site, and so on. Macromedia Dreamweaver (Adobe Systems, San Jose, CA) was the basic software used to create the Web site and is used for Web site presentation, navigation, and uploading. Women are directed to complete a five-question assessment as they enter the site. Content of the Web site is matched to women's responses to the assessment questions about their key characteristics. Process Hazard Analysis (PHP) engine and associated logic scripting were used for the dynamic instruction to seamlessly tailor content. The site is rich with photographs of wildflowers and women between the ages of 35 and 55 years of all races and ethnicities. Photographs of foods are used to illustrate categories of foods high in calcium. Headers and sidebars are used to indicate categories of information, and linkages are underlined. All screens contain linkages to return to the main page, return to previous page, or continue to the next page. Pop-up boxes have been included within select screens, enabling women to further select information of potential interest to them. Macromedia Fireworks was used for design features and Adobe Photoshop was used for image reworking (both published by Adobe). Content experts built the initial decision logic and wrote the script in Microsoft Word (Microsoft, Redmond, WA). This Word document was transferred to the Web site. The site is currently being used to deliver an intervention for a research study and is housed on a university system behind a password-protected firewall (Tailored Computerized Intervention for Behavior Change, funded by the National Institute for Nursing Research 1R15NR009021-01A2).
The goal of the handheld computer is to assist people in developing self-regulation skills.29-31 Self-regulation development is an iterative process including goal setting, self-monitoring, understanding and using feedback, self-reflection, and problem solving. A Dell Axim X50 (http://accessories.US.dell.com) pocket computer is programmed with activities designed to facilitate engagement in these self-regulation behaviors. The handheld computer is approximately 5 in long, 3 in wide, and 2/3 in deep and weighs less than 1 lb. Unlike men, who carry handheld computers in their pockets, women prefer to carry these little computers in their purses, bags, and briefcases. The screen has a protective plastic sheet, and the device comes with a soft protective case. The original version of the handheld computer had a Microsoft 3Ce operating system that has been updated to Windows Mobile (both programs by Microsoft). Participants can interact with the computer in four ways: stylus, alphabet board, directly writing on the screen, or an external portable Bluetooth, fold-up keyboard. The keyboard is the same size as the handheld computer and weighs less than a pound. The computer can be used with or without the keyboard.
The handheld computer is programmed so the study icon (a wildflower) appears on the screen (for examples of screens, see Figure 3). Clicking on the icon brings up the program and the links to the major sections of the intervention: specifically, "Goals" (goal setting), "Intake" (self-monitoring and feedback), "Diary" (reflective journaling), and "Helpful Facts and Helps & Tips" (resource information). Data are encrypted within the handheld computer and are automatically downloaded at the end of the intervention into an Access database for analysis.
Each screen is designed with links to the main page and the previous screen. Because the screen of the handheld computer is smaller than a desktop computer screen, the layout of functional buttons differs from the Web site. For example, the top and bottom parts of the handheld computer are used for rapid navigation rather than the top and left-side locations in the Web site. Key information is categorized and located on the bottom of the page, so participants can, via a link, move to the section of interest. Screens have the scroll bar or "elevator" feature, enabling rapid movement through pages extending beyond the visible screens. The program is designed to facilitate interaction and provide feedback. Participants enter personal information into the computer by "filling in the box" and selecting their choice from preconstructed lists. Feedback is provided automatically both in narrative and graphs.
Usability, one component of human engineering,26,32-35 is an iterative process in which feedback is systematically obtained during stages of development with the goal of improving the speed, ease, accuracy, and satisfaction that people experience as they interact with a Web site or computer program.4,36-40 Usability studies have been conducted to assist with the development of a variety of technologies including use of touch screens,38 bar-code scanning for use by patients,41 medication order entry,42 use of Web-based portals,43 defibrillator use among student nurses,44 development of handheld computers,45,46 and for tailored interventions.37,47 Usability testing can be conducted at each stage of development, including creative conceptualization of a new idea, design, development, and implementation of Web sites or computer programs.12,40,48,49 During the initial phase of site development, it is critical to understand the needs, desires, and characteristics of intended users to ensure that the site contains appropriate information and presentation. Site development focuses on meeting goals in a way that is technically feasible and cost-effective, balancing the goals with available resources. Site and program use can be evaluated first within a laboratory setting and then in a real-world environment. Hence, different types of usability testing (eg, expert review [heuristics], cognitive walk-through, testing via software, or performance usability)26 can be conducted during different stages of intervention development. Each method of usability testing is designed to focus on different aspects of the Web site or computer program. For this program, heuristics was used to evaluate the congruence of this site with its underlying theory,19 and systematic testing was conducted in the laboratory to evaluate overall navigation, time to load, page layout, and seamless tailoring and to ensure accurate functioning of all links. At each step, adjustments were made to the Web site and program.
Performance usability is conducted in a natural setting, mirroring the intended use of the program. It is a dynamic assessment of the interaction between a user and the device used to deliver the intervention.48,50 The goal of performance usability is to validate the ability of people to use a technical device. Performance usability focuses on the person rather than the device per se. Performance usability is an activity that is goal directed. Farzanfar and colleagues39 found that patients and professionals differ in the criteria they use to evaluate the usability of an e-delivery method, making it important to match the usability testing to persons and goals.
The goals for use of the device are specified, and users are evaluated using the device to determine whether they are able to achieve these goals. Performance usability determines parameters such as how long it takes people to complete a task, how linkages are used,51 errors, frustration, and accuracy. Performance usability tests (studies) help to identify problems or issues that arise during real-world use. Performance usability has been tested using self-report surveys38,43,45,46,52 or both observations and self-report.41,45 The results of these studies indicate that it is the combination of self-report and observation that appears to provide the most accurate evaluation of usability. Information learned from this testing is used to modify the site or program prior to efficacy testing or actual use. It has been estimated that usability testing can detect up to 85% of the errors that people experience using a new device.33 Evaluation of performance usability is critical as it is 10 times less costly to correct errors during the development phase than during actual use.26,32,33
The purpose of this study was to determine the performance usability of the Web site and handheld computer in an environment in which the effectiveness of the intervention is being formally tested. Specifically, could women use the newly developed Web site and program in a handheld computer accurately and in a timely manner in real-world settings?
This study combined quantitative and qualitative performance usability methods.26,32,33,53 Complete a Task, a quantitative method,53 was used to evaluate women's ability to actually use the Web site and computer program. The goals of the Complete a Task study were to identify the percentage of participants who could (a) complete each of the 15 tasks for the Web site and the handheld computer program within a predetermined time limit for each task, (b) accurately complete the tasks, and (c) complete the task in less than 2 minutes.
The second performance usability method used was the Talk Aloud Method (also known as "Think Aloud Method").54,55 This method is a qualitative approach that provides subjective data of participants' experiences using the Web site and computer program. Understanding participants' perceptions of their use of the Web site and handheld computer program can pinpoint areas of confusion as well as provide overarching responses to the e-delivery method, content, layout, terminology, and meaningfulness of the intervention.
A convenience sample of 15 women between 35 and 55 years of age, an age group in which little usability testing has been conducted,36 were recruited from ambulatory clinics, a health education center, and a health fair. Women were eligible to participate if they perceived themselves to be healthy. Women were required to speak, read, and write English. No prior computer knowledge or skill was required. Usability studies require small sample sizes, generally five to eight persons.26,34,56 Twice the required number of women was recruited for this usability study to evaluate differences across experience with computer use and use in different real-world environments.26
The Complete a Task method of performance usability requires each participant to complete a series of goal-oriented tasks. Complete a Task requires women to understand the setup, components, logic, links, and organization of the Web site or program on the handheld computer. As the participants complete the task, their performance is evaluated for accuracy and timeliness as a measure of the utility of the e-delivery method. The Web site and handheld device were developed to achieve a number of goals; specifically, this intervention was developed to increase women's knowledge of bone health, enhance their health beliefs, and promote use of skills and activities to increase self-regulation and engagement in social facilitation activities. The specific tasks included in this usability testing evaluated participant's abilities to use the newly developed electronic intervention to achieve these goals.
One set of tasks was appropriate for the Web site and one for the handheld computer. To complete these tasks, women were required to navigate the Web site and the program on the handheld computer. Individual tasks included things such as completing assessment information, finding the list of common foods high in calcium, finding information in all four different dimensions (required navigation in and out of each of the dimension), and using the index to locate information related to a topic. Personal information related to age, education, and usual computer use was collected.
Permission to conduct this study was obtained from the required institutional review boards. Women were recruited via flyers, posters, or personal contact. For access to the Web site, women used desktop computers within one of two patient education areas in an outpatient clinic, a portable computer at a health fair, or on their personal home computers. A "virtual," fully functional copy of the Web site was loaded onto a laptop for use at the health fair. All participants received two printed forms, each containing a list of 15 tasks.
The procedure was explained to the participant, and the observer opened the site/program to the splash page to begin the evaluation session. The splash page of the Web site was available on the screen at the time the study began. Women navigated the Web site with a mouse and the handheld computer with a stylus touchscreen. Women were allotted 2 minutes to find content on the Web site or in the program on the handheld computer and answer each question or complete each task. The researcher stressed that the purpose of this activity was to evaluate the Web site and the handheld computer and that this activity was not an evaluation of their knowledge or computer proficiency.
The researcher was positioned to observe both the screen and the participant. The observer used a digital timer affixed to a clipboard and evaluation sheet. Participants were requested to verbally report when they had completed the task and to proceed to the next task from the location in the program where they had completed the last task (some usability testing returns participants to the splash page following each test item-as we were evaluating use in the real world, participants needed to find their way through the site without returning to the splash page). Women were asked to "talk aloud" as they answered questions and performed tasks for the researcher to more clearly identify and understand problems and issues and their perspectives.
Following the hands-on evaluation of the handheld computer, participants completed the personal information questionnaire. All participants were provided written information on osteoporosis prevention prepared by the Arthritis Foundation and received several different types of calcium samples and coupons for calcium supplements.
Quantitative data were obtained via Complete a Task and consisted of task completion, accuracy, and time required completing the task. Completion, accuracy, and completion of a task in less than 2 minutes were evaluated as yes/no. Time was recorded in 30-second intervals, so time to complete a task could be approximated if it took longer or shorter than the estimated 2 minutes. When the electronic intervention functioned as designed, accuracy and time of task completion were determined. However, if a component in the intervention did not function as designed, the participant could not complete the task, and neither time nor accuracy could be determined.
Qualitative data were obtained in two ways. Observation of participants during Complete a Task provided data that could be used to understand problems that users experienced in completing the tasks. Talk Aloud is a qualitative method designed to provide explanations of cognitive thinking and decision making. These qualitative data were analyzed to identify patterns of problems related to function and language.
The mean age of participants was 48.3 years; 53.4% graduated from college, while 20% had high school diplomas. Most women were white (73%), 20% were African Americans, and 6% were Hispanic. The majority (80%) reported using a computer daily for e-mail and Internet shopping (93.4%), word processing (86.7%), calendar and work-related files (40%), and occasionally for managing finances (13%). Sixty percent were very comfortable using computers, and 20% were only somewhat comfortable or uncomfortable.
Of the 15 tasks used to evaluate the Web site, 12 (80%) were completed; three tasks could not be completed. Of the 12 tasks that could be completed, all were completed accurately in less than 2 minutes. All of the 12 tasks were completed accurately in 1 minute or less, and nine of these 12 tasks were completed accurately in 30 seconds or less.
Of the 15 tasks used to evaluate the handheld computer, 13 (86%) were completed; two tasks could not be completed. Of the 13 tasks that could be completed, all were completed accurately in less than 2 minutes. All 13 tasks were completed accurately in 1 minute or less; no one completed them in less than 30 seconds.
Results included identification of functional and language problems (Table 3). Functional problems included identification of a broken link, inconsistent appearance of pop-up boxes (sometimes in front, but at other times behind the screen), and failure to program the Web site to "remember" the participant's responses to the assessment questions for the duration of the session, limiting their ability to move between tailored and general content.
Several factors affected women's ability to use the handheld computer. Because of the size of the handheld computer, general information is provided at the top, similar in many respects to a Web site, and at the bottom, different from most Web sites. Change in screen layout distracted from the use of the handheld computer in several ways. Participants were unclear how to exit the "Goal-setting" section because the exit button was at the bottom. When participants completed their assessment of calcium intake, the "Total" button needed to be tapped or the calcium intake remained incorrect. No participants used the elevator button.
Informational problems included a determination that the questions used to assess menopausal status were too complex to be easily understood. A sidebar term that was not intuitively understood was used, and participants requested additional linkages and increasingly detailed information about foods. Women did not know the amount of calcium in the supplement they were taking, leading to inaccurate or incomplete completion of the assessment questions.
An interesting observation was how participants processed information. A pattern of processing information common to most participants was noted. Participants began by diligently following the tasks to be completed. Once they were aware that they could complete the tasks without problem and in much less time than allotted, they began to read through the content on the Web site for their own interest. Before they could find anything informative or new to them in the Web site, they appeared to need to scan the information to ascertain whether they agreed with the content. They would read through information and say things such as "Yes, that's true," "Where do you talk about 'X,' oh here's that information," and "Yeah, that worked for me." Only after the information passed their scrutiny were participants ready to take in new information exemplified by comments such as "Oh, I didn't know that," "Isn't that interesting," or "Oh that's a great idea."
SUMMARY AND DISCUSSION
Task completion rates of 80% for the Web site and 86% for the handheld computer identified specific opportunities for improvement of the intervention. Conducting performance usability of these newly developed delivery methods resulted in identification of several problem affecting the ability of participants to use the Web site and handheld computer. Once identified, the problems were easily resolved, with the exception of women's knowledge of the amount of calcium in their supplements. Conducting usability studies in an environment similar to the environment in which they were used identifies issues and problems not apparent in laboratory or office settings. The results of this usability testing enabled the intervention to be improved by eliminating technical and language problems that prevented women from receiving the intervention as intended.
Tailored content was integral to this intervention on the Web site. To provide tailored content, participants need to be able to correctly complete preassessment data. Self-monitoring, a dimension of the handheld computer, requires women to know the amount of calcium in their supplements. Women's lack of knowledge related to calcium content in supplements was a major finding requiring redesign of the assessment components of the Web site and the program on the handheld computer. Without usability testing, this problem would not have become apparent, and outcomes of the next phases of research testing could have been seriously affected.
None of the participants in this study had ever seen a handheld computer, yet with very minimal instruction, they were able to navigate through this program. Women responded very favorably to this component of the intervention, and all had the dexterity to use it. As a result of usability testing, we were able to pinpoint with very few participants the major problems that users would have experienced during actual use. This study provided a great example of the importance of including performance usability as one component in the development of an intervention delivered via technology.
1. Putnam J. Bowling alone: Americas declining social capital. J Democr. 1995;6(1):65.
2. Kawachi I, Kennedy BP. The Health of Nations: Why Inequality Is Harmful to Your Health. New York, NY: New York Press; 2002.
3. Bull FC, Holt CL, Kreuter MW, Clark EM, Scharff D. Understanding the effects of printed health education materials: which features lead to which outcomes? J Health Commun. 2001;6(3):265-279.
4. McRoy SW, Liu-Perez A, Ali SS. Interactive computerized health care education. J Am Med Inform Assoc. 1998;5(4):347-354.
5. Rector C. An integrative review of computer-based simulation in the education process. Comput Inform Nurs. 2002;20(5):203-211.
6. Revere D, Dunbar PJ. Review of computer-generated outpatient health behavior interventions: clinical encounters "in absentia". J Am Med Inform Assoc. 2001;8(1):62-79.
7. Carroll JM, Rosson MB. Paradox of the active user. In: Carroll JM, ed. Interfacing Thought: Cognitive Aspects of Human-Computer Interaction. Cambridge, MA: MIT Press; 1987:80-111.
9. U.S. Department of Health & Human Services (HSDHHS). Usability Gov: A Guide for Developing & Useful Web Sites. Designing educational booklets for the Web. In: Services DoHaH, ed. Usability: FirstGov.gov
. Accessed July 13, 2009.
10. Ammenwerth E, Iller C, Mahler C. It-adoption and the interaction of task, technology and individuals: a fit framework and a case study. BMC Med Inform Decis Mak. 2006;6:3.
12. Hesse BW, Shneiderman B. eHealth research from the user's perspective. Am J Prev Med. 2007;32(58):s97-s103.
13. Johnson CW. Why did that happen? Exploring the proliferation of barely usable software in healthcare systems. Qual Saf Health Care. 2007;15(S1):i76-i81.
14. Shekelle PG, Morton SC, Keeler EB. Costs and Benefits of Health Information Technology: Evidence Report/Technology Assessment No. 132. Rockville, MD: Agency for Healthcare Research and Quality; 2006. Publication no. 06-E006.
15. Kools M. A focus on the usability of health education materials. Patient Educ Couns. 2007;65:275-276.
16. Whittemore R, Grey M. The systematic development of nursing interventions. J Nurs Scholarsh. 2002;34(2):115-120.
17. Campbell M, Firzpatrick R, Haines A, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321(16):694-696.
18. Karsh B-T. Beyond usability: designing effective technology implementation systems to promote patient safety. Quality Saf Health Care. 2004;13:388-394.
19. Ryan P. Integrated theory of health behavior change: background and intervention development. Internet J Adv Nurs Pract. 2008;23(3):161-170.
20. Staggers N, Thomas CR, Happ B. An operational model for patient-centered informatics. Comput Nurs. 1999;17(6):278-285.
21. Baker L, Bundorf K, Royalty A, Galvin C, McDonald K. Consumer-Oriented Strategies for Improving Health Benefit Design: An Overview. Rockville, MD: Agency for Healthcare Research and Quality; 2007. Publication no. 07-0067.
22. Dollarhide AW, Rutledge T, Weinger MD, Dresselhaus TR. Use of a handheld computer application for voluntary medication event reporting by inpatient nurses and physicians. J Gen Intern Med. 2007;23(4):418-422.
23. Farrell MJ, Rose L. Use of mobile handheld computers in clinical nursing education. J Nurs Educ. 2008;47(1):13-19.
24. Kuiper R. Use of personal digital assistants to support clinical reasoning in undergraduate baccalaureate nursing students. Comput Inform Nurs. 2008;2:90-98.
25. McGowan JJ. The pervasiveness of telemedicine: adoption with or without a research base. J Gen Intern Med. 2008;23(4):505-507.
26. Koyanl SJ, Bailey RW, Nall JR, et al. Research-Based Web Design & Usability Guidelines. Bethesda, MD: AHRQ. NIH publication 03-5424; 2003.
27. Ryan P, Lauver DR. The efficacy of tailored interventions. J Nurs Scholarsh. 2002;34(4):331-337.
28. Osteoporosis Prevention, Diagnosis, and Therapy. NIH Consensus Statement. Bethesda, MD: NIH. 2000;17(1):1-45.
29. King AC, Ahn DK, Oliveria BM, Atienza AA, Castro CM, Gardner CD. Promoting physical activity through hand-held computer technology. Am J Prev Med. 2008;34(2):138-142.
30. Wilkinson SA, Miller YD. Improving health behaviours during pregnancy: a new direction for the pregnancy handheld record. Aust N Z J Obstet Gynaecol. 2007;47:464-467.
31. Wing RR, Tate DF, Gorin AA, Raynor HA, Fava JL. A self-regulation program for maintenance of weight loss. N Engl J Med. 2006;355(15):1563-1571.
35. Staggers N. Human-computer interaction in health care organizations. In: Englebardt SP, Nelson R, eds. Health Care Informatics: An Interdisciplinary Approach. St Louis, MO: Mosby; 2002:321-345.
36. Armbruster C, Sutter C, Ziefle M. Notebook input devices put to the age test: the usability of trackpoint and touchpad for middle-aged adults. Ergonomics. 2007;50(3):426-445.
37. Eakin B, Brady JS, Lusk SL. Creating a tailored, multimedia, computer-based intervention. Comput Inform Nurs. 2001;119(4):152-160.
38. Edwards SL, Slattery ML, Murtaugh MA, et al. Development and use of touch-screen audio computer-assisted self-interviewing in a study of American Indians. Am J Epidemiol. 2007;165(11):1336-1342.
39. Farzanfar R, Finkelstein J, Friedman RH. Testing the usability of two automated home-based patient-management systems. J Med Syst. 2004;28(2):143-153.
40. Pearce GF, Williamson J, Harrell JS, Wildemuth BM, Solomon P. The children's computerized physical activity reporter: children as partners in the design and usability evaluation of an application for self-reporting physical activity. Comput Inform Nurs. 2007;25(2):93-105.
41. Boissy P, Jacobs K, Roy SH. Usability of a barcode scanning system as a means of data entry on a PDA for self-report health outcome questionnaires: a pilot study in individuals over 60 years of age. BMC Med Inform Decis Mak. 2006;6:6-42.
42. Eslami S, Abu-Hanna A, de Keizer NF. Evaluation of outpatient computerized physician medication order entry system: a systematic review. J Am Med Inform Assoc. 2007;14:400-406.
43. Koivunen M, Valimaki M, Pitkanen A. A preliminary usability evaluation of Web-based portal application for patients with schizophrenia. J Psychiatr Ment Health Nurs. 2007;14:462-469.
44. Monsieurs KG, Vogels C, Bossaert LL, Meert P, Calle PA. A study comparing the usability of fully automatic versus semi-automatic defibrillation by untrained nursing students. Resuscitation. 2005;64:41-47.
45. Treadwell I. The usability of personal digital assistants (PDAs) for assessment of practical performance. Med Educ. 2006;40:855-861.
46. Chen Y-C, Chiu H-C, Tsai M-D, Chang H, Chong C-F. Development of a personal digital assistant-based wireless application in clinical practice. Comput Methods Programs Biomed. 2007;85:181-184.
47. Effken JA, Doyle M. Interface design and cognitive style in learning an instructional computer simulation. Comput Inform Nurs. 2001;19(4):164-171.
48. Beuscart-Zephir MC, Brender J, Beuscart R, Menager-Depriester I. Cognitive evaluation: how to assess the usability of information technology in healthcare. Comput Methods Programs Biomed. 1997;54:19-28.
49. Stinson JN, Petroz GC, Tait G, et al. e-Ouch: usability testing of an electronic chronic pain diary for adolescents with arthritis. Clin J Pain. 2006;22(3):295-305.
50. Rees M, Dineschandra J. Monitoring clinical performance: the role of software architecture. Health Care Manag Sci. 2005;8:197-203.
51. Ling J, vanSchaik P. The effects of link format and screen location on visual search of Web pages. Ergonomics. 2004;22:907-921.
52. Beuscart-Zephir MC, Watbled L, Carpentier AM, Degroisse M, Alao O. A rapid usability assessment methodology to support the choice of clinical information systems: a case study. In: American Medical Informatics Association 2002 Annual Symposium Proceedings
. 2002:46-50. http://www.pubmedcentral.nih.gov
. Accessed July 13, 2009.
53. Rubin J. The Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. New York, NY: John Wiley & Sons; 1994.
54. Fonda SJ, Paulsen CA, Perkins J, Kedziora RJ, Rodbard D, Bursell S-E. Usability test of an Internet-based informatics tool for diabetes care providers: the comprehensive diabetes management program. Diabetes Technol Ther. 2007;10(1):16-24.
55. Neufeld A, Harrison MJ, Rempel GR, et al. Practical issues in using a card sort in a study of nonsupport and family caregiving. Qual Health Res. 2004;14(10):1418-1428.
For more than 23 additional continuing education articles related to education, go to NursingCenter.com\CE.
© 2009 Lippincott Williams & Wilkins, Inc.