Skip Navigation LinksHome > February 2003 - Volume 33 - Issue 2 > A Model for Examining Predictors of Outcomes of Telephone Nu...
Journal of Nursing Administration:

A Model for Examining Predictors of Outcomes of Telephone Nursing Advice

Valanis, Barbara DrPH, RN; Tanner, Christine PhD, RN; Moscato, Susan Randles EdD, RN; Shapiro, Susan MS, RN; Izumi, Shigeko MSN, RN; David, Marion PhD; Keyes, Connie MSN, RN; Mayo, Ann DNSc, RN

Free Access
Article Outline
Collapse Box

Author Information

Authors’ affiliations: Kaiser Permanente Center for Health Research, Portland, Ore (Drs Valanis and David); Oregon Health & Science University, School of Nursing, Portland, Ore (Dr Tanner, Ms Shapiro, Ms Izumi); University of Portland, School of Nursing, Ore (Dr Moscato); Kaiser Permanente Northwest, Health Systems Department, Portland, Ore (Ms Keyes); Kaiser Permanente Southern California, Patient Care Services, San Diego (Dr Mayo).

Corresponding author: Barbara Valanis, DrPH, RN, Center for Health Research, 3800 North Interstate Ave, Portland, OR 97227 (

Collapse Box


Although telephone advice nursing is the fastest-growing nursing specialty, useful information to guide managers’ decisions about how best to structure and support advice services to achieve desired outcomes is unavailable. We identified issues and variables relevant to outcomes of telephone advice from the perspectives of callers, nurses, and the system. Subsequently, we derived a model for studying factors affecting nursing advice outcomes that will help managers identify modifiable factors to improve outcomes of care.

Health plans struggle to improve access to appropriate healthcare services and maintain or improve member satisfaction while controlling costs. Telephone nursing advice services are one approach used to meet these goals, making telephone nursing care the most rapidly growing nursing specialty. 1–6 These services are intended to provide prompt and efficient responses to health plan members’ questions, manage demand by directing members to the appropriate level of care, enhance members’ self-care ability, and improve members’ satisfaction with the health plan. 7

Telephone advice nursing services in all regions of Kaiser Permanente (KP), a large, nonprofit, group-model HMO, have experienced rapid increases in numbers of calls in recent years. The structure of telephone advice services and support for nurses who deliver services vary widely, raising questions as to the best way to structure and support this service. Little is known about how the advice service fits into the care process or about outcomes of advice, so managers have begun to question whether the outcomes are worth the increasing investment.

The literature has not addressed the crucial issues that would provide the information needed for administrative decision-making, such as relationships of telephone advice outcomes with care processes, structural characteristics of call centers, characteristics of callers, and nursing practice. Rather, it contains descriptions of how telephone advice is used, the volume and dispositions of calls, approaches to documentation, and discussions of the pros and cons of protocol use and legal liability and licensure issues. The few empirical studies that have been published suggest that patients are generally satisfied with telephone advice 8–11 and that telephone services may reduce unnecessary emergency department and office visits. 12,13 Research is needed to identify the factors that affect relevant outcomes of the calls.

In response to this need, we are conducting a four-phase study that examines the relationships of caller, nurse, and system characteristics to telephone nursing advice outcomes. The phases are as follows:

1. Preliminary studies and development of the conceptual approach;

2. Development of tools to measure key variables;

3. A pilot/feasibility phase to finalize instrument development and test the logistics of data collection; and

4. The full-scale study and data analysis, now in progress.

In this article, we describe the conceptual and research processes we used to arrive at our working conceptual model for studying telephone nursing advice outcomes, and present the model. We also present some findings from developmental and pilot studies.

Back to Top | Article Outline

Identifying Key Variables

To identify relevant variables, we reviewed the literature on interactive communication, factors affecting patient satisfaction and compliance, and costs of care. We also identified three perspectives to be considered: that of the callers, that of the nurses delivering advice, and that of the healthcare system.

To validate our basic approach, we scanned the KP environment by talking with managers of telephone nursing advice in several KP regions, reviewing written reports, and examining issues on the agenda of the KP Interregional Nursing Council to identify what was already known about telephone advice nursing from the three perspectives.

Back to Top | Article Outline
The Caller Perspective

The Advice Nurse Quality Improvement Project, conducted in the Northwest region of Kaiser Permanente (KPNW) in 1993, 14 identified all calls to advice nurses occurring during one day, then surveyed 407 of the callers. Caller satisfaction was high: 55% described themselves as very satisfied, and 34% as somewhat satisfied. Key reasons for satisfaction included good advice, friendly/sympathetic nurses, and having an appointment quickly scheduled. Dissatisfied members mentioned service components (eg, being on hold too long, being kept from seeing a physician, or no call back) and receiving advice that was not understandable or appropriate. These results highlighted the importance of assessing the interpersonal and service aspects of the call interaction, as well as the quality of the advice given.

Back to Top | Article Outline
The Nurses’ Perspective

A doctoral dissertation 15 and two other studies conducted in KP and non-KP call centers in Southern California by Dr. Mayo 16,17 provided insight into the perspective of advice nurses and identified important dimensions for the research to address. The three studies used qualitative interviews and analyses of taped advice calls. The dissertation examined workflow and advice nurses’ experiences, 16 the second study examined the interpersonal and service aspects of the call interaction and advice quality, 15 and the third rated call interactions on assessment, planning, implementation, and evaluation aspects of the nursing process. 17

Content analysis from study one identified themes, including the nurses’ ability to “see the big picture,” “deciding to not use protocols,” “avoiding making medical diagnoses,” and “experiencing fulfillment when they connected with patients.” The nurses perceived that telephone advice/triage nursing involves “fitting pieces into puzzles.” As one nurse said, “It’s not only what I hear, it’s how I hear it, and what’s not said. It’s using a lot of different resources and information.”15

The second study classified the nurses into three groups: those over-confident in their decision-making skill who believed unequivocally in the “rightness” of their protocols; those comfortable with their decision-making skills and aware of what they do and do not know; and those always uncomfortable with decision-making, who seek consultation to validate most of their decisions and advice. 16 These findings suggested a need to examine how advice nurses’ backgrounds and experience relate to their perceptions of supports and barriers to their practice and their actual performance.

The third study unexpectedly found a negative relationship (r = −0.395, P < .000) between availability of protocols and the overall quality of the nursing process. 17 Use of protocols will be a key variable in our model.

Back to Top | Article Outline
The System Perspective

Managers of nursing advice services noted that KP telephone advice services differ by site in their use of protocols, prescreening of calls, and availability of patient information and medical or pharmaceutical consultation at the time of the call. Managers also confirmed that member complaints about nursing advice continue to focus on service-related factors.

Quality assurance data from the KP call centers focused on hold times, hang-up rates, and length of the calls, which contribute little understanding of how call center organization and nursing processes contribute to call outcomes. Little was known about which members used advice, how often they called, how often they followed the advice given, and whether the advice was used in lieu of other services—questions of great interest to managers.

Based on this information and our review of the literature, we identified five potential outcomes of interest: patient satisfaction, compliance with the advice given, disposition of the call, appropriateness of the call disposition, and shifts in use of services resulting from the nursing advice service.

Back to Top | Article Outline

Medical Record Review on Use of Advice Services

To determine who among the HMO membership generally used telephone nursing advice services, we examined the electronic medical records of a random sample of 200 KPNW members for evidence of telephone advice use. Fewer than half (47%) had used telephone advice nursing services in a 1-year period beginning in November 1997. Most called once or twice, 2% called four or more times, and 5% called at least 12 times. Patient ages distributed across the age spectrum, but calls representing children and the elderly were more frequent. Only 23% of men had called (or had someone call), versus 59% of women.

The number of health problems noted on the medical records’ problem list correlated positively with the number of advice calls (r = 0.43;P = 0.000). Consistent with another study, 18 we found a positive correlation between number of advice calls and number of office visits (r = 0.46;P = 0.001). Over half of the calls (51.5%) were about new symptoms or a change in an existing health condition, 12.6% were questions related to medications or postvisit follow-up, and 22.9% were for prescription refills or test results. These data provided a preliminary scheme for classifying the reason for the call and indicated a need to further examine which subset of the membership uses advice services. It also seemed important to understand why the advice nurse service was chosen for obtaining test results or prescription renewals, and to investigate the entire episode of care for calls triggered by questions following medical office visits.

Back to Top | Article Outline

Review of Taped Calls

Of 27 advice nurses working in a centralized call center, 25 returned our peer rating form, but only 7 agreed to have their calls recorded. From these 7, whose experience in advice nursing ranged from newly employed in advice nursing to 20 years of experience, we recorded 270 calls. Members of the investigative team and several experienced advice nurses listened to the same subset of taped conversations and independently made lists of key nursing practice behaviors they thought distinguished more experienced from less experienced nurses. The list of behaviors agreed upon by the team was used to develop a preliminary instrument for coding the nurse behaviors from the tapes. The categories of behaviors that appeared to distinguish more experienced from less experienced nurses were interpersonal communication style, quality of assessment, and quality of nursing judgment. We believed that these behaviors could contribute to call outcomes and should be included as mediating variables in the conceptual model. We also developed measures to assess these behaviors.

Back to Top | Article Outline

Interviews With Callers

One investigator on our team conducted debriefing interviews with a convenience sample of 40 of the 270 callers whose calls were taped. Using a semi-structured interview format, the interviewer asked callers to tell their stories about why they called, what they would have done without the advice service, what advice they were given and its clarity and relevance, their satisfaction with the advice, and their follow-through. Using an emergent design, responses from previous interviews were used to ask additional questions in later interviews. We reviewed the tapes and notes from the interviews to identify themes related to the advice experience. This review process identified important caller factors that might affect outcomes, including callers’ perceptions of affective support, decisional control, the advice nurses’ familiarity with their doctor’s practice and likely recommendations, and the nurses’ competence. It also suggested call outcomes important from the caller’s perspective: the callers’ confidence in the call disposition, their capacity for self-care, and their follow-up on advice given. In addition to suggesting important variables for the conceptual model, interview responses provided important content and wording for development of a draft caller survey, a call description form, and a tool to assess nurse interpersonal communication skills.

Back to Top | Article Outline

Focus Groups With Telephone Advice Nurses and Site Observations

To identify additional aspects of the structure and processes of telephone nursing advice services that might affect outcomes, we conducted site observations at centralized call centers and medical office advice sites in four KP regions. We also conducted focus groups with call center nurses, medical office nurses, and some physicians in these four regions. Comments of the nurses during focus groups and our pilot site observations indicated that autonomy and control issues, limitations on the nurses’ involvement in decision-making, and the stress in the work environment interfered with the nurses’ ability to meet the needs of the caller. The nurses’ relationships with physician colleagues were also identified as supporting or hindering their advice practice. These work environment factors were also incorporated into our model.

Back to Top | Article Outline

A Model for Organizing Variables

Through our literature review and preliminary work, we identified a large number of potential variables that might affect the practice of telephone advice nurses and outcomes of telephone nursing advice. We used a systems orientation, wherein existing structural factors and processes operating in the health system are expected to influence outcomes, to organize the variables under consideration into structure, process, and outcome variables.

Traditionally, social models have focused on the individual and have not sufficiently considered the social and institutional contexts in which individuals function. Our working model (Figure 1) places the practice of telephone advice nurses within the context of the institutional environment in which they function. The left side of the model shows structural characteristics of the system and inherent characteristics of the nurses and callers that are present prior to the call. These existing factors are expected to influence processes operating at the time of the call, shown in the center area of the model. The major study outcomes, shown on the right side of the model, are grouped as (1) nurse/call outcomes; (2) caller outcomes; and (3) system outcomes. We expect that these outcomes would be influenced primarily by process variables operating at the time of the call, in the absence of an expert advice nurse.

Figure 1
Figure 1
Image Tools

We hypothesize that an expert advice nurse will mediate the negative effects on the call outcome of both pre-existing system characteristics and system barriers that operate at the time of the call (eg, appointment unavailability and prescriptive protocols). The expert advice nurse will work around these barriers, guiding the patient to what the nurse views as the most appropriate level of care given medical need, patient preference, and patient barriers to compliance with particular care options. We expect that assessment and judgments that are sensitive to caller needs and preferences will result in higher caller satisfaction, higher perceived support, and increased likelihood of follow-through on the advice given.

We derived our model (Figure 1) through an empirical process and used it to select variables for inclusion in the full-scale study and guide development of our study design. The model is conceptually consistent with a theoretical model recently postulated by Larson-Dahn 19 for integrating outcomes into telephone nursing practice. In our earlier models, we had included some additional factors that are in her model, but we later deleted them, either because they did not emerge during our preliminary work as important barriers or facilitators of advice practice outcomes or because they could not be measured reliably in order to study them. In our model, we focus on factors that are modifiable, either by changing structure and processes or through providing training and support to improve the practice of advice nurses. This approach is most likely to produce results that can inform the decisions of managers.

Data collection for the full-scale study was recently completed. Analyses are in progress. We will use the data from the full-scale study to test and refine the model. Subsequently, we hope to test it in other healthcare systems to confirm that the same key factors predict outcomes in settings with different structures and processes for delivering nursing advice. Knowing what factors affect outcomes of nursing advice will help nursing managers decide how to structure and support the delivery of telephone nursing advice to achieve optimal outcomes.

Back to Top | Article Outline


The work reported in this article was supported by grants from the following sources: Oregon Health & Science University, School of Nursing, Faculty Intramural Research Award; University of Portland, School of Nursing, Butine Award; Kaiser Permanente Garfield Memorial Fund.

Back to Top | Article Outline


1. American Academy of Ambulatory Care Nursing (AAACN). Telephone Nursing Practice Administration and Practice Standards. Pitman, NJ: Anthony J. Jannetti; 1997.

2. Blanchfield KC. Telephone nursing practice SIG reports on study of telephone interventions in ambulatory care. AACN Viewpoint. 1996; 18( 5):265–267.

3. Glasper EA. Telephone triage: a step forward for nursing practice? Br J Nurs. 1993; 2( 2):108–109.

4. Kastens JM. Integrated care management: aligning medical call centers and nurse triage services. Nurs Econ. 1998; 16( 6): 320–322.

5. Lazarus I. Medical call centers: an effective demand management strategy. Managed Healthcare. 1997; 56:58–59.

6. Lippman H. Career options: RNs take over for Denver pediatricians. RN 1995; 58( 3):52–54.

7. Kaiser Permanente. Report of the Call Center Survey: Overview and Results. Oakland, Calif: Presented at Kaiser Permanente National Call Center Conference, 1999.

8. Dale J, Crouch R, Patel A, Williams S. Patients telephoning A & E for advice: a comparison of expectation and outcomes. J Accident Emerg Med. 1997; 14:21–23.

9. Egleston CV. Use of a telephone advice line in an A & E department. Br Med J. 1994; 308:31.

10. Poole SR, Schmitt BD, Carruth T, Peterson-Smith A, Slusarski M. After-hours telephone coverage: the application of an area-wide telephone triage and advice system for pediatric practices. Pediatrics. 1993; 92:670–679.

11. Srinivas S, Poole F, Redpath J, Underhill TJ. Review of a computer-based telephone helpline in an A & E department. J Accident Emerg Med. 1996; 13:330–333.

12. Greenberg ME. Evaluation of an Established Telephone Triage Program [doctoral dissertation]. Tucson, Ariz: University of Arizona; 1999.

13. O’Connell JM, Johnson DA, Stallmeyer J, Cokingtin D. A satisfaction and return-on-investment study of a nurse triage service. Am J Managed Care. 2001; 7( 2):159–169.

14. Keyes C, Shapton L, eds. Advice Nurse Caller Survey Report. Portland, Ore: Kaiser Permanente Northwest Region; 1993.

15. Mayo A. The Experience of Decision-making Among Telephone Advice Triage Nurses [doctoral dissertation]. San Diego, Calif: University of San Diego; 1998.

16. Mayo A. The role of the telephone advice/triage nurse. AAACN Viewpoint. 1998; 20( 2):9–10.

17. Mayo A, Chang B, Omery A. Use of protocols and guidelines by telephone advice nurses. J Clin Nurs Res. 2002; 11( 2);204–219.

18. Daugird AJ, Spencer DC. Characteristics of patients who highly utilize telephone medical care in a private practice. J Family Pract. 1989; 29( 1):59–64.

19. Larson-Dahn M. Tel-eNurse practice: quality of care and patient outcomes. JONA 2001; 31( 3):15–152.

Cited By:

This article has been cited 9 time(s).

American Journal of Managed Care
Communicating effectively and managing care
Smith, DG
American Journal of Managed Care, 10(8): 529-530.

Journal of Clinical Nursing
Commentary on Pascoe SW and Neal RD (2004) Primary care: questionnaire survey of alternative forms of patient and nurse face-to-face consultations. Journal of Clinical Nursing 13, 406-407
Patterson, E
Journal of Clinical Nursing, 15(3): 374-375.

Journal of Advanced Nursing
A comprehensive model of the process of telephone nursing
Greenberg, ME
Journal of Advanced Nursing, 65(): 2621-2629.
Journal of Clinical Nursing
Commentary on Pascoe SW and Neal RD (2004) Primary care: questionnaire survey of alternative forms of patient and nurse face-to-face consultations. Journal of Clinical Nursing 13, 406-407
Patterson, E
Journal of Clinical Nursing, 14(3): 403-404.

Family Practice
Follow-through after calling a nurse telephone advice line: a population-based study
De Coster, C; Quan, HD; Elford, R; Li, B; Mazzei, L; Zimmer, S
Family Practice, 27(3): 271-278.
Journal of Telemedicine and Telecare
Telephone advice nursing services in a US health maintenance organization
Shapiro, SE; Izumi, S; Tanner, CA; Moscato, SR; Valanis, BG; David, MR; Gullion, CM
Journal of Telemedicine and Telecare, 10(1): 50-54.

Journal of Pediatric Health Care
Evaluation of a telephone advice nurse in a nursing faculty managed pediatric community clinic
Beaulieu, R; Humphreys, J
Journal of Pediatric Health Care, 22(3): 175-181.
Implementation Science
Managing symptoms during cancer treatments: evaluating the implementation of evidence-informed remote support protocols
Stacey, D; Bakker, D; Ballantyne, B; Chapman, K; Cumminger, J; Green, E; Harrison, M; Howell, D; Kuziemsky, C; MacKenzie, T; Sabo, B; Skrutkowski, M; Syme, A; Whynot, A
Implementation Science, 7(): -.
ARTN 110
Journal of Nursing Administration
Organization and Processes of Telephone Nursing Advice Services: Making it Work
Mayo, A; Valanis, B; Moscato, S; Tanner, C; Shapiro, S; Izumi, S; David, M
Journal of Nursing Administration, 33(4): 216-223.

PDF (106)
Back to Top | Article Outline

© 2003 Lippincott Williams & Wilkins, Inc.