Skip Navigation LinksHome > May/June 2009 - Volume 27 - Issue 3 > Factors Influencing Outcomes of Clinical Information Systems...
CIN: Computers, Informatics, Nursing:
doi: 10.1097/NCN.0b013e31819f7c07
Continuing Education

Factors Influencing Outcomes of Clinical Information Systems Implementation: A Systematic Review


Free Access
Continued Education
Article Outline
Collapse Box

Author Information

Author Affiliations: Acute Care Clinical Applications Team, Information Systems, Alberta Health Services (Ms Gruber); Faculty of Nursing, University of Alberta, Alberta, Canada (Dr Cummings and Professor Smith); and A Division of Covenant Health, Grey Nuns Community Hospital, Edmonton, Alberta, Canada (Ms LeBlanc).

Disclaimer: Authors declare no conflict of interest.

Corresponding author: Greta G. Cummings, RN, PhD, Faculty of Nursing, University of Alberta, Clinical Sciences Building, 5-125 CSB, Edmonton, AB, Canada T6G2G3 (

Collapse Box


Healthcare agencies spend significant resources to acquire or develop clinical information systems. However, implementation of clinical information systems often report significant failures. A systematic review of the research literature identified processes and outcomes of clinical information system implementation and factors that influenced success or failure. Of 124 original papers, 18 met the primary inclusion criteria-clinical systems implementation, healthcare facility, and outcome measures. Data extraction elements included study characteristics, outcomes, and implementation risk factors classified according to the Expanded Systems Life Cycle. The quality of each study was also assessed. Forty-nine outcomes of clinical information system implementation were identified. No single implementation strategy proved completely effective. The findings of this synthesis direct the attention of managers and decision makers to the importance of clinical context to successful implementation of clinical information systems. The highest number of factors influencing success or failure was reported during implementation and system "go-live." End-user support or lack thereof was the important factor in both successful and failed implementations, respectively. Following the Expanded Systems Life Cycle management model instead of a traditional project management approach may contribute to greater success over time, by paying particular attention to the underrecognized maintenance phase of implementation.

Clinical information systems (CISs) are "large computerized database management systems used by clinicians to access a range of patient data to plan, implement, and evaluate care."1 In this study, we use the expanded definition of CISs that includes a point-of-care, patient-focused computer system that replaces all or some of the paper medical records.2 Other terms for CIS are patient care information systems, computer-based patient records, medical information systems, patient care management systems, health information systems, nursing intervention systems, and electronic medical records. In the broad context, CIS implementation is viewed as including all phases of an application's full life cycle, from planning through maintenance.

Healthcare is increasingly mediated by technology, and the implementation of CISs represents a new era of technological possibilities. As more sophisticated systems to manage patient information become available, there are rising expectations that these CISs will achieve outcomes for systems, users, managers, and patients. These outcomes, in turn, are expected to improve the efficiency and effectiveness of healthcare services. A successful implementation process is critical to gaining the economic and competitive advantages that innovation offers, but not enough is understood about the implementation process because efforts are often either complete or partial failures.3 Success may depend in part on developing a better understanding of the factors that influence the success or failure of CIS implementation.4

The value in examining CIS implementation is in relation to the implementation process itself and whether the outcomes expected are actually achieved within the context of clinical healthcare environments. Examining the role of nursing in implementing CISs is also valuable, as nurses may be unprepared for the requisite technological changes to their work. This could result in a significant lag between the creation of the new healthcare enterprise and the end user's ability to fulfill its potential.5

This area of study is important for several reasons. Clinical environments are unique and merit their own study, as what has been successful in business arenas has not always worked in healthcare.6 Registered nurses are the predominant healthcare providers in the healthcare sector,2 and nursing is healthcare's largest constituency.7 The implementation of CISs influences the clinical practice environment, nurses, and nursing practice, as well as patient safety and quality of care, management decision making, and funding. It is now possible to capture physiologic observations, communicate with ancillary services such as laboratories, assist bedside clinicians to formulate nursing and medical care plans, and use a sophisticated CIS to capture all of the necessary data required to make sound clinical decisions.2,8

The importance of CISs has increased, and efforts to implement automated systems have intensified, as funding methods for health services become increasingly focused on outcomes. Accreditation, chronic disease management, physician remuneration models, and primary care networks all demonstrate a shift to outcomes-based funding. Although literature describing the implementation of computer systems in healthcare organizations is extensive, in a preliminary scoping of the literature, most studies examining CIS implementation were exploratory-descriptive in design, described by Brink and Woods9 as level 1 research, which is appropriate for a developing field of knowledge. We could not find a synthesis of evidence that describes successful or failed CIS implementation. A review of current literature is important to identify and describe the best available evidence about CIS implementation. The study reported in this article was undertaken to address this gap in knowledge. The strength of our review lies in the rigorous assessment of the literature in a field that is new and lacks common definitions and understanding of the process of CISs implementation and outcomes.

Back to Top | Article Outline


A systematic review of the research literature was undertaken to determine the current evidence about the process and outcomes of the implementation of CISs in healthcare facilities. The research question guiding the review was "What is the present state of knowledge about the process of CIS implementation and outcomes for the system, users, management or patients?" Content analysis of the literature identified outcomes, which were classified by system, user/provider, management, and patient.

Back to Top | Article Outline


A key assumption guiding this study was that the context in which a CIS is implemented can influence success or failure of implementation. Therefore, we determined that a theoretical model that takes account of the role of context in implementation would be used to guide this systematic literature review and selected the Expanded Systems Life Cycle (ESLC) framework by Thompson et al.10 This model illustrates the full cycle of CIS implementation within complex clinical contexts. The ESLC model (Figure 1) proposes a logical process for planning, executing, and managing system life cycle activities for all types and sizes of healthcare settings.10 The system life cycle management process is conceptualized in five major steps, which include planning, analysis, design, implementation, and maintenance.10 In the planning stage, the process is initiated; however, planning continues throughout the entire life cycle. Analysis focuses on requirements for optimal solution. Design relates to the development or customization of the CIS for the clinical setting. Implementation is then initiated. A final phase, solution maintenance, follows implementation.10

Figure 1
Figure 1
Image Tools

In the ESLC model depicted in Figure 1, the dotted line surrounding the system life cycle management process conveys the continuous influence of dynamic contextual factors occurring within the healthcare environment.10 Each segment of the system life cycle process has an identified risk zone and corresponding risk factors, further detailed in Figure 2. Although risk factors exert influence in a particular zone, they can also influence other system life cycle segments.10 These downstream effects can also shape the progression and outcomes of CIS implementation.10 Case study analysis has become a frequently used research method to study implementation strategies and context. The stories in case studies reveal how implementation strategies and contextual conditions mutually influence each other and work together to produce implementation outcomes.11

Figure 2
Figure 2
Image Tools
Back to Top | Article Outline
Outcomes Defined

We identified and used several types of outcomes relevant to CIS implementation. System outcomes refer to the results of a CIS implementation such as documentation that is legible. User outcomes refer to the "end users" of the CIS who have hands-on interaction with the system in the course of providing patient care. User-friendly refers to a clinical system that is (1) "easy" to use, with as few "key strokes" as possible to enter data; (2) "intuitive," whereby moving from screen to screen is logical to clinical end users because it matches the work process and flow of information in the clinical setting; and (3) "interfaced" such that information is entered once, but used by many. Management outcomes refer to aspects of CIS that assist in managerial decision making, operational management, meeting government regulations, benchmarking organizational performance, and funding decisions within and external to the organization. Patient outcomes are those that are directly affected by CIS implementation for patients. Reduced wait times for procedures resulting from system efficiencies, reduced frustration when not being asked the same information by multiple care providers, having direct system access to book their own appointments online, and having access to their information or to faster results reporting to care providers are examples of patient outcomes.

Back to Top | Article Outline


The search strategy was initiated by searching online bibliographical databases-CINAHL, Medline, and HealthStar from 1995 to 2005. HealthStar yielded the highest number of titles and abstracts. An online search of Computers in Nursing was completed for 1997 to 2003; Computers, Informatics, Nursing, for 2003 to 2005; and EBM Cochrane Systematic Review. The primary inclusion criteria were that the paper was written in English, involved clinical system implementation, had a healthcare facility/acute care as the setting, and included system, user, management, or patient outcome measures. Only papers that met the primary inclusion criteria were retained for review. Secondary inclusion criteria were used to determine whether research findings/evidence were being used to guide the CIS implementation (nursing informatics, theoretical model or framework, planning or implementation based on literature, adding to nursing domain knowledge, broad context for clinical system implementation, and the importance of this area of study).

Back to Top | Article Outline

The total yield from online database and online journal searches was 178 titles and abstracts, which were screened using the primary inclusion criteria. The primary author retrieved and screened 124 papers using the three primary inclusion criteria and secondary inclusion criteria of interest. The 124 papers were predominantly exploratory-descriptive in the form of case studies. A second reviewer randomly screened 20 of the 124 papers, blinded to the primary author's review. Discrepancies on four papers were discussed and consensus was reached. Of the original 124 papers, 21 were retained for review once screening criteria were applied.

Back to Top | Article Outline
Data Extraction

The following data were extracted from the 21 selected papers: author, journal, year, acute care healthcare facility, clinical system implementation, lessons learned, or outcome measures. The second level of extraction included the use of a conceptual model or framework to guide implementation, nursing informatics, nursing domain knowledge, CIS implementation in a broad context, and the relevance of the papers to this area of study.

Back to Top | Article Outline
Quality Review

The authors searched for a quality assessment instrument to review and categorize the included studies according to how well they examined and reported on CIS implementation and outcomes; however, none was found. Therefore, the primary author developed a screening tool that included the inclusion criteria of interest to reflect the dimensions of how research, nursing informatics, and nursing knowledge were applied by the authors. (A copy of the quality assessment tool may be obtained by contacting the primary author.) Possible scores could range from 1 to 16 points. A score greater than 9 was required to meet the minimum inclusion criteria.

Back to Top | Article Outline


Search Results

Only 18 studies met the mandatory inclusion criteria by achieving a score greater than 9. All 18 studies2,6,11,13-27 in the final inclusion group were published between 1995 and 2003, with 16 completed in the United States, one in Canada, and one in Australia. Of the 21 retained studies, three studies scored less than 9 on the quality assessment tool28-30 and were eliminated. The quality review-screening tool proved effective in differentiating the quality of the papers for final selection. A summary of the quality assessment scores and characteristics of the final 18 studies is presented in Table 1.

Table 1
Table 1
Image Tools
Table 1
Table 1
Image Tools
Table 1
Table 1
Image Tools
Table 1
Table 1
Image Tools

The study objective reported in all studies was the implementation of CISs in acute care facilities. Eight were identified as integrated systems, nine implied integration based on the assumption that point-of-care and nursing documentation systems were more sophisticated systems dependent on integration, and one was a beta stand-alone system. The articles included 14 case studies of exploratory-descriptive design and four research studies of preimplementation/postimplementation design. The four pre/post design studies used surveys, and of the 14 case studies, four used a survey as part of their preimplementation planning process to target appropriate end-user training.

The study populations included clinical nurses (registered and licensed practical nurses) and other healthcare practitioners. Nurses were identified as the only participants in eight of the 18 studies, whereas in seven studies, nurses were combined with other study participants such as physicians and staff from ancillary departments. An implementation project manager was identified in only three studies. In each of these, the project manager was reported to be a nurse. Project leadership by a nurse was reported in eight studies, leadership by a nursing committee in four studies, and no leadership type identified in 6 studies. Of the 18 studies, nine reported using a theoretical model to guide planning or implementation; eight referred to nursing informatics; six referred to nursing knowledge and/or the practice of nursing; and five based the CIS implementation on nursing or other literature, such as change management or adult teaching theories.

The study settings included facilities ranging in size from 100 to 1200 beds, with 15 studies in the 250- to 1200-bed size range. The study setting unit type was identified in nine studies as critical care only, critical care combined with medical-surgical, or medical-surgical settings alone. Nine studies did not identify a specific setting.

Back to Top | Article Outline
Study Outcomes

The studies were examined to identify the outcomes of CIS implementation that were anticipated by each agency. The outcomes were then categorized as system, user, management, or patient outcomes. As shown in Table 2, this content analysis resulted in 49 outcomes that consisted of nine system outcomes, 29 user/provider outcomes, nine management outcomes, and two patient outcomes.

Table 2
Table 2
Image Tools

The outcomes anticipated by the agency in each study were compared with those actually measured and achieved. A summary of anticipated outcomes of CIS implementation in the retained 18 papers and the number of times each outcome was reported is presented in Table 3. These data include factors that were observed, measured, or self-reported as leading to success or failure of implementation. System, user, management, and patient outcomes were reported, with the greatest number relating to user outcomes. In the four research studies, five outcomes were measured, with a mixed result of successes and failures. In the 14 case studies, which reported a total 45 anticipated outcomes, only 26 were achieved, of which 23.5 were identified as successes.

Table 3
Table 3
Image Tools

As described earlier in this paper, the ESLC framework (Figure 1) was chosen to categorize the factors associated with success and failure in each study into risk zones to determine if any patterns emerged. The summary of risk zone factors found to influence CIS implementation success/failure is presented in Table 4. Of the final 18 studies, 17 reported a combination of factors leading to success and failure, one reported failure only, and none reported success only. Analysis of each article revealed risk zone factors identified with each implementation in one to four of the risk zones, with risk zone factors crossing greater than or equal to three of the four zones in 15 of the 18 articles.

Table 4
Table 4
Image Tools
Table 4
Table 4
Image Tools

This analysis revealed a total of 47 success factors and 38 failure factors across several risk zones. The highest number of success and failures were in zone 3 (implementation-the zone focusing on preimplementation training and "go-live" of the system). Figure 2 provides the original risk factors associated with influencing outcomes of the ESLC. Table 4 adds additional detail to some of the risk zone factors currently listed inFigure 2 and presents the additional factors found in this review.

Back to Top | Article Outline


This study was completed to identify the state of knowledge about the implementation of clinical systems and outcomes for systems, end users, management, and patients. Applying the ESLC framework to identify risks at each phase of the implementation in the included studies yielded evidence of an emphasis on go-live dates and limited support for end users. End-user support or lack thereof was an important factor in both successful and failed implementations, respectively. This study may also be the first to test the application of the ESLC in examining CIS implementation in a systematic review. In doing so, we have suggested additional risk factors based on the empirical findings and their associations with success or failure of implementation.

This systematic review demonstrates that implementation of CIS, as point-of-care clinical systems in healthcare facilities, is of significant interest to healthcare administrators, nurses, and the nursing profession and to furthering nursing knowledge. Although the case studies are predominantly descriptive in nature, they discuss expected and achieved outcomes related to the system, users, management, and patients. Successful implementation was largely dependent on the "context" of the clinical environment, system maintenance, end-user support, evaluation to ensure system benefits, and key actors to ensure success. This review has also demonstrated the value of case study research to better understand the complexities of the implementation process.

Implementation methodology is complex, and despite research and experience, it is not possible to designate a single implementation approach that will work in all situations.11 More in-depth research is needed to identify how costly errors in implementation can be avoided.12 The findings of this synthesis suggest several areas where ongoing research could be focused or strengthened. First, future research into the implementation of CIS should consider the influence of context in the clinical environment. Second, it is important for researchers to report a clear description of the CIS interventions. The expected outcomes and approaches to measurement of the intervention and outcome should also be reported at the levels of the system, users, managers, and patients. Effects on operational procedures such as patient wait-lists and on broader issues such as system sustainability should also be documented. Longitudinal investigation is required to observe the complexity of CIS, to develop strategies to mitigate risks associated with CIS implementation, and to support and maintain outcomes over time.

The findings of this review also suggest the need to establish regular processes for auditing electronic documentation to measure and ensure data quality, target areas of improvement, and increase confidence in decision making based on the data. Long-term attention to end-user training and support is key to data quality and reliability for management decision making and healthcare funding decisions. To shortchange training and long-term support when implementing a CIS jeopardizes anticipated return on investment in quality patient care.

It is important to develop evidence-based funding decisions and CIS postimplementation support models that help decision makers determine how best to support clinical end users and that provide direction to information technology departments and administrators on how best to support and maintain CIS on an ongoing basis.

The importance of involving and educating nurses in all aspects of CIS implementation and support is evident from the findings of this review. Nurses can and should be educated in nursing informatics and project management methods to prepare for, participate in, and lead clinical systems implementation. Nurses should also be informed about the importance of data integrity to support decision making related to patient care and organizational outcomes. Research into training methods best suited to this workforce is needed with the recognition that an investment in training can bring about the best of all that is already being invested in a highly skilled and specialized workforce.5

These findings also suggest that it is important for nurses to recognize the potential influence that clinical systems implementation can have on nursing practice and the nursing profession and on furthering nursing knowledge. Nurses use advanced information technology to provide care, document that care, research better treatment methods, and transfer knowledge to colleagues at every level of experience, in every specialty and in every care setting imaginable.8 It is also valuable for nurses to recognize the opportunity that CIS can bring about in standardizing, describing, and studying the contribution of nursing in achieving positive patient outcomes.

This review has three potential limitations. First, a potential reporting bias may exist by including only published studies that tend to overreport positive findings. However, in this review, we have balance, as both negative and positive findings were reported in most of the literature reviewed, with none of the published studies reporting positive findings only. Second, only US, Canadian, and Australian articles describing the process of CIS implementation and outcomes were reported in the literature. Although implementation of computer systems in healthcare organizations is extensive, it is apparent that more countries are experiencing CIS implementation than are publishing their findings. Third, our search strategy restricted titles or abstracts to English language because it was the language of proficiency of our research team. This may have resulted in overlooking additional evidence of specific cultural factors that influence CIS implementation in nursing.

Back to Top | Article Outline


In this systematic review, we have demonstrated that research into the process of CIS implementation and outcomes is (1) relatively recent, (2) of significant interest, (3) primarily descriptive, and (4) an evolving field given the existence of very little theoretical work. The findings of this review confirm the rapid advancement of CISs, the complexity of CIS implementation, and the importance of viewing system implementation in the context of diverse clinical environments. Use of the ESLC model confirms that support within and across each risk zone is key to facilitating clinical system implementation and to achieving defined outcomes.9 Time pressures are often associated with the implementation of clinical systems, and research is urgently needed to mitigate the risks of failure, guide the implementation process within specific clinical environments, and achieve expected outcomes. Technology is leaving its mark on nursing,7 and nursing professionals will become active participants and leaders in the development and implementation of CIS as they recognize the impact and potential of CIS on nursing practice.

Back to Top | Article Outline


1. Axford R, Carter B. Impact of clinical information systems on nursing practice: nurses' perspective. Comput Nurs. 1996;14:156-163.

2. Marasovic C, Kenney C, Elliott D, Sindhusake D. Attitudes of Australian nurses toward the implementation of a clinical information system. Comput Nurs. 1997;15:91-98.

3. Linton J. Implementation research: state of the art and future directions. Technovation. 2002;22:65-79.

4. Struder M. The effects of organizational factors on the implementation of EMR system implementation-what have we learned? Electr Healthc. 2005;4(2):92-98.

5. Simpson R. Are staff prepared for the new information-based hospital enterprise? Nurs Adm Q. 1997;21(2):85-87.

6. Anderson LK, Stafford CJ. The "big bang" implementation: not for the faint of heart…hospital replaces its outdated mainframe with a new integrated information system during the course of one weekend. Comput Nurs. 2002;20:14-22.

7. Simpson R. Where will we be in 2015? Nurs Manag. 2004;35:38-44.

8. Simpson R. Healing healthcare, healing nursing in the 21st century. Nurs Adm Q. 2002;26(5):94-98.

9. Brink P, Woods M. Advanced Design in Nursing Research. Newbury Park, CA: Sage Publications; 1989:319.

10. Thompson CB, Snyder-Halpern R, Staggers N. Analysis, processes, and techniques: case study. Comput Nurs. 1999;17(5):203-206.

11. Pare G, Elam JJ. Introducing information technology in the clinical setting. Lessons learned in a trauma center. Int J Technol Assess Healthc. 1998;14:331-343.

12. Ball MJ. Hospital information systems: perspectives on problems and prospects. Int J Med Inform. 2003;69:83-89.

13. Dillon TW, Lending D, Crews TR II, Blankenship R. Nursing self-efficacy of an integrated clinical and administrative information system. Comput Nurs. 2003;21:198-205.

14. Larrabee JH, Boldreghini S, ElderSorrells K, et al. Evaluation of documentation before and after implementation of a nursing information system in an acute care hospital. Comput Nurs. 2001;19:56-58.

15. Nahm R, Poston I. Measurement of the effects of an integrated, point-of-care computer system on quality of nursing documentation and patient satisfaction. Comput Nurs. 2000;18:220-229.

16. Barr BJ. Managing change during an information systems transition. AORN J. 2002;75(6):1085-1092.

17. Brooks Massanari K. Implementation of NANDA nursing diagnoses online. Comput Nurs. 1998;16:320-326.

18. Cheung WT, Hamilton M. End-user support improvement at Memorial Sloan Kettering cancer center. Health Inf Manage. 1997;11:13-20.

19. Doyle K, Kowba M. Connecting points: automation: managing the human side of change to automation. Comput Nurs. 1997;15:67-68.

20. Halley EC, Kambic PM. Concurrent process redesign and clinical documentation system implementation: a 6-month success story. Top Health Inf Manage. 1996;17(1):12-17.

21. Johnson T. Functional health pattern assessment on-line: lessons learned. Comput Nurs. 2000;18:248-254.

22. LaDuke S. Online nursing documentation: finding a middle ground. J Nurs Adm. 2001;31:283-286.

23. Lytle KS, Bailey DW, Dorman KF, Moos MK. Just a beta…. Proc AMIA Symp. 1999:580-584.

24. Murphy F, Karmali K, McFarlane G, Augustin K. Education strategy for the implementation of computerized nursing documentation. Medinfo. 1995;8(Pt 2):1390-1393.

25. Stebbins JK, Gentes JM, KagenFishkind JE, LaMountain S, Sheehan SM, Colburn CA. A clinical information system transition: a nursing perspective. J Nurs Staff Dev. 1997;13:309-313.

26. Travers D. Innovative informatics. The need for quick computer response with minimal input: development of an ED computer system. J Emerg Nurs. 1997;23:259-264.

27. Whitman BL, Hamann SK, Vossler BL. A training plan for bedside computers. J Nurs Staff Dev. 1997;13:33-36.

28. Husting PM, Cintron L. Healthcare Information systems: education lessons learned. J Nurs Staff Dev. 2003;19(5):253-257.

29. Littlejohns P, Wyatt JC, Garvican L. Evaluating computerized health information systems: hard lessons still to be learnt. BMJ. 2003;326(7394):860-863.

30. Webb J. Breaking through the pain barrier-the reality of EPR implementation. Br J Healthc Comput Inf Manage. 1999;16(6):19-21.

Cited By:

This article has been cited 4 time(s).

Artificial Intelligence in Medicine
Impact of four training conditions on physician use of a web-based clinical decision support system
Kealey, E; Leckman-Westin, E; Finnerty, MT
Artificial Intelligence in Medicine, 59(1): 39-44.
Proceedings of the 18Th European Congress of Physical and Rehabilitation Medicine (Prm)
The Information System Technologies in rehabilitation procedures. The "Arogi" case
Giannakitzidis, C; Hatzoglou, N; Gkoumas, C; Vorniotakis, P; Loizidis, T
Proceedings of the 18Th European Congress of Physical and Rehabilitation Medicine (Prm), (): 56-59.

International Journal of Medical Informatics
Organizational issues in the implementation and adoption of health information technology innovations: An interpretative review
Cresswell, K; Sheikh, A
International Journal of Medical Informatics, 82(5): E73-E86.
2011 4Th International Conference of Education, Research and Innovation (Iceri)
Training of Nurses for the Effective Implementation of Information and Communication Technology in Health Care Systems
Galimany, J; Roca, M; Falco, A
2011 4Th International Conference of Education, Research and Innovation (Iceri), (): 973-976.

Back to Top | Article Outline

Clinical information systems; Expanded Systems Life Cycle; Implementation; Information management; Nursing informatics

© 2009 Lippincott Williams & Wilkins, Inc.



Article Tools


CE Test


Article Level Metrics

Search for Similar Articles
You may search for similar articles that contain these same keywords or you may modify the keyword list to augment your search.