Secondary Logo

Stakeholders External Evaluation of the Clinical Nurse Transition Program

Seimetz, John A., DM, MSN, RN, NE-BC

Journal for Nurses in Professional Development: May/June 2019 - Volume 35 - Issue 3 - p E6–E10
doi: 10.1097/NND.0000000000000535
Articles: Online Only
Free

Nurse transition programs have predominantly used internal evaluations to evaluate quality outcomes, and thus, there appears to be a lack of current literature on external evaluation of these programs. External evaluation provides an objective and valuable tool for the nursing professional development practitioner to demonstrate return on investment. The results support the use of the innovative and groundbreaking Clinical Nurse Transition Program Evaluation Tool to externally evaluate the nurse transition program.

John A. Seimetz, DM, MSN, RN, NE-BC, is Director, CNTP, Brooke Army Medical Center, JBSA Fort Sam Houston, Texas.

The views expressed herein are those of the author and do not reflect the official policy or position of Brooke Army Medical Center, the U.S. Army Medical Department, the U.S. Army Office of the Surgeon General, the Department of the Army, the Department of the Air Force, the Department of Defense, or the U.S. Government.

The author declares no conflicts of interest.

ADDRESS FOR CORRESPONDENCE: John A. Seimetz, Brooke Army Medical Center, 3551 Roger Brooke Drive, JBSA Fort Sam Houston, TX 78234-6200 (e-mail: John.A.Seimetz.civ@mail.mil).

This project was conducted in the largest Department of Defense Tri-Service regional medical facility and Level 1 Continental United States Trauma Center, with both inpatient and outpatient services. Care is provided by approximately 8,900 staff members, including active duty military personnel from each of the uniformed services, federal civilian employees, and contractors. This medical center is located in Southwest United States.

The Clinical Nurse Transition Program (CNTP) began its journey at this medical center in October 2008, and through June 2018, there have been 450 graduates. As part of an ongoing improvement process is the requirement for the use of stakeholders to evaluate quality outcomes. Evaluating outcomes can be accomplished using internal or external validation. Internal validation is subjective and conducted by individuals who are intrinsically connected with the project. A main tool used is the survey. The advantage of internal validation is firsthand knowledge, but the disadvantage is the potential for bias. External validation is conducted by an individual, frequently using an evaluation tool, who is not directly associated with the project. A distinct advantage to external validation is the objectivity, mitigation of bias, and a fresh perspective (Church & Waclawski, 1989).

In examining how other transition programs are evaluated, many organizations use a common technique used for evaluation, which relies on internal assessment by the nurses via surveys (Hatzenbuhler & Klein, 2018; McDonald, Brown, & Knihnitski, 2018; Wilson, Weathers, & Forneris, 2018). In addition, other organizations measure the longevity of their nurses after program completion as a measure of success (Burr & Malagon-Maldonado, 2014; Cochran, 2017; Innes & Calleja, 2018; Trepanier, Early, Ulrich, & Cherry, 2012; Wall, Fetherston, & Browne, 2018). There are organizations that use internal evaluations to evaluate programs; however, there appears to be a lack of external evaluation of nurse transition programs (Aggar, Gordon, Thomas, Wadsworth, & Bloomfield, 2018; Cochran, 2017; Innes & Calleja, 2018).

This project began with three goals. The first goal was to obtain a tool to externally evaluate the CNTP. Because there were discussions, at the executive decision-making level, regarding the appropriate length of the CNTP, one suggesting 12 weeks versus the established 24 weeks, the second goal was to evaluate the CNTP nurses at 12 weeks into the program and again at the completion of the program (24 weeks). The third goal was to evaluate the performance of the CNTP nurse at least 6 months after graduation to evaluate CNTP return on investment. For all three goals, there was no tool to externally evaluate CNTP nurses that was both accessible and cost-effective for this organization. Furthermore, there was no supporting data for the second or third goals.

In addressing the second and third goals of this project, the elements of nursing professional development (NPD) became foundational. By definition, NPD enhances the growth and development of nurses as they progress from novice to expert (Benner, 1984; Harper & Maloney, 2016; Warren & Harper, 2017). The external evaluation of CNTP and the valuable lessons learned are one way to evaluate the return on this investment and provided the NPD practitioner another substantial tool to “demonstrate their value to the organization” (Harper, 2016, p. 3).

Back to Top | Article Outline

METHODS

In exploring potential tools to externally evaluate the transition nurses, many published evaluation tools were either too complex and/or expensive. As a result, a tool was required to be long enough to capture the necessary information, but not too long as to create any survey fatigue in the participant (Church & Waclawski, 1989). Thus, the Clinical Nurse Transition Program Evaluation Tool (CNTP-ET) was specifically designed for the purpose of this project.

Back to Top | Article Outline

The CNTP-ET

The CNTP-ET was derived from the Assessment Tool to Evaluate the Struggling CNTP RN, Appendix J, CNTP Guidelines (U.S. Department of the Army, 2015). Once developed, the CNTP-ET was modified by a small working group of three CNTP directors. The tool was reviewed collectively by nine other CNTP directors within the program and underwent multiple reviews and revisions until a suitable tool was ready for submission. The finalized tool was approved by the Dean, School of Nursing Science, Chief, Department of Nursing Science, Army Medical Department Center and School, Health Readiness Center of Excellence, Joint Base San Antonio Fort Sam Houston, Texas.

The CMTP-ET is a one-page evaluation tool (see Figure 1), which consists of seven elements: Ability, Standards, Knowledge and Skills, Delegation, Feedback, Environment, and Professional Attitude. Under the seven elements are 15 specific subskills. The CNTP nurse is evaluated based on observed performances by their Clinical Nurse Officer in Charge or their Assistant Clinical Nurse Officer in Charge using the CNTP-ET by means of a 5-point Likert scale based on Benner’s model. Level 1 Observe (Novice) observed the clinical activity performed by others. Level 2 Assist (Novice) assisted the preceptor to perform the clinical activity. Level 3 Direct Supervision (Advanced Beginner) performed the entire activity under preceptor’s direct supervision and coaching. Level 4 Indirect Supervision (Advanced Beginner) performed the entire activity with indirect supervision (requires follow-up or partial presence). Level 5 Independent (Competent) performed the entire activity to standard without need for supervision or coaching (includes proper documentation; Benner, 1984).

FIGURE 1

FIGURE 1

An external evaluation by supervisors has a long standing history in both the civilian sectors and the military. The CNTP-ET is an extension of that external evaluation and is prescriptive in nature to evaluate the observer performance of the recent CNTP graduate nurse. The information collected in this project will be forwarded to senior leadership to assist in strategic planning and decision-making.

The next step was to test the tool using Cronbach’s alpha to measure internal consistency because the CNTP-ET had not been used. The CNTP-ET was evaluated by nurse leaders prior to use on CNTP nurses. The Clinical Nurse Officer in Charge, Assistant Clinical Nurse Officer in Charge, charge nurse, and staff nurses were invited to evaluate the CNTP-ET. These leaders were from eight different and diverse nursing units where CNTP nurses are assigned.

Although most of these nurses were not NPD practitioners, their competence was consistent with the NPD generalist competencies for collegiality and advocacy (Harper & Maloney, 2016). The medical center is a training platform, and many of the CNTP nurses become preceptors after 1 year. Nurses evaluating the CNTP-ET were required to have greater than 1 year nursing experience, and no contractors were used. The CNTP-ET was sent to 228 nurse leaders, of which 109 responded (47.81%) with a useable response rate of 43%. “Internal surveys will generally receive a 30–40% response rate (or more) on average, compared to an average 10–15% response rate for external surveys” (Fryrear, 2015).

The education degree of the nurse evaluators ranged from associate’s to master’s, with bachelor’s degree dominating at 73.47% (n = 72), master’s degree at 15.31% (n = 72), and associate’s degree at 7.14% (n = 7). The years of nursing experience for the nurse evaluators ranged from 1 year to greater than 20 years. The 1–4 years of experience groups comprised 52.04% (n = 51), whereas the 4–10 years of experience group was 15.31% (n = 15), the 10–14 years of experience group was 13.27% (n = 13), the 14–20 years of experience group was 11.22% (n = 11), and the greater than 20 years of experience group was 5.10% (n = 5). The status of the nurse evaluators was well distributed with the Army at 43.88% (n = 43), the General Schedule (GS-civilian) at 32.65% (n = 32), and the Air Force at 20.41% (n = 20).

Information from the nurse evaluators using the CNTP-ET and rating the CNTP nurse at 12 and 24 weeks was processed using SPSS Version 22 for a Cronbach’s alpha of .965 (see Table 1). In further exploring the Cronbach’s alpha if items were deleted revealed the following values: Questions 1–17, 20, 22–25, 27, and 29: α = .964; Questions 18 and 26: α = .963; Questions 19, 21, 28, and 30: α = .965.

TABLE 1

TABLE 1

Back to Top | Article Outline

CNTP 12 weeks versus 24 weeks

Using the premise that a nurse will function at a higher level of clinical proficiency after 24 weeks of CNTP experience rather than with 12 weeks of CNTP experience, the CNTP-ET was used to collect information on the CNTP nurses in the program (Cochran, 2017). The information collected was exclusively obtained at this medical center, and groups were small ranging from 7 to 12 CNTP nurses per group. Over the last year, four groups were evaluated: Group 43 (n = 8), Group 44 (n = 7), Group 45 (n = 9), and Group 46 (n = 12). The information collected from these small groups were combined for investigation (n = 36).

Using SPSS Version 22, for a paired t test, the correlation was .485, with a significance of .003. The paired differences indicated the 24-week CNTP nurses’ observed performance was 5.99914 greater than the 12-week CNTP nurses’ observed performance, at a .05 confidence interval and p = .000 (two-tailed), respectively (see Table 2).

TABLE 2

TABLE 2

Back to Top | Article Outline

CNTP-ET 6 months after graduation

The third goal of the project was to evaluate the performance of the CNTP nurse at least 6 months following graduation as a means to evaluate CNTP return on investment. The premise was a CNTP graduate nurse would be performing above Level 3 (Level 3 Direct Supervision, Advanced Beginner) Performed the entire activity under preceptor’s direct supervision and coaching 6 months post CNTP graduation as defined in the CNTP-ET.

The information collected was exclusively obtained at this medical center, and groups were small ranging from four to eight CNTP graduate nurses per group. Over the last year, six groups were evaluated: Group 38 (n = 6), Group 39 (n = 5), Group 40 (n = 6), Group 41 (n = 8), Group 42 (n = 4), and Group 43 (n = 8). The information collected from these small groups were combined for investigation (n = 37).

Using SPSS Version 22, for a standard t test, indicated the CNTP graduate nurses were performing above Level 3 (Level 3 Direct Supervision, Advanced Beginner) Performed the entire activity under preceptor’s direct supervision and coaching 6 months post CNTP graduation as defined in the CNTP-ET. This information was at a .05 confidence interval and p = .000 (two-tailed), respectively (see Table 3).

TABLE 3

TABLE 3

Back to Top | Article Outline

Restrictions

Valuable information was obtained for this quality improvement project. The CNTP-ET was tested exclusively at this medical center. All of the groups were small, and information was combined from the groups for investigation. All the lessons learned are specific to this medical center.

Back to Top | Article Outline

LESSONS LEARNED

Commercial tools to externally evaluate nurse transition programs were cost prohibitive for this medical center. This medical center now has the CNTP-ET with a Cronbach’s alpha of .965 to externally evaluate CNTP nurses. Also, information supports the 24-week CNTP versus the 12-week CNTP (correlation .485; significance .003, and a 5.99914 higher score for 24 weeks vs. 12 weeks). In addition, this medical center has information that externally validates CNTP nurses’ performance 6 months post CNTP graduation at greater than a level 3 (p = .000, for the last six CNTP groups, n = 37). This is a significant step in the external evaluation and consistent with the standards of the NPD by providing tangible evidence to support the contributions to the organization’s quality and safety environment (Harper, 2016).

Back to Top | Article Outline

CONCLUSION

External validation of CNTP by NPD practitioners fully supports the standards of NPD (Harper, 2016). The CNTP-ET can be piloted in other healthcare originations to externally validate the clinical performance of the CNTP graduate and provide leadership with quantifiable decision-making data. Based on trends or individual performance, the CNTP-ET can also be used to identify areas in need of remedial training. Although not totally inclusive, the CNTP-ET can be expanded and tailored to include other clinical performance elements indigenous to one’s respective healthcare organization.

Back to Top | Article Outline

References

Aggar C., Gordon C. J., Thomas T. H. T., Wadsworth L., & Bloomfield J. (2018). Evaluation of a community transition to professional practice program for graduate registered nurses in Australia. Nurse Education in Practice, 32, 101–107.
Benner P. (1984). From novice to expert: Excellence and power in clinical nurse practice. Menlo Park, CA: Addison-Wesley.
Burr S. G., & Malagon-Maldonado G. (2014). Implementation of a nurse residency program in a women’s’ and newborn's specialty hospital: A magical transformation. Journal of Obstetric, Gynecologic & Neonatal Nursing, 43(Suppl. 1), S57.
Church A. H., & Waclawski J. (1989). Designing and using organizational surveys: A seven-step process. San Francisco, CA: Josses-Bass.
Cochran C. (2017). Effectiveness and best practice of nurse residency programs: A literature review. Medsurg Nursing, 26(1), 53–63.
Fryrear A. (2015, July 22). What's a good survey response rate? SurveyGizmo. Retrieved from https://www.surveygizmo.com/survey-blog/survey-response-rates/
Harper M. G. (2016). NPD roles: Generalists vs. specialist. Retrieved from http://www.anpd.org/blog/npd-roles-generalist-vs.-specialist
Harper M. G., & Maloney P. (2016). Nursing professional development: Scope and standards of practice (3rd ed.). Chicago, IL: Association for Nursing Professional Development.
Hatzenbuhler N. J., & Klein J. E. (2018). Educational preparation for clinical practice: Reflections of newly graduated RNs. Nurse Educator. Advance online publication. doi:10.1097/NNE.0000000000000550.
Innes T., & Calleja P. (2018). Transition support for new graduate and novice nurses in critical care settings: An integrative review of the literature. Nurse Education in Practice, 30, 62–72.
McDonald M., Brown J., & Knihnitski C. (2018). Student perception of initial transition into a nursing program: A mixed methods research study. Nurse Education Today, 64, 85–92.
Trepanier S., Early S., Ulrich B., & Cherry B. (2012). New graduate nurse residency program: A cost–benefit analysis based on turnover and contract labor usage. Nursing Economic$, 30(4), 207–214.
U.S. Department of the Army. (2015, December 15). BG(R) Anna Mae Hays Clinical Nurse Transition Program (CNTP) guidelines. Joint Base San Antonio Fort Sam Houston, TX: Headquarters, U.S. Army Medical Department Center and School, U.S. Army Health Readiness Center of Excellence.
Wall P., Fetherston C., & Browne C. (2018). Understanding the enrolled nurse to registered nurse journey through a model adapted from Schlossberg's transition theory. Nurse Education Today, 67, 6–14.
Warren J. I., & Harper M. G. (2017). Transforming roles of nursing professional development practitioners. Journal for Nurses in Professional Development, 33(1), 2–12.
Wilson T., Weathers N., & Forneris L. (2018). Evaluation of outcomes from an online nurse residency program. Journal of Nursing Administration, 48(10), 495–501.
© 2019 by Lippincott Williams & Wilkins, Inc.