Secondary Logo

Journal Logo


A Formative Program Evaluation of Electronic Clinical Tracking System Documentation to Meet National Core Competencies

Smith, Lynette S. PhD, APRN, PMHNP-BC, FNP-BC; Branstetter, M. Laurie DNP, APRN, FNP-BC

Author Information
CIN: Computers, Informatics, Nursing: September 2016 - Volume 34 - Issue 9 - p 393-401
doi: 10.1097/CIN.0000000000000254


Electronic clinical tracking systems (ECTSs) have been utilized in advanced practice RN (APRN) education to document students’ clinical experiences.1–7 Electronic clinical tracking system has been defined as technology that is used to track students’ clinical information, which includes healthcare populations and management information for review and analyses by APRN faculty.8 The use of ECTS is underpinned by the Institute of Medicine and Technology Informatics Guiding Education Reform initiatives, which support the use of electronic health records to provide quality and safety in APRN education.9,10 Electronic clinical tracking system provides students with experience documenting patient encounters in clinical settings,8 thus promoting student synthesis of their knowledge and skills related to APRN clinical and nonclinical situations.

The National Organization of Nurse Practitioner Faculties (NONPF) has developed population-focused core competencies for APRN education.11 The NONPF competencies provide entry-level expertise that is population focused throughout the lifespan.11 Graduate nursing programs are burdened with providing evidence that APRN students are meeting population-focused core competencies. The Family/Across the Lifespan NP Competencies were reviewed and only leadership and independent practice population foci competency areas were identified by NONPF as pertinent for family nurse practitioner (FNP) programs. These two population foci will serve as the definition of core competencies for the remainder of this study.

Documentation of students’ clinical experiences in an ECTS should provide information in meeting core competencies in an FNP program. To date, there is a paucity of research studies examining whether ECTS has the capacity to document students’ clinical experiences to meet core competencies. However, one integrative review was identified as evaluating only NONPF Technology and Information Literacy competency.8 Therefore, program-selected ECTS should be evaluated for potential to document core competencies. One ECTS commercially available to nursing programs is Medatrax (Medatrax, Pulaski, TN).


Medatrax is a Web-based ECTS designed to be utilized by undergraduate and graduate nursing students in their respective educational programs.12 The system serves as a permanent searchable repository for individual student clinical experiences across specific courses within an educational program. Faculty has the ability with ECTS to aggregate individual and cohort student data across clinical experiences. These objective data provide population summaries and performed skill sets. A clinical e-portfolio, with summaries of each population and skill sets performed, is available at the end of their educational program for students to utilize in seeking employment postprogram completion. The e-portfolio allows for a student to display each of the competency-based accomplishments in an objective format.1 A student’s documentation of clinical encounters reflects his/her performance and is an indicator of attaining core competencies, which is a reflection of student readiness for entry into advanced practice. The benefits of this Web-based ECTS are the configurable and adaptable functionality of the system.12 This flexibility provides individual programs with options for customization to provide cultural and academic diversity requirements of each institute of higher education.12 Lastly, Medatrax provides faculty and student support for questions and concerns. For the remainder of this study, the term ECTS will be used to identify Medatrax. The purpose of this study is to evaluate ECTS using a formative program evaluation approach to determine whether students’ clinical documentation meets core competencies in a midsouthern FNP program.


Formative program evaluation underpins the process to determine if an ECTS is effective in meeting core competencies in an FNP program. A formative program evaluation is defined as the evaluation of a program in process to determine a program outcome.13 Formative program evaluation consists of inputs, activities, outputs, outcomes, and impacts. Inputs or resources in this evaluation are ECTS and core competencies. Activities describe ECTS documentation properties of students’ clinical experiences and are paired to the core competencies to determine which ECTS functions reflect individual core competencies. Outputs are demonstrated by developing a side-by-side table matching core competencies to corresponding ECTS documentation. Outcomes are found in the Results and report specific ECTS documentation to meet core competencies. Impacts of ECTS documentation in meeting core competencies are found in the Discussion. A logic model is used to illustrate the formative program evaluation framework (Table 1).

Table 1
Table 1:
Formative Program Evaluation Framework


Electronic Clinical Tracking System

Advanced practice RN education focuses on providing students with substantive clinical experiences in which to implement attained knowledge and skills acquired through didactic instruction. Historically, clinical experiences were documented in handwritten clinical logs or electronic listings of patients evaluated.14,15 This process left much to be desired in evaluating the overview of students’ clinical experiences. Problems with handwritten clinical logs included illegibility, absent data, and timeliness of evaluation.14,15 Electronic clinical tracking system allows for legibility of clinical logs, complete data sets, and real-time evaluations of students’ clinical experiences. Cumulative data of students’ clinical encounters are available to faculty throughout program progression.16–18

Core Competencies

The NONPF published the first set of core competencies in 1990 for APRN entry into practice.19 These seminal documents have been published for APRN specialty practice and as national guidelines for APRN education.19 Updates and modifications have occurred to meet educational needs of an ever-changing healthcare environment. The NONPF supports credentialing bodies in serving the public through the application of core competencies, which set standards for higher education in nursing.10 The APRN programs are obligated to develop specialty practice didactic and clinical experiences for students that reflect the integration of core competency areas.


The potential in ECTS to provide documentation of competency areas was identified. First, the ability of ECTS to document individual student encounters was evaluated, including demographic information, encounter times, vital signs, advanced practice examination skills, evaluation/management services, International Classification of Diseases (ICD) diagnostic codes, and Current Procedural Terminology (CPT) code assignments. Other documentation includes patient counseling provided, medications prescribed, and interdisciplinary consultations, when applicable. Second, ECTS provides a venue for clinical note templates utilized in students’ documentation of program-specific clinical encounters. These functions within ECTS allow for faculty review and overview of students’ clinical experiences to meet program outcomes, which reflect the core competencies.16–18


A table was developed to illustrate alignment of ECTS documentation to specific core competencies in the areas of leadership and independent practice (Table 2). Each competency component was reviewed. Then, ECTS documentation was subjectively determined by the authors to meet respective components. Not all core competency components could be matched with a documentation method in ECTS.

Table 2
Table 2:
Alignment of ECTS in Meeting Core Competencies



Students’ clinical documentations in ECTS were evaluated to meet core competencies. The results were categorized into each core competency being directly met, indirectly met, or not met. Each process of the formative program evaluation framework was reviewed. The inputs for this evaluation are ECTS and core competencies; activities include a review of ECTS documentation and alignment with specific core competencies; and outputs resulted in the development of Table 2. The outcomes of this study evaluate documentation in an ECTS to meet core competencies. This study finds that ECTS documentation partially meets leadership and independent practice core competencies.


Leadership core competencies are based on tenets of complex and advanced leadership roles, critical and reflective thinking, collaboration and communication, and utilizing principles of change.19 A clinical experience with a preceptor provides a venue for students to develop professional relationships, access resources, and attain knowledge of patient care strategies in the clinical setting. Practice is an effective means to build on leadership qualities in students’ experiences.20 Electronic clinical tracking system provides a searchable and retrievable means of documenting students’ clinical experiences and illustrating some leadership competencies.

Leadership competencies are partially met through students’ documentations in ECTS, directly and indirectly. Direct evidence of meeting competencies includes the ability to objectively categorize and document specific data points. Indirect evidence requires preceptors’ and faculty’s subjective appraisals of students’ clinical decision-making rationales as accurate and appropriate in documenting objective information for meeting core competencies. Interdisciplinary referrals, patient education, and counseling provided are directly evidenced in documentations of clinical experiences. Indirect evaluation of mutual respect and shared values in leadership core competencies are reflective of students’ critical thinking, collaboration, and communication that are required in the medical plan process through practicing coordinated patient-centered care. However, ECTS does not document other forms of leadership expression in clinical practice, such as oral communications, participation in professional organizations, or formal writing. Therefore, leadership core competencies are partially met through students’ documentations in ECTS.

Independent Practice

Independent practice and accountability in clinical practice are goals of APRN education. Optimal clinical practice includes achievement and internalization of concepts from the program sciences: pharmacology, physiology, and pathophysiology.20 Components of patient encounters in students’ clinical logs lend themselves easily to ECTS documentation to reflect scientific concepts. The details of individual patient encounters are configurable in ECTS to requirements specified by individual educational institutions and data points selected by faculty to reflect course objectives.12

Independent practice competencies are partially met through students’ ECTS documentations; however, not all competencies are met. Core competency demonstration was evaluated by determining direct or indirect evidence in ECTS documentation. It became apparent as the authors worked through the competencies that the logical approach to analyze the independent practice core competencies was to report the findings according to the principles of American Nurses Association (ANA) Standards of Practice.21 The ANA is recognized as the national professional association of nursing.21 The ANA Standards of Professional Performance underpins each of the tenets described in the ANA Standards of Practice. The six professional nursing practice standards are essential to evaluating the core competencies of APRN practice. The first six Standards of Practice are used to present the results of independent practice core competencies.21 These Standards of Practice are assessment, diagnosis, outcome identification, planning, implementation, and evaluation.21

The first ANA standard, assessment, involves the comprehensive collection, prioritization, and documentation of patient data with synthesis and interpretation.21 The problem-oriented medical record (POMR) or assessment, plan, subjective, and objective (APSO) notes22 are an accepted method of documentation of clinical encounters specific to a clinical facility. For the remainder of this article, clinical note will be the term used to describe POMR or APSO note. Core Competencies 1, 2, and 3 are met directly by students’ clinical note documentations in ECTS. However, Core Competency 5 is met indirectly. The acute or chronic diagnoses and the social history, inclusive of family support, can be documented directly in ECTS, but students’ understanding of illnesses that affect families are evaluated by either or both preceptors and faculty, indicating an indirect relationship with Competency 5.

The next ANA standard, diagnosis, involves a diagnostic process that includes a systematic approach in developing differential diagnoses utilizing clinical reasoning based on physiological, psychological, and developmental variations.21 Independent practice Core Competency 10 is met directly by students’ differential diagnoses documentation in clinical notes and illustration of pertinent positive and negative subjective and objective evidence in support of the final diagnosis in ECTS. Core Competency 9 is met indirectly. Diagnostic tools are documented objectively in ECTS, but preceptors and faculty are needed to determine the appropriateness of students’ considerations of strategies, risks, benefits, and costs to patients.

Outcome identification is the third ANA standard that utilizes evidence-based practice and cost-effectiveness, which leads to expected clinical outcomes.21 Electronic clinical tracking system has the capacity to document assessments of patients with and without acute or chronic disease, or age-related physical changes. However, analytical processes performed by students require evaluation in the clinical setting to interpret findings as pathological versus age-related in origin. Although ECTS directly documents Competencies 6 and 20, these competencies require indirect interpretation by preceptors or faculty for validation of students’ performances. Achievement of cost-effectiveness can be assessed through accurate diagnoses and evidence-based interventions resulting in expected patients’ clinical outcomes. Clinical outcomes are not part of ECTS documentation, rather a function of patients’ clinical records that are subjectively evaluated over time, therefore not an expected function within ECTS.

Planning and implementation21 are separate principles yet function as a fluid process in practical application. The ANA Standards 4 and 5 use expert clinical knowledge to synthesize holistic care planning and collaboration for mutually safe, culturally, and socially acceptable healthcare for patients that may include pharmacologic, therapeutic, and interprofessional care when appropriate.21 Core Competencies 4, 15, 18, 21, and 24 focus on the planning aspects in students’ achievements of evidence-based entry-level practice competencies. Clinical note documentations in ECTS have the capacity to directly document students’ decisions regarding identification of individualized interventions, culturally sensitive plans of care, promotion of patients’ self-care needs, and referrals. However, students’ reasoning processes utilizing developmental and family theories essential for individual care, family care, and referrals require preceptor or faculty collaboration for clinical appropriateness.

Students’ ECTS documentations of clinical encounters include special populations across the lifespan. Planning and interventional implementation are the focus of Core Competencies 11, 12, 13, 14, 16, and 17. Prescribing medications, therapeutic devices, and performing primary care procedures for acute and chronic conditions in all populations directly meet the core competencies via students’ encounter logs and clinical notes. The safety and appropriateness of medications and interventions to holistically meet unique and complex population needs require supervisory evaluations. These core competencies are indirectly met using ECTS. Palliative care, end-of-life care, and referrals reflect collaborations among patients, students, preceptors, social support, and financial resources to meet patients’ complex needs during life’s transitions. Students’ documentations in ECTS clinical notes indirectly reflect collaborative decisions in patient care encounters requiring supervisory guidance.

Students’ documentations in ECTS do not meet Core Competencies 7, 8, 19, and 23. Electronic clinical tracking system documentations can reflect decisions made in clinical encounters, but cannot assess students’ syntheses of knowledge. Comprehensive facets in decision-making processes, quality of student-patient consultations, multiple sources of data syntheses, and referrals require preceptors and faculty to observe synthesis of students’ appropriate decision-making methods and self-efficacy required for independent practice.

The final ANA standard, evaluation, encompasses diagnostic accuracy and interventional effectiveness when evaluating patient outcomes with adaptation of healthcare plans to attain expected outcomes.21 Students’ documentation in ECTS does not meet Core Competencies 22 and 25. Students have the ability to document diagnoses and coordination of care in clinical notes. However, students’ documentations in ECTS cannot directly or indirectly monitor outcome effectiveness or evaluate illness and situational impacts on families and communities as this is a function of patients’ clinical records and is subjectively evaluated by providers over time, therefore not an expected function within ECTS.



In this formative program evaluation, impacts are determined by evaluation of students’ clinical documentations in ECTS as they relate to core competencies. Core competencies can be illustrated through students’ clinical ECTS documentations in one of three ways: directly met, indirectly met, and not met using students’ documentations in ECTS.

Competencies may be directly met through students’ clinical documentation in ECTS. Complex skills performed by students are reflected within ECTS documentation, for example, coding of evaluation and management services, ICD diagnostic codes, and CPT code assignments. Electronic clinical tracking system is also a permanent repository for complex decisions made in the clinical encounter such as students’ referrals, educational counseling, prescribing practices, and therapeutic modalities. Age-appropriate advanced practice skills lists were developed for FNP students within ECTS to reflect individual student physical examination experiences in the clinical setting. Faculty and preceptors are able to review and evaluate summaries of students’ skills lists in order to tailor clinical experiences to optimize students’ preparation for evidence-based entry into practice. Skills lists can and should be representative of the population foci served. These areas represent direct evidence of students’ documentations in ECTS to objectively categorize and document specific data points.

Aspects of students’ clinical performances involve validation by preceptors or faculty. Indirect evidence of meeting core competencies requires subjective appraisals of students’ clinical decision-making rationales as accurate and appropriate in documenting objective information. Clinical reasoning, use of diagnostic tools and referrals, accuracy of physical assessments and findings, differential diagnoses, and prescribing of medications and therapeutic modalities require supervisory expertise in guiding and evaluating students’ evidence-based practice. Preceptors and faculty play an integral role in addressing cultural sensitivities and collaborations for students’ skills performed and evaluated.

Using a formative approach, the authors found that ECTS meets some leadership and most independent practice core competencies through students’ clinical documentations in ECTS in a midsouthern FNP program. Core competency areas that were not met through students’ documentations in ECTS encompassed participation in leadership activities, which displayed attributes of professional nursing leaders, syntheses of knowledge, and management and monitoring of clinical outcomes for individuals and families. Students can observe and learn from preceptors’ leadership qualities, as well as management and monitoring of individuals’ and families’ clinical outcomes, but there is no ECTS objective documentation of this longitudinal relationship established in clinical settings. Students’ syntheses of knowledge are best explored by mechanisms outside ECTS.

There are three limitations of this study. The first is that only one ECTS (Medatrax) was evaluated, and results are not generalizable to other ECTSs. The next limitation is that only one midsouthern FNP program’s use of a customized ECTS was evaluated to meet core competencies. Therefore, this customized ECTS may not be generalizable to other programs. The final limitation is that a formative program evaluation, by definition, is an evaluation of a program in process to determine a program outcome. Therefore, the formative program evaluation is an evolving process, which is not static.


The use of a formative program evaluation approach supports the use of students’ clinical documentations in an ECTS in meeting some leadership and most independent practice population-focused core competencies. This research provides evidence that ECTS documentation has a valid and effective role in FNP programs in meeting most entry into practice competencies. The significance of this study is that it provides novel evidence to support the use of an ECTS to assist a midsouthern FNP program in meeting national core competencies. Future program evaluation research is necessary to evaluate continued support of ECTS in education as technology evolves. Further studies should include aggregate data from multiple graduate nursing programs across geographic regions, which would add to the evidence in supporting generalizability of ECTS to meet core competencies.



1. Baker AL, Dubose TJ. Electronic systems for student clinical records. J Diagn Med Sonogr. 2009;25(5): 277–281.
2. Hatfield AJ, Bangert MP. Implementation of the clinical encounters tracking system at the Indiana university school of medicine. Med Ref Serv Q. 2005;24(4): 41–58.
3. Kowlowitz V, Slatt LM, Kollisch DO, Strayhorn G. Monitoring students’ clinical experiences during a third-year family medicine clerkship. Acad Med. 1996;71(4): 387–389.
4. Olivieri R. Clinical recordkeeping Using a personal computer. Part 1. Preparation. Nurse Educ. 1984;9(1): 43–48.
5. Johnson VK, Michener JL. Tracking medical students’ clinical experiences with a computerized medical records system. Fam Med. 1994;26(7): 425–427.
6. Rowe BH, Ryan DT, Mulloy JV. Evaluation of a computer tracking program for resident-patient encounters. Can Fam Physician. 1995;41: 2113–2120.
7. Kuehn AF, Hardin LE. Development of a computerized database for evaluation of nurse practitioner student clinical experiences in primary health care. Report of three pilot studies. Comput Nurs. 1999;17(1): 16–26.
8. Branstetter ML, Smith LS, Brooks AF. Evidence-based use of electronic clinical tracking systems in advanced practice registered nurse education: an integrative review. Comput Inform Nurs. 2014;32(7): 312–319; quiz 320–311.
9. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.
10. US Department of Health and Human Services, Health Resources and Services Administration. Nurse Practitioner Primary Care Competencies in Specialty Areas: Adult, Family, Gerontological, Pediatric, and Women’s Health. Rockville, MD: Bureau of Health Professions Division in Nursing; 2002.
11. NONPF. Population-Focused Nurse Practitioner Competencies. 2013. Accessed May 15, 2015.
12. Medatrax. Medatrax: clinical data tools for health professional education. 2015. Accessed May 30, 2015.
13. Funnell SC, Rogers PJ. Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. San Francisco, CA: Jossey-Bass; 2011.
14. Joy L, Berner B, Tarrant D. Evaluating the implementation of an online clinical log system for family nurse practitioner students. Comput Inform Nurs. 2012;30(1): 29–36.
15. Longworth JC, Lesh D. Development of a student nurse practitioner computerized clinical log. J Am Acad Nurse Pract. 2000;12(4): 117–122.
16. Cullen D, Stiffler D, Settles J, Pesut DJ. A database for nurse practitioner clinical education. Comput Inform Nurs. 2010;28(1): 20–29; quiz 30–21.
17. Walker DS, Pohl JM. Web-based data collection in midwifery clinical education. J Midwifery Womens Health. 2003;48(6): 437–443.
18. Fontana SA, Kelber ST, Devine EC. A computerized system for tracking practice and prescriptive patterns of family nurse practitioner students. Clin Excell Nurse Pract. 2001;5(2): 68–72.
19. National Organization of Nurse Practitioner Faculties. Nurse Practitioner Core Competencies. 2012. Accessed May 15, 2015.
20. NONPF. NP Core Competencies with Curriculum Content. 2014. Accessed May 15, 2015.
21. American Nurses Association. Standards of Professional Nursing Practice. Nursing: Scope and Standards of Practice. 2nd ed. Silver Spring, MD: American Nurses Association; 2010: 31–62.
22. Ball JW, Dains JE, Flynn JA, Solomon BS, Stewart RW. Recording information. In: Seidel’s Guide to Physical Examination. 8th ed. St Louis, MO: Elsevier Inc; 2015: 616–631.

Advanced practice RN; Core competencies; Electronic clinical tracking systems; Formative program evaluation; Nursing education

Copyright © 2016 Wolters Kluwer Health, Inc. All rights reserved.