Journal Logo

Concise Definitive Review

A Guide to Performance Evaluation for the Intensivist: Ongoing Professional Practice Evaluation and Focused Professional Practice Evaluation in the ICU

Sterling, Michael MD, FCCM1,2; Gregg, Sara MHA, FCCM1; Bakshi, Vishal PA-C, FCCM1; Moll, Vanessa MD, PhD, FCCM1,3,4

Author Information
doi: 10.1097/CCM.0000000000004441
  • Free
  • Editor's Choice

Abstract

Creating a successful quality and patient safety program requires a multifaceted approach that systematically reviews overall systems and processes, and creates a standardized framework for evaluating individual provider performance on an ongoing basis.

The Joint Commission (TJC) is the largest accrediting body in healthcare (1). TJC accreditation emphasizes improved quality of care, decreased liability, a pathway to government regulatory compliance, and value as benefits of its accreditation process. Ongoing Professional Practice Evaluation (OPPE) and Focused Professional Practice Evaluation (FPPE) are mandated by TJC and form an important cornerstone for ensuring adequate provider performance and knowledge base (2).

The intent is to ensure that privileging and credentialing decisions are informed by a regular assessment, establishing that medical staff meets the quality of care standards. In 2008, TJC implemented a new standard mandating a detailed evaluation of provider performance. The OPPE was designed to provide ongoing performance evaluation as opposed to periodic evaluation. The FPPE was designed to evaluate the performance of providers new to the medical staff or providers who are requesting new privileges. FPPE and OPPE are required for any practitioners who are granted privileges at a hospital that is accredited by TJC. However, the details remain obscure to many frontline staff, and there remains a substantial opportunity for education related to the role and importance that these two processes play (3,4).

To our knowledge, no prior publications address the implementation of OPPE/FPPE in critical care medicine and it is likely the implementation of these processes is highly variable across this specialty. This concise review aims to fill the knowledge gap about the OPPE/FPPE process in critical care, references the practices in other specialties, and describes how the OPPE/FPPE process has been implemented at our institution.

ONGOING PROFESSIONAL PRACTICE EVALUATION

The OPPE must cover every provider, with elements of performance (EP) that are clearly defined in advance, and the data gathered should inform decisions regarding privileges within that healthcare setting (5).

The specific assessment and timing are left to the organization to determine. In order to meet the threshold for “ongoing,” OPPEs must occur more than one time in a yearly cycle. The frequency of evaluation has to balance resources with effectiveness. Although 3 months might seem like an ideal interval, the resources consumed to perform an ongoing 3-month evaluation may not be sustainable for many organizations. Hence, the ideal balance between effectiveness and sustainability is based on the institution’s available resources. Medical staff must also collect physician-specific data regarding Accreditation Council for Graduate Medical Education (ACGME) six core competencies (patient care, medical/clinical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice). OPPE is specifically oriented to evaluate the care provided to identify providers who are not practicing to an acceptable standard, and although it may reflect and overlap with the core competencies, they are separate requirements. Recommended options for OPPE include chart review, direct observation, monitoring of techniques, patient satisfaction, or peer and administrative feedback (6).

The information gathered during this OPPE process is used as a factor in deciding whether to maintain, revise, or revoke existing practitioner privileges before or at the end of the required license and privilege renewal cycle.

If an OPPE discovers a potentially actionable issue, a FPPE can be initiated. The intention of the FPPE is to better understand the magnitude of the problem and follow the effectiveness of any corrective action over a defined period of time.

TJC defines objectives within each standard called EP. To meet the standard of OPPE (MS.08.01.03) (7), three EPs must be met:

  • 1) There is a clearly defined process in place that facilitates the evaluation of each practitioner’s professional practice.
  • 2) The type of data to be collected is determined by individual departments and approved by the organized medical staff.
  • 3) Information resulting from the OPPE is used to determine whether to continue, limit, or revoke any existing privilege(s).

All three EPs must be delineated in the organization’s formal OPPE policy.

FOCUSED PROFESSIONAL PRACTICE EVALUATION

Although OPPE is defined for providers with existing privileges, TJC recognized the need for a professional evaluation of additional provider privilege categories (8). This evaluation is termed FPPE. The FPPE standard as defined by TJC requires a period of focused review for the following:

  • 1) Initially requested privileges
  • 2) A provider requesting a new privilege
  • 3) A provider has exceeded a threshold metric from an OPPE, a review, or a complaint

The FPPE standard has nine EPs (FPPE—MS.08.01.01) (7):

  • 1) Applies to all new privileges
  • 2) Defined duration
  • 3) Defined metrics
  • 4) Defined process
  • 5) Defined reviewer
  • 6) Defined threshold values
  • 7) Used to continue, limit, or revoke privileges
  • 8) Applies to all providers
  • 9) Approved by medical staff

Note the differences between OPPE and FPPE. FPPE is required for all initially requested privileges, while OPPE is an ongoing evaluation of the provider’s practice. All newly appointed critical care providers on any hospital medical staff must undergo an FPPE when being considered for initial core privileges, within 6 months after commencing practice, and before granting of additional or special privileges. In addition, an FPPE is required for any critical care provider after a routine OPPE raised questions about professional competency. Such an evaluation is essentially a focused review.

OPPE is defined as a frequency of evaluation, while FPPE is defined as the duration of the evaluation. The time period for conducting the FPPE varies from institution to institution but must be sufficient to assess the adequacy of clinical activity and performance quality (9).

THE RATIONALE FOR INTRODUCING OPPE AND FPPE

Many factors have driven the need for increased organizational accountability, including changes in tort law and the widespread publicity behind the nation’s epidemic of medical errors.

Darling versus Charleston Hospital (1965) was the first case recognizing the right of patients to recover damages under the doctrine of hospital corporate negligence (10). As a result of this case, hospitals could be held accountable for the actions of clinicians that are credentialed by the hospital. This decision became a powerful motivator for hospitals to re-engineer their credentialing processes to ensure that clinicians were competent. The Institute of Medicine’s “To Err is Human: Building a Safer Health System,” and its 2001 follow-ups “Crossing the Quality Chasm: A New Health System for the 21st Century” and “Crossing the Global Quality Chasm: Improving Healthcare Worldwide” suggest that safe, effective, patient-centered, timely, efficient, and equitable care are the keys to reducing errors and improving quality (11–13). It was a logical step forward to incorporate these principles into ongoing professional evaluation. Finally, the ACGMEs adoption of the six core competencies for medical training programs codified the need for competency-based training and by extension, competency-based practice (14).

The EP that make up the OPPE/FPPE standard, address potential quality of care issues by providing powerful tools for healthcare organizations to use to ensure their providers are providing the highest quality of care to their patients and assisting their providers in maintaining that quality of care (15). The OPPE and FPPE processes are inter-related. Although the FPPE is triggered when a practitioner requests new privileges or when a significant adverse event occurs, the OPPE is intended to ensure competency and adequate performance on a continuous basis.

LOOKING ACROSS SPECIALTIES

TJC suggests the implementation of general measures that apply to all medical staff and specialty-specific data that will be agreed upon by clinical departments. Examples of the former include medical record delinquency, dating, and timing entries into the medical record, and “do not use” abbreviations, while examples of specialty-specific data include morbidity and mortality, blood utilization, core measures, and the National Surgical Quality Improvement Program. The only specialty-specific examples noted by TJC that relate to critical care are anesthesiology related examples: reintubations, dental damage (and spinal headache). As the metrics are up to departments to decide, one large academic anesthesia department described the utilization of the electronic medical record to avoid the high resource and time costs associated with traditional methods of clinical performance evaluation. Metrics chosen were a preinduction blood pressure measurement, recorded end-tidal co2 monitoring and timely documentation of compliance statements that make a record billable no more than 120 minutes after the end of anesthesia care (16). Recently Walker et al (17) described current approaches in emergency medicine and identified consensus-based best practice recommendations.

Peer review cases with an opportunity for improvement, Centers for Medicare and Medicaid Services metrics, and timely chart completion were metrics assessed by the modified Delphi method addressing self-identified leaders in emergency medicine.

In the field of radiology, the RADPEER process (sponsored by the American College of Radiology, that allows peer review to be performed during routine image interpretation) involves peer review of a prior study during routine interpretation of current images. The data from RADPEER are submitted to the American College of Radiology, which creates online reports available for peer review and ongoing practice evaluation (18). Similarly, in the field of interventional cardiology, ongoing and focused evaluations of providers in a procedure rich environment can help inform the process of quality of care by interventional cardiologists for the purposes of credentialing and privileging. Cardiology department directors and catheterization laboratory directors are responsible for re-credentialing and re-privileging members of their departments. TJC has taken the position that within an institution, privileging requirements should be the same for all physicians performing that procedure, regardless of their specialty (4).

OPPE and FPPE can also be used successfully in institutions and specialties that are built on interprofessional teams such as critical care medicine. Critical care nurses, advanced practice providers, pharmacists, respiratory care practitioners, rehabilitation specialists, dieticians, social workers, case managers, spiritual care providers, intensivists, and nonintensivist physicians each provide distinct expertise and perspectives to patient care and therefore play an important role in a team that addresses the diverse needs of patients and families in the ICU. When multidisciplinary teams are involved, OPPE/FPPEs are easiest done within the individual professions, with quality indices and metrics developed by each department. Team assessments, although immensely important for patient care and safety in critical care medicine, are currently not a focus of the OPPE/FPPE process. Assessing team performance bears additional unique challenges. The OPPE/FPPE process has been used successfully in alternative ICU staffing models that use advanced practice providers (nurse practitioners, and physician assistants) overseen by a board-certified intensivist to manage critically ill patients.

The OPPE/FPPE process can also be tailored to the critical care model using hospitalists to care for ICU patients, or an electronic ICUs (eICUs) (tele-ICU) to provide support to ICU providers. For hospitalists who provide care in ICUs and for eICU based providers, the OPPE/FPPE process is similar with minor adjustments required in the peer review, and the metrics portion. Direct observation can be carried out by the appropriate medical director, chief of service, or designated provider. At the time of this writing, there are no published guidelines for deploying an OPPE/FPPE process in critical care medicine.

Before implementing an OPPE/FPPE system, a series of questions must be answered relating to all steps of the process, to ensure consistency, to eliminate bias, and to be uniform and fair to all members of the department. It appears easiest to base an OPPE/FPPE system on the ACGME core competencies (Table 1).

TABLE 1. - Examples of Accreditation Council for Graduate Medical Education Core Competency-Based (19) Ongoing Professional Practice Evaluation Metrics for Critical Care
Accreditation Council for Graduate Medical Education Core Competencies Examples of Critical Care Medicine Metrics
Patient care and procedural skills Documentation turnaround timea
Documentation standards and errors (chart review)
Compliance with policies (sepsis bundle, lung protective ventilation, etc.)
Compliance with policies for The Joint Commission’s Universal Protocola
Procedure competencies (can include adequate monitoring and treatment of hemodynamic compromise, safety)
Procedure outcomesa
Incident reports and/or sentinel eventsa
Medical knowledge Maintenance of certification statusb
Fellowship training (subspecialty board certification)b
Continuing medical education requirements metb
Multisource feedback (inter-professional team)
Academic rank
Academic productivity, invited lectures, scholarship
Practice-based learning and improvement Participation in practice quality improvement projects
Participation in committees
Teaching profile
Trainee teaching scores and feedback
Interpersonal and communication skills Data from multisource feedback surveys
Works effectively within an interprofessional team
Compliments and complaints from stakeholders
Documentation is complete and follows national reporting standards
Professionalism Conduct with colleagues, staff, and patients and their families
Data from multisource feedback surveys
Punctuality
Timely billing practicesa
Medical staff, committee, and department meeting attendance and participation
Accountable for personal behavior
Compliance with required training programs
Timely response to administrative requests and surveys
Staff complaints (department, peers)
Complaints from trainees
Systems-based practice Peer review submissions and data
Participation in practice quality improvement efforts
Compliance with the initiatives of The Joint Commission’s National Patient Safety Goals
Participation in multidisciplinary conferences
Compliance with required education testsb
Compliance with required standards (Advanced Cardiac Life Support, etc.)
Feedback from mentor, section, or division chief
a Metrics that can be deducted from the electronic medical record.
b Metrics that are collected by the hospital.

At Emory Healthcare, we have developed a robust model of critical care using advanced practice providers (nurse practitioners and physician assistants) directed by board-certified intensivists. The OPPE/FPPE process works well for this ICU model, with the direct observation portion recently being eliminated in favor of a chart review (minimum of five charts). The peer reviews are provided by a questionnaire in a Likert scale format, and the metrics are similar for both physician and nonphysician providers but do not have to be identical. As long as the EP as defined by TJC are met, organizations can define the process to best meet their organizational needs.

EMORY HEALTHCARE: AN EXAMPLE

At Emory Healthcare, our challenge was to design an OPPE/FPPE process for critical care that met TJC EP but could be adapted to the six hospitals within our system. A workgroup made up of stakeholders from several of the hospitals and with knowledge about, and experience with the processes in each hospital was convened to develop both the OPPE and FPPE processes for our organization. Once the process was elucidated, it was initially tested on the critical care leadership. After making minor adjustments, its use was then expanded to the other providers within the system. As long as the EP are incorporated into the OPPE/FPPE process, the processes can be adjusted for individual hospitals within a larger system. As previously noted, each organization will need to balance its resources against the need to make the process meaningful and sustainable.

We share below our own methodology for using OPPE and FPPE within a single institution of the Emory Healthcare system.

We divided the OPPE evaluation into three parts, including a peer evaluation using a Likert scale (Fig. 1) compliance with predefined metrics and a five chart review. We initially included direct observation by the unit medical director but found the process logistically difficult and abandoned it.

Figure 1.
Figure 1.:
Example of a 270° feedback survey that is part of the Ongoing Professional Practice Evaluation (OPPE) at Emory Critical Care Center. The survey will be sent to (A) 75% of advanced practice providers and attending physicians within one ICU and (B) 3–5 evaluations from other members of the multiprofessional team. NA = not applicable.

The peer review is a 270° evaluation that includes domains about clinical care, communication, behavior, and scholarship. The next portion of our OPPE consists of predefined metrics (Table 2). According to the TJC EP, these metrics are departmentally defined and need to be approved by the medical staff. OPPEs are performed twice per year and are discussed in person with the respective supervisors. In the future, additional metrics including competence in critical care relevant procedures such as central line placements are being added to the FPPE/OPPE process.

TABLE 2. - Examples of Ongoing Professional Practice Evaluation Metrics at Emory Critical Care Center
Metrics Explanations
Finalized notes Notes consist of history and physical, consultations, procedure, and discharge notes. All notes should be finalized within 72 hr of the service date. Our predefined standard is 80%
Billing over past 6 mo Our predefined standard of 8 hr of time documentation a day for attendings or 75% time documentation for affiliates
Charge lag Our predefined standard consists of all bills entered within 1 d
Chart review Review of five charts. EICU physician review charts from all four hospitals within our healthcare system
Ongoing Professional Practice Evaluations are performed twice annually at our institution. These metrics were defined and agreed upon by stakeholders and made transparent to the critical care medicine providers.

We acknowledge that our current OPPE metrics reflect more the business side of patient care rather than patient care quality metrics such as prevalence of ventilator-associated pneumonia, central line-associated bloodstream infections, catheter-associated urinary tract infections, length of stay, etc. These metrics depend on the whole ICU team and are difficult to assign to one physician or provider. This reflects one of the main issues of measuring ICU provider performance: individual attribution. Each episode of care involves multiple intensivists, advanced practice providers, other practitioners, including and consulting teams (20).

We, therefore, decided to monitor these care metrics via a quality dashboard that is shared with the respective ICU teams. For each chosen metric, acceptable parameters need to be established. At our institution, available national, international, and internal averages are incorporated. Annual goals are set by the critical care leadership team.

FUTURE DIRECTIONS

Role of Simulation

Simulation training is rapidly improving. It is likely to serve increasingly large role in initial training, Entrustable Professional Activities (EPA), maintenance of expertise, particularly for low-volume procedures and can serve for OPPE and FPPE processes (21,22). Especially for an interprofessional field such as critical care medicine, simulation might pose a way to evaluate critical team function (23). The challenges of simulation-based OPPE/FPPE evaluation include; prolonged implementation time, time-consuming process in itself, expensive, observer bias, and observer effect.

Automation

The electronic medical record may help inform OPPE criteria selection. The extraction of readily available data/metrics minimizes the administrative burden and reduces the effect of observation and bias in measurements. An automatic, reliable, and cost-effective process can be applied on a continuous, ongoing basis. The extraction from the electronic medical chart can easily be modified, metrics can be replaced or added (16,24).

Utilization of Applications

Mobile Web-based platforms or applications utilizing smartphones might be used to improve feedback validity, timeliness and decrease the administrative burden of the performance evaluation process (24). Applications such as UpToDate offer education and trackable Continuing Medical Education credits (25). Especially in graduate medical education, new applications are at the forefront (26,27). EPA are activities to be entrusted to a trainee once he (or she) has attained sufficient competence and are used in practice (28). EPAs were recently successfully introduced into an intensive care medicine training program in the Netherlands (29).

This framework is also being adopted by the ACGME (19) and has the potential to be used in ongoing or focused performance reviews in the future (30).

CONCLUSIONS

Although OPPE and FPPE are mandated by TJC, they can provide a valuable tool for both ongoing and focused evaluations that do far more to ensure provider competence then the annual or biennial processes of the past. Developing OPPE and FPPE processes can be tailored to individual departments within an organization as well as to institutions within a healthcare system and can be easily adapted to both physician and nonphysician providers. In the near future, simulation and the utilization of applications may play a role in the OPPE/FPPE process and the ability to automate provider data extraction and 270° evaluations will allow OPPEs to occur in real-time, in a true ongoing fashion.

REFERENCES

1. The Joint Commission: Facts About The Joint Commission.Available at: https://www.jointcommission.org/about-us/facts-about-the-joint-commission/. Accessed January 5, 2020
2. The Joint Commission: Joint Commission Accreditation Guide Hospitals.Available at: https://www.jointcommission.org/assets/1/18/171110_Accreditation_Guide_Hospitals_FINAL.pdf. Accessed August 28, 2019
3. Harned RK 2nd. Familiarity with current practices of granting and maintaining privileges in pediatric interventional radiology–a worldwide survey of the members of the Society for Pediatric Interventional Radiology (SPIR). Pediatr Radiol 2012; 42:1316–1321
4. Blankenship JC, Rosenfield K, Jennings HS 3rd. Privileging and credentialing for interventional cardiology procedures. Catheter Cardiovasc Interv 2015; 86:655–663
5. The Joint Commission: Standards Frequently Asked Questions.Available at: https://www.jointcommission.org/standards_information/jcfaq.aspx. Accessed August 28, 2019
6. The Joint Commission: Interpreting Joint Commission Standards: FAQs.Available at: https://www.jointcommission.org/standards/standard-faqs/. Accessed January 5, 2020
7. The Joint Commission: Standards BoosterPak for Focused Professional Practice Evaluation/Ongoing Professional Practice Evaluation (FPPE/OPPE).Available at: https://www.mc.vanderbilt.edu/documents/CAPNAH/files/Forms/Competency%20Evaluation%20Forms/TJC%20Booster%20Pack%20FPPE-OPPE.pdf. Accessed January 14, 2020
8. The Joint Commission: The Joint Commission Medical Staff Role in Performance Improvement.Available at: https://www.jointcommission.org/assets/1/6/MSE17_Sample_PageslTOC_Intro_pp1to7.pdf. Accessed August 28, 2019
9. Hunt JL. Assessing physician competency: An update on the joint commission requirement for ongoing and focused professional practice evaluation. Adv Anat Pathol 2012; 19:388–400
10. Case Brief for Law Students | Casebriefs: Darling v. Charleston Community Memorial Hospital.Available at: https://www.casebriefs.com/blog/law/health-law/health-law-keyed-to-furrow/liability-of-health-care-institutions/darling-v-charleston-community-memorial-hospital/. Accessed August 28, 2019
11. Institute of Medicine (US) Committee on Quality of Health Care in America: To Err is Human: Building a Safer Health System. 2014Washington, DC, National Academies Press (US).
12. Institute of Medicine, Committee on Quality of Health Care in America: Crossing the Quality Chasm: A New Health System for the 21st Century. 2001Washington, DC, National Academies Press (US).
13. National Academies of Sciences, Engineering, and Medicine; Health and Medicine Division; Board on Health Care Services; Board on Global Health; Committee on Improving the Quality of Health Care Globally. Crossing the Global Quality Chasm: Improving Health Care Worldwide. 2018Washington, DC, National Academies Press (US).
14. Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the accreditation council for graduate medical education: A systematic review. Acad Med 2009; 84:301–309
15. The Joint Commission: Standards Interpretation.Available at: https://www.jointcommission.org/standards_information/jcfaq.aspx. Accessed August 28, 2019
16. Ehrenfeld JM, Henneman JP, Peterfreund RA, et al. Ongoing Professional Performance Evaluation (OPPE) using automatically captured electronic anesthesia data. Jt Comm J Qual Patient Saf 2012; 38:73–80
17. Walker LE, Phelan MP, Bitner M, et al. Ongoing and focused provider performance evaluations in emergency medicine: Current practices and modified Delphi to guide future practice. Am J Med Qual 2019 Sep 13. [online ahead of print]
18. Steele JR. The role of RADPEERTM in the joint commission ongoing practice performance evaluation. J Am Coll Radiol 2011; 8:6–7
19. NEJM Knowledge+ Team: Exploring the ACGME Core Competencies (Part 1 of 7) - NEJM Knowledge+. 2016Available at: https://knowledgeplus.nejm.org/blog/exploring-acgme-core-competencies/. Accessed January 12, 2020
20. Oropello JM, Kvetan V, Pastores SM. Lange Critical Care. 2016New York City, NY, McGraw Hill Professional.
21. Gorrindo T, Goldfarb E, Birnbaum RJ, et al. Simulation-based ongoing professional practice evaluation in psychiatry: A novel tool for performance assessment. Jt Comm J Qual Patient Saf 2013; 39:319–323
22. Croft H, Gilligan C, Rasiah R, et al. Development and inclusion of an entrustable professional activity (EPA) scale in a simulation-based medicine dispensing assessment. Curr Pharm Teach Learn 2019; 12:203–212
23. Houzé-Cerfon CH, Boet S, Marhar F, et al. [Simulation-based interprofessional education for critical care teams: Concept, implementation and assessment]. Presse Med 2019; 48:780–787
24. Kruskal JB, Eisenberg RL, Ahmed M, et al. Ongoing professional practice evaluation of radiologists: Strategies and tools for simplifying a complex process. Radiographics 2018; 38:1593–1608
25. UpToDate: Smarter Decisions. Better Care.2013Available at: https://www.uptodate.com/home. Accessed March 25, 2020
26. Connolly A, Goepfert A, Blanchard A, et al. myTIPreport and training for independent practice: A tool for real-time workplace feedback for milestones and procedural skills. J Grad Med Educ 2018; 10:70–77
27. Chen F, Arora H, Zvara DA, et al. Anesthesia myTIPreport: A web-based tool for real-time evaluation of accreditation council for graduate medical education’s milestone competencies and clinical feedback to residents. A A Pract 2019; 12:412–415
28. Wölfel T, Beltermann E, Lottspeich C, et al. Medical ward round competence in internal medicine - an interview study towards an interprofessional development of an Entrustable Professional Activity (EPA). BMC Med Educ 2016; 16:174
29. van Bockel EAP, Walstock PA, van Mook WNKA, et al. Entrustable professional activities (EPAs) for postgraduate competency based intensive care medicine training in the Netherlands: The next step towards excellence in intensive care medicine training. J Crit Care 2019; 54:261–267
30. Wagner JP, Lewis CE, Tillou A, et al. Use of entrustable professional activities in the assessment of surgical resident competency. JAMA Surg 2018; 153:335–343
Keywords:

critical care medicine; Focused Professional Practice Evaluation; healthcare quality; Ongoing Professional Practice Evaluation; safety; The Joint Commission

Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.