THE VALIDITY OF USING A HANDOVER OBJECTIVE STRUCTURED CLINICAL EXAMINATION TO INFORM ENTRUSTMENT AND COMPETENCY DECISIONS ABOUT PATIENT HANDOFFS
Joseph Lopreiato, MD, MPH1, Kathleen Wortmann, BS1, Amy Flanagan, BA, MFA1, Andrea Creel, BA, MSW1, Daniel West, MD2,
1UNIFORMED SERVICES UNIVERSITY OF THE HEALTH SCIENCES, BETHESDA MD, USA
2UNIVERSITY OF CALIFORNIA, SAN FRANCISCO CA, USA
Introduction: Ensuring trainees can effectively handover patients to other providers has implications for patient safety and is required by the ACGME. However, methods to assess handover skills are needed.
Objective: To test the validity of a simulated Handover Objective Structured Clinical Examination (Handover OSCE) to assess resident handover skills.
Methods: In 2013 and 2014, pediatric residents at the end of their first year of training from 2 different pediatric residency programs participated in a 5-patient OSCE. In the final station residents handed over 3 standardized patients they had seen during the OSCE to a faculty rater playing the role of a receiving resident. In 2013, 33 residents from one program received no formal handover training. Residents in the other program in 2013 and residents in both programs in 2014 (N = 57) received training using the IPASS handover method. After training to use the assessment tool, a faculty rater scored each resident’s performance in the handover station using a 10-item assessment tool aligned with the I-PASS handover curriculum. All items were rated on a 5-point scale indicating the frequency (1 = never; 2 = rarely; 3 = sometimes/fairly often; 4 = usually/very often; 5 = always/very often) that the desired skill/behavior was observed. To generate validity evidence, we assessed reliability and tested whether the handoff OSCE could measure the higher level of performance expected after training.
Results: Variance proportions from a Generalizability study are shown in the Table. Handover OSCE scores demonstrated high reliability to discriminate between low and high performing residents (G-coefficient 0.80) and a very good ability to make criterion-based decisions about an individual’s handover skills (phi = 0.72). The handover OSCE scores for residents who received handover training were significantly higher than residents who did not (P = 0.001, T-test).
Conclusions: A handover OSCE scored using a 10-item assessment was highly reliable for both relative and criterion-based decisions and was able to measure the expected difference in the performance after handover training. These findings provide strong validity evidence and support the use of the handover OSCE to make high stakes decisions about competency and entrustment and to meet ACGME requirements.
STANDARDIZED PATIENTS’ UNSCRIPTED DIALOGUE AND BEHAVIORS DURING A THIRD YEAR MEDICAL STUDENT OB/GYN STANDARDIZED PATIENT EXAMINATION
Amy Thompson, MD, Maureen Asebrook,
UNIVERSITY OF CINCINNATI COLLEGE OF MEDICINE
Introduction: Standardized patient (SP) examinations allow standardization of patient encounters for formative or summative assessment. In spite of SP training, student interactions cannot be entirely scripted and the SP must portray the patient character and reveal the information in a standard way to the student in spite of unpredictable variation between students. Reliability of the SP examination has been reported using quantitative methods. This study used qualitative methods to describe, categorize and quantify unscripted SP dialogue and behaviors.
Methods: Thirty videos were reviewed from two stations of the third year, OB/Gyn clerkship, summative OSCE. Encounter videos from five standardized patients were viewed and video observation notes collected. During video review, attention was given to behaviors not prescribed in SP training, hence review was completed by the faculty member who wrote the OSCE material and assisted with SP training. Notes were used to generate a coding scheme of unscripted behaviors. Video review continued for each SP until no new codes were generated for an individual SP. Sixteen videos of eight standardized patients in a second OSCE station were reviewed, and video observation notes collected and coded for occurrence and frequency of behavior codes. No new codes were generated from video reviews of the second station.
Results: Both verbal and non-verbal behaviors were observed. Most verbal behaviors represented content that was spontaneously generated by the standardized patient. While all standardized patients exhibited unscripted dialogue and/or behaviors, the frequency and type varied among standardized patients. Most standardized patients demonstrated a correct response to atypical questions from students and asked unscripted follow-up questions to student statements. Behaviors that represented case content error did not correlate with standardized patient experience
Conclusion: Qualitative research methods were used to create a descriptive coding scheme of unscripted standardized patient behaviors during a summative assessment. This coding scheme can guide standardized patient training and evaluate standardized patient performance. Although errors in case portrayal were observed, most unscripted dialogue and behaviors had a positive impact, by adding realism to the encounters. The unique skills of some standardized patients, such as well-timed follow up questions to student statements, can be helpful for standardized patient training.
Implementing A Longitudinal, Inpatient, Multiday OSCE
Marta M. Brewer, BS, MBA, Tom C. Weber, BS, Christopher J. Mooney, MA, MPH, Sarah Peyre, EdD,
UNIVERSITY OF ROCHESTER: SCHOOL OF MEDICINE AND DENTISTRY, ROCHESTER NY, USA
Introduction: A Comprehensive Assessment (CA) for 3rd-year medical students has long been used in our institution to formatively assess students’ knowledge, skills and behaviors, to foster reflection and direct future learning.1 SP encounters were largely ambulatory-based, incongruent to the third year clinical environment. Developing an OSCE that replicated stages of care typical to inpatient experiences provided opportunities to establish patient/family relationships and incorporated elements of collaborative and patient and family centered care, reflecting both institutional and national care models.1,2
Project Description: 104 medical students participated in a longitudinal OSCE that simulated an evolving inpatient scenario. Day 1 and 2, students worked individually; days 3 and 4 they worked in pairs seeing the same SP and three different “family members”. Day 1 involved an altered mental state, post-operative patient. Days 2 and 3 required administering informed consent and delivering bad news. Day 4, discharge, included 5 participants: two medical students, a nursing student, an SP patient, and SP family member. The first three days concluded with expert role-modeling in large group debriefings. Day 4 concluded with a structured small group, faculty-led debriefing session. All were videotaped.
Outcomes: We successfully implemented an OSCE aligned with students’ clinical experiences. Student and SP evaluations were positive; many praising the interprofessional component. The AAMC Graduation Questionnaire indicated over 96% of our students participated in a required curricular activity with other health professional students, which far exceeds national standards.
Conclusions/Discussion: More longitudinal experiences incorporating simulated patients, family members as well as other health professional students are needed to mirror real practice and increase the realism of OSCEs.
- Epstein RM, Dannefer EF, Nofziger AC, Hansen JT, Schultz SH, Jospe N, Connard LW, Meldrum SC, Henson LC: Comprehensive assessment of professional competence: the Rochester experiment. Teaching and Learning in Medicine. (2004);16(2):186–196.
- Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Vol. 6. Washington, DC: National Academy Press; 2001.
NEW METHODOLOGY FOR USING INCOGNITO STANDARDISED PATIENTS FOR TELEPHONE CONSULTATION IN PRIMARY CARE
Hay Derkx, MD, PhD,
MAASTRICHT UNIVERSITY, MAASTRICHT, THE NETHERLANDS
Introduction: Many countries are now using call centres as an integral part of out-of-hours primary care. While there has been research on safety issues within telephone consultations, there has been no published research on how to train and/or use standardised patients calling for medical advice and on the accuracy of role-playing. Objectives: To assess the feasibility and validity of using Telephone Incognito Standardised Patients (TISPs), and the accuracy of role plays and the rate of detection. To explore the experiences of being a TISP and the difficulties encountered with self-recording the calls.
Methods: Twelve TISPs were trained in role-play by presenting their problem to a general practitioner and a nurse and to self-record calls. Calls were made to 17 different out-of-hours centres from home. Of the four or five calls per evening, one call was assessed for accuracy of role play. Retrospectively, the out-of-hours centres were asked whether they had detected any calls made by a TISP. The TISPs filled in a questionnaire concerning their training, the self-recording technique and personal experiences.
Results: The TISPs made 375 calls during 84 evenings. The accuracy of the roleplay was close to 100%. A TISP was called back the same evening for additional information in 11 cases. Self-recording caused extra tension for some TISPs. All fictitious calls remained undetected.
Conclusion: Using the method described, TISPs can be valuable both for training and assessment of performance in telephone consultation carried out by doctors, trainees, and other personnel involved in medical services. NB: This research was part of a research project into the quality of telephone consultations at out-of- hours centres in the Netherlands.
THE “TELEOSCE:” USING STANDARDIZED PATIENTS IN REMOTE TELEMEDICINE ASSESSMENTS
Andrew Fredette, BA1, Tru Chatelain, BA1, Ryan Palmer, EdD2,
1DEPARTMENT OF SIMULATION, OREGON HEALTH AND SCIENCE UNIVERSITY, PORTLAND OR, USA
2DEPARTMENT OF FAMILY MEDICINE, OREGON HEALTH AND SCIENCE UNIVERSITY, PORTLAND OR, USA
Introduction: Distance learning is a large part of many medical school curricula, however assessing distance students can be challenging. Similarly, there are few opportunities to assess local students in their ability to conduct remote clinical examinations. In order to meet these challenges, a Standardized Patient (SP)- based telemedicine case, the “TeleOSCE” was developed and evaluated for its cost-effectiveness and teaching value. This case utilized real-time faculty and SP feedback to assess local and distance learners’ ability in these types of situations.
Project Description: The “TeleOSCE” was developed to simulate a rural telemedicine consultation with a diabetic patient as part of the formative 3rd year Family Medicine Clerkship Objective Structured Clinical Exam (OSCE). The case includes specific elements to assess learners’ ability to use the technology and conduct an appropriate remote exam. SP’s used for the case were specifically trained to use the remote technology and in how to give valuable feedback. A qualitative analysis was done to assess the utility of the “TeleOSCE” from both student and SP perspectives. SP interviews were conducted to analyze trainability and effectiveness.
Outcomes: A cost analysis was conducted to access economic feasibility and a qualitative analysis of learners’ transcripts was conducted to assess case effectiveness. Qualitative analysis revealed that the case was both cost effective and a valuable learning tool for the both rural and local students1. Analysis also suggested that SP’s could effectively be utilized for remote cases in an OSCE setting.
Conclusions/Discussion: Implementing SP-based telemedicine cases is a cost-effective and valuable learning tool that can be used to asses and train distance learners and local learners.1 SP’s are highly effective in these situations, and will help learners adapt to rapidly developing technologies. The cost-effectiveness suggests that similar strategies could easily be employed by other teaching institutions.
- Brisson A.M., Steinmetz P, Oleskevich S, Lewis J, Reid A. 2005. A Comparison of Telemedicine Teaching to In-Person Teaching for the Acquisition of an Ultrasound Skill - a Pilot Project. Journal of Telemedicine and Telecare. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/25766853.
LEARNING FROM STANDARDIZED PATIENT (SP) EXPERIENCE: USING SP METHODOLOGY TO TEACH PATIENTS SHARED DECISION-MAKING
Sarita Kundrod, Joseph Plaksin, BS, Andrew Wallach, MD, Natalie Levy, MD, Andrew Chang, MD, MPH, Sondra Zabar, MD, Lisa Altshuler, Ph.D., Adina Kalet, MD, MPH,
NEW YORK UNIVERSITY SCHOOL OF MEDICINE, NEW YORK NY, USA
Introduction: Standardized Patients (SPs) are a valid, reliable, and frequently used method of assessing the clinical competency of healthcare providers (HCPs). As a result of their experience, SPs often report an increased ability to judge the skills of HCPs and increased collaboration with their HCPs. They have also reported becoming proactive in terms of their own health behaviors. Based on these findings, we have developed a Patient Empowerment Program (PEP) that will utilize SP training methodology to prepare real patients with diabetes to take on a more active role in their medical encounters.
Project Description: The PEP consists of two two-hour sessions. In the first session, participants discuss the roles of the patient and HCP in the medical encounter in terms of shared decision-making (SDM). They then observe video clips of patient-HCP interactions and rate them based on how well they participated in SDM. In the second session, participants role-play several scenarios with Standardized Healthcare Providers (SHPs), rate their performances, and practice giving feedback. They then apply what they have learned to an individualized role-play scenario with a SHP, created specifically for each participant based on self-reported difficulties with diabetes. Finally, participants debrief with the SHP and research personnel to create an action plan for diabetes self-care and future medical encounters.
Outcomes: Planned outcomes include patient activation, as measured by the Patient Activation Measure (PAM), and self-reported measures of patient preferences for control in medical decision-making. These measures will be assessed: 1) at baseline, 2) immediately after the intervention, and 3) three months after the intervention. Hemoglobin A1c will also be collected to determine if a change in attitude translates into a change in disease severity.
Conclusions/Discussion: The methodology used to train SPs can be adapted into a training program for patients to prepare them to be full partners in their care, making the HCP-patient relationship more collaborative and improving the quality of care, thus leading to improved health outcomes. This program also seeks to expand the role of SPs in healthcare education, from training HCPs to addressing both sides of the medical encounter by using SHP interactions to empower patients.
BIOETHICS EDUCATION: BRIDGING THE GAP BETWEEN KNOWLEDGE AND SKILL
Terry Sommer, BFA1, Nada Gligorov, PhD1, Rosamond Rhodes, PhD1, Ellen Tobin, BA1, Robert Baker, PhD2, Lily Frank, PhD1, Sean Philpott, Ph.D., MS2,
1ICAHN SCHOOL OF MEDICINE AT MOUNT SINAI, NEW YORK NY, USA
2UNION GRADUATE COLLEGE, SCHENECTADY NY, USA
Introduction: The Master of Science degree in Bioethics is jointly conferred by a major medical school and a well-established graduate college. This hybrid program consists of both online and onsite courses. Students apply from medicine, law, philosophy, and many other backgrounds. As part of their practical, onsite training in clinical ethics, students engage in competency-based Standardized Patient (SP) encounters at the medical school. These encounters offer the learners an opportunity to simultaneously apply their knowledge of ethical concepts and practice their interpersonal skills.
Project Description: Students completing the Bioethics Program have a total of four SP encounters during their studies, two during the Clinical Ethics Practicum (formative encounters) and two during the Capstone course (summative encounters). Clinical Ethics Practicum and Capstone Logistics: 12–16 learners participate in two 20-minute SP encounters each. Using checklists to evaluate learners’ performances, SPs provide feedback on Interpersonal Communication, and members of the ethics faculty provide feedback on ethics. Sample Practicum Encounter: Ana Petrovska is terminally ill with pancreatic cancer. Citing cultural differences, the family requests the patient not be informed of her prognosis. Her oncologist, Dr. Fisher, meets with the clinical ethicist to determine whether she should disclose the prognosis despite the family’s request. The primary case dilemma involves a conflict between truth-telling and minimizing harm by withholding the bad news from the patient. For all encounters learners must: 1) elicit the relevant concerns from the SP; 2) identify the key ethical principles underlying the client’s dilemma; 3) help the client prioritize competing ethical principles; 4) identify a potential resolution to the dilemma; and 5) formulate a plan to implement the resolution. In addition, the learner must respond respectfully and empathically to client concerns, and remain non-judgmental throughout the encounter.
Outcomes: All students who participated in the 2014 Practicum rated the SP exercise favorably. Some remarked that it was the highlight of the course. 71% of the students expressed that feedback from both faculty and the SP was the most formative aspect of the exercise.
Conclusions/Discussion: Next steps include research on the impact of SP exercises and assessments on the Bioethics students’ ability to successfully engage in ethics consultation.
INTRODUCING HEALTH PROFESSIONAL AND LEGAL STUDENTS TO MEDICAL MALPRACTICE USING SIMULATION
Christopher Woodyard, JD1, Donald Woodyard, BS2,
1CHARLOTTE SCHOOL OF LAW, CHARLOTTE NC, USA
2INDIANA UNIVERSITY, BLOOMINGTON IN, USA
Introduction: Much of the resentment between the legal and medical professions is a result of the tort reform debate, which has pitted doctors versus lawyers in a very public and political battle of eye catching headlines: 54 cents of every dollar that injured patients received were used to pay legal fees; $200B spent annually on defensive medicine because doctors fear lawsuits. However, lawyers and doctors are much more likely to interact on a professional level outside the realm of the dreaded medical malpractice lawsuit. From trying to understand the cause of death of a homicide victim to ridding a neighborhood of environmental health hazards, the opportunity for meaningful collaboration is substantial.
Methods: Medical, nursing, and pharmacy students enrolled in an Inter-Professional Teamwork & Communication Course had an interactive experience with law students centered on medical malpractice. This included a 90-minute didactic session followed by a 3-hour mock medical malpractice trial. The didactic session gave an overview of the judicial system followed by an in depth look at the process of medical malpractice litigation. At its conclusion, students were assigned to roles for a mock trial based on a true medication error case involving complaints against all three disciplines. Defendants and Expert Witnesses were assigned counsel by Law students and required to meet each other outside of class. Law faculty, acting as judge, managed the mock trial that used Standardized Patients as jurors and patients/family. Students later completed a retrospective survey on the experience.
Results: 30 Nursing, Pharmacy, Medical, and Law students participated in the trial.
Surveys of Health Affairs students (N = 24) showed that 100% enjoyed the experience and felt more comfortable working with lawyers. While all students reported fearing litigation, the mean dropped from 4.0 to 3.5.
Conclusion: This data expands on research presented in 20101 and reaffirms those findings. Some participants indicated reduced fear of litigation, but all students reported fear. All students favorably reported more comfort working with lawyers, understanding that lawsuits are a rare occurrence, and recognition that many of the myths surrounding torts are not true. This experience provided an opportunity for law students to practice their skills with healthcare professionals and juries. More collaborative opportunities among these professions are needed.
- Woodyard C, Woodyard D, Scolaro K, Durham C: The mock medical malpractice trial: Introducing health professional students to the legal field using simulation. Simulation in Healthcare. 2010; 5(6):420.