Improving Medical Students’ OSCE Performance in Telehealth: The Effects of a Telephone Medicine Curriculum

Abstract

Purpose 

To report on the implementation of a telephone medicine curriculum as part of the core clerkship in pediatrics for students at Johns Hopkins University School of Medicine and evaluate the curriculum’s effect on student performance on a telephone medicine case as part of a required objective structured clinical exam (OSCE).

Method 

Using a prospective cohort design with a convenience sample of third-year medical students during the 2016–2017 and 2017–2018 academic years, the authors compared the OSCE scores of students assigned to the curriculum with both historical and concurrent control groups of students who had not received the curriculum. Additionally, the authors compared the costs of the recommended testing by students in each group using the 2018 Centers for Medicare and Medicaid Services Clinical Laboratory Fee Schedule.

Results 

Students assigned to the telephone medicine curriculum (students in the intervention group) had a significantly higher mean overall score on the simulated OSCE telephone medicine case compared with the students in the control groups who did not receive the curriculum (the mean score for students in the intervention group was 7.38 vs 6.92 for students in the control groups, P = .02). Additionally, the intervention group had statistically significantly lower costs for their recommended testing compared with the control groups (the median value for tests ordered by students in the intervention group was $27.91 vs $51.23 for students in the control groups, P = .03).

Conclusions 

Implementing a dedicated telephone medicine curriculum for medical students improves their overall performance and delivery of high-value care via telephone medicine as part of an OSCE. Medical educators should pursue ongoing research into effective methods for teaching medical students and residents how to navigate digital encounters.

The triage, assessment, and management of a patient without physical contact with a medical provider is an increasingly common practice.¹ One 1989 study reported that 28.5% of pediatric patient encounters occurred via telephone.² This percentage was higher than those in family practice (18.5%), internal medicine (24.6%), and dermatology (8.9%). Given the ongoing movement of patient encounters to a virtual space, medical students need intentional and rigorous instruction across medical disciplines in the skills necessary to provide safe and high-quality telehealth, a field that encompasses “the delivery of healthcare services where patients and providers are separated by a distance.”³

Health care providers are increasingly delivering care via telehealth in a myriad of settings with the potential for further growth. For example, in one study, electronic patient portal messaging in pediatric primary and subspecialty care grew rapidly in volume in the years following its introduction in 2008.⁴ Given the development of new technologies and opportunities for reimbursement, medical educators and clinicians should anticipate further expansion of telehealth.¹ For health systems, the care provided to patients via telehealth represents a promising pathway to reducing health care system costs, decreasing school absences for children, and decreasing visits to the emergency department.¹ As a result, residency programs and schools of nursing have created curricula for teaching telephone triage, noting improved history taking, appropriateness, and relevance of the curriculum for residents going into primary pediatric practice and, possibly, improved access and satisfaction for patients.^5–8

However, a recent Cochrane review concluded that high-quality studies in the field of telephone consultation training are urgently needed.⁹ Further underscoring the importance of this topic, telephone encounters have long been a component of the United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills.¹⁰ In this study, we discuss how we developed a telephone medicine curriculum (TMC) for medical students completing their core clerkship in pediatrics (CCP) at the Johns Hopkins University School of Medicine (JHUSOM) and evaluated the effect of the curriculum on student performance on a telephone medicine case included in their required objective structured clinical exam (OSCE). Additionally, we compare the associated costs of the recommended tests ordered by students who received the curriculum and those who did not.

Method

We performed a prospective cohort study using a convenience sample of third-year medical students at JHUSOM during the 2016–2017 and 2017–2018 academic years. The study was deemed exempt from review by the JHUSOM institutional review board. Before graduation, medical students at JHUSOM must pass a 12-station OSCE covering content from all of the core disciplines. The OSCE is administered at the end of the third year of medical school and includes a pediatric telephone case depicting an infant with vomiting and diarrhea. We designed and implemented a TMC as part of the CCP. We then assessed how completing at least half of or the entire curriculum (receiving the intervention) affected student performance on the simulated telephone medicine encounter included in the OSCE.

We implemented the TMC in August 2017 as a 1-hour classroom session included in the 2-hour educational block during each quarter of the CCP. Approximately 20 of roughly 25 clerkship students per quarter attended. Students were absent from the session primarily when they were post-call. Students who did not attend the classroom session were still included in the intervention arm for analytic purposes.

Before the session, educators provided students with a sample page from the triage service we designed for the study. The page stated: “13 m/o female. Poor appetite. Cough. Fever. Poor appetite? Please call.” Before the session, students were assigned a textbook reading that provided instructions on how to conduct a telephone interview on a pediatric patient with a cough.¹¹

The classroom session opened with a 10-minute introductory lecture on telephone medicine followed by 2 simulated patient phone interviews involving a student role-playing a phone call with a parent to take a history and determine appropriate management. A faculty member assumed the role of the parent. The remaining students observed the interview and could be called on to provide input to their colleague. Both cases involved the same chief concern of cough, but there was variability in the severity of illness, which required different management plans. In the first scenario, the patient had an upper respiratory infection that could be managed at home. In the second scenario, the patient had pneumonia with impending respiratory failure that necessitated that the parent take the child to the emergency department. After the session, students submitted telephone notes for each patient and faculty provided written feedback to the students on the notes.

The telephone medicine case was included in the OSCE before we developed the TMC, allowing us to assess the curriculum’s effect on student performance before and after the TMC’s implementation. We selected as historical controls all students who had completed the OSCE in 2017. As a concurrent control, we evaluated the performance of students who took the OSCE in 2018 but completed their CCP before implementation of the TMC. Since not all students had completed their CCP when they took the OSCE (based on the structure of our school’s curriculum), we conducted a subanalysis to assess for confounding based on CCP completion. In this subanalysis, we compared performance on the OSCE only for those students in each arm of the study who had completed their CCP. For students who took the OSCE during their CCP, we considered them to have completed the CCP if they were more than halfway through the clerkship at the time of the OSCE.

As part of the OSCE, each student completed a postencounter note. We graded notes on a 10-point scale broken into 4 domains: history (4.0 points), diagnoses (3.5 points), diagnostic tests (0.5 point), and management plan (2.0 points). A single faculty member graded all notes. Masking of students’ names did not take place during grading. The note template and grading were modeled on the USMLE Step 2 Clinical Skills. For history, students received 1 point for each of the following: characterizing the emesis, characterizing the diarrhea, assessing the child’s ability to take fluids by mouth, and assessing the child’s hydration status. For the diagnoses, students received 1 point for each diagnosis in the differential that matched a list of acceptable diagnoses (up to a total of 3 points). They received an additional 0.5 points for ranking the correct diagnosis of gastroenteritis highest on the differential. Students received 0 points for ordering tests and 0.5 points for ordering no tests. For management, students earned 1 point for recommending oral rehydration and an additional point for giving callback instructions. We compared the performance of students who received the intervention with the performance of both control groups: the historical cohort (those who completed the OSCE in 2017 before TMC implementation) and the concurrent cohort (those who took the OSCE in 2018 but completed the CCP before TMC implementation).

We used descriptive statistics to characterize the percentage of students in the control and intervention groups by age, gender, ethnicity, race, and completion of the CCP before taking the OSCE. We calculated the mean overall score and mean score for the diagnoses section of the OSCE for the intervention and control groups. We made comparisons using Student’s t test to test for significant differences. We treated the scores for diagnostic tests ordered, management plans, and history as categorical variables and made comparisons using a chi-square analysis to test for significant differences between groups. Given the exploratory nature of the analysis, we did not adjust for multiple comparisons.

To assess the effect of the intervention relative to the quality and value of care in the setting of telephone medicine, we used the 2018 Centers for Medicare and Medicaid Services Clinical Laboratory Fee Schedule to determine the costs that would be incurred for the recommended testing by the students in each study arm.¹² We calculated the mean and median costs by study arm as well as the mean differences. We then compared study arms using a Kruskal–Wallis test, as the data were not distributed normally. All statistical analyses were performed using SAS 9.4 (SAS Institute, Cary, North Carolina).

Results

This study included 245 students and 67 students received the intervention. There were 48 students in the concurrent control group and 130 students in the historical control group. The distribution of students in the study arms as well as student characteristics are presented in Table 1. Students who received the TMC had a significantly higher mean overall score on the simulated OSCE telephone medicine case compared with the students in the control groups who did not receive the curriculum (the mean score for the intervention group was 7.38 vs 6.92 for the control groups, P = .02). Students in the intervention group also scored significantly higher on the history and diagnostic portions of the tests, as documented in their notes. Table 2 presents these and additional student results on the OSCE.

Table 1: Participant Characteristics in a Study on the Effects of a Telephone Medicine Curriculum During Third-Year Medical Students’ Core Clerkship in Pediatrics, Johns Hopkins University School of Medicine, 2016–2018

Table 2: Student Scores on the Telephone Medicine Case Section of a Required Objective Structured Clinical Exam, Johns Hopkins University School of Medicine, 2016–2018

To account for the differences between study arms regarding completion of the CCP (see Table 2), we performed a subanalysis using only those students who had either completed or were past the halfway point in their CCP. This subanalysis demonstrated similar findings to the overall analysis presented in Table 2 regarding the mean overall score and mean score for diagnoses for the intervention and control groups. Additionally, the percentages for the categorical variables (diagnostic tests, management plans, and history) were similar. In the subanalysis, the P value for the difference in the mean overall score between the intervention and combined control groups was greater than .05 (the mean overall score for the intervention group was 7.41, while the mean overall score for the combined control groups was 6.98, P = .30)—a change from the overall analysis where the P value was .02. However, findings of statistical significance for the remaining variables remained consistent with the overall findings between the intervention and combined control groups.

We determined the mean and median costs for the tests ordered by students (see Table 3) during the OSCE. Students who participated in the TMC averaged $61.93 for their recommended tests with a median of $27.91. These values are less than those of both control arms, which, when combined, were an average of $84.68 with a median of $51.23 (P = .03). Using a Kruskal–Wallis test to account for the lack of a normal distribution in the data on the costs of the recommended tests, we found that the intervention group had statistically significantly lower charges compared with the concurrent control group and the combined control group.

Table 3: Costs for Tests^a Ordered by Students Participating in the Telephone Medicine Case Section of a Required Objective Structured Clinical Exam, Johns Hopkins University School of Medicine, 2016–2018

Discussion

The results demonstrate the potential of dedicated medical student instruction in telephone medicine for improving the quality and value of medical care provided in the virtual environment, a setting that has received added attention in the context of the COVID-19 pandemic. Our study has 2 primary findings. First, the TMC improved the quality of care provided by participants in terms of their history taking and recommended diagnostic testing. This is evident in the higher scores on these sections of the OSCE for the intervention group. Second, the TMC improved the value of care provided by participants in that they ordered fewer tests that were not indicated. While our study design raises concerns about confounding based on whether a student completed the CCP before taking the OSCE, the subanalysis affirms that the intervention had similar effects on performance whether or not the student completed the clerkship before taking the OSCE.

Our findings are consistent with other studies on the positive effect of TMCs on medical student performance in simulated environments.¹³ Central to our curriculum—and the curricula of other interventions aimed at improving medical student performance in the telehealth environment—is the assumption that effective communication and medical management when the patient is not physically present are unique skill sets. A corollary to this assumption is that developing these skill sets through dedicated instruction has the potential to enhance patient care. Our findings contribute to the validity of these assumptions and, along with the currently developing literature on the subject,^13,14 serve to emphasize the continued need for the development and evaluation of focused telehealth curricula.

Notably, our study is limited by its use of a simulated clinical environment at a single medical school and by its use of a convenience sample. We cannot say with certainty that our curriculum’s effect would be the same in an actual clinical environment or across institutions. Further research on telehealth education necessitates continued studies using randomized, masked methodologies and set in the clinical environment to better determine the effect of dedicated curricula. We do not know whether the effect we observed is applicable beyond the medical school setting. Students transitioning to residency may require reinforcement in the practice of high-quality telephone medicine. Our institution, as well as others, has a program designed to assist senior medical students transitioning to residency that could serve as a setting for such reinforcement.

Medical educators must continue to adapt to the growth in technologies and use of smart phones and mobile health applications and their effect on patient expectations and provider services. Given the sustained technological advances and the developing reimbursement infrastructure for virtual encounters,¹⁵ we anticipate an increasing need to instruct students in the practice of safe and high-quality telehealth. As such, medical educators should pursue ongoing research into effective methods for teaching medical students and residents how to navigate digital encounters. By actively understanding and teaching the processes and nuances of virtual patient encounters, educators will serve not only the needs of medical students but also the countless patients whom current and future physicians will treat.