Skip Navigation LinksHome > November 2010 - Volume 85 - Issue 11 > Evaluation of Medical Student Performance on Objective Struc...
Academic Medicine:
doi: 10.1097/ACM.0b013e3181f849dc
Performance on OSCEs

Evaluation of Medical Student Performance on Objective Structured Clinical Exams With Standardized Patients With and Without Disabilities

Brown, Rachel S. MD; Graham, Catherine Leigh MEBME; Richeson, Nancy MD; Wu, Junlong; McDermott, Suzanne PhD

Free Access
Article Outline
Collapse Box

Author Information

Dr. Brown is associate professor, Department of Family and Preventive Medicine, University of South Carolina School of Medicine, Columbia, South Carolina.

Ms. Graham is a rehabilitation engineer, Department of Family and Preventive Medicine, University of South Carolina School of Medicine, Columbia, South Carolina.

Dr. Richeson is assistant dean for clinical assessment, University of South Carolina School of Medicine, Columbia, South Carolina.

Mr. Wu is research assistant, Department of Epidemiology and Biostatistics, Arnold School of Public Health, University of South Carolina, Columbia, South Carolina.

Dr. McDermott is director of the research division, Department of Family and Preventive Medicine, University of South Carolina School of Medicine, Columbia, South Carolina.

Correspondence should be addressed to Dr. Brown, Family Medicine Center, 3209 Colonial Drive, Columbia, SC 29203; telephone: (803) 434-6113; fax: (803) 434-6265; e-mail: rachel.brown@uscmed.sc.edu.

First published online September 28, 2010

Collapse Box

Abstract

Purpose: To investigate whether medical students' performance on a family medicine clerkship objective structured clinical exam (OSCE) differed when the standardized patient (SP) had a disability versus when the SP did not have a disability.

Method: SPs with spinal cord injury (SP-SCI), SPs with intellectual disability (SP-ID), and SPs without a disability participated separately in two OSCE scenarios that were administered by the University of South Carolina School of Medicine's Department of Family and Preventive Medicine from 2007 to 2009. OSCE scores were determined based on the number of critical actions completed by the student, and scores were analyzed to determine differences among scenarios.

Results: Students scored lower in history, physical exam, lab tests, and interpersonal skills with an SP-SCI, and lower in history, physical exam, and lab tests with an SP-ID than did students interacting with SPs without a disability. The odds ratio for ordering a hemoglobin A1c in one scenario was 4.16 times higher in cases when the SP did not have a disability (95% confidence interval [CI] 1.78–9.17, P = .001). In the second scenario, the odds ratio was 3.08 times higher for ordering a urinalysis (95% CI 1.34–7.08, P = .006) and was 2.15 times higher for providing lifestyle counseling (95% CI 1.04–4.44, P = .038) in students interacting with SPs without a disability.

Conclusions: Students performed better when the SP did not have a disability. This suggests that greater emphasis should be placed on teaching appropriate care of patients with a disability.

Because patients with physical and intellectual disabilities often do not receive adequate health services,1–3 it is important that medical students receive training in health care provision for people with disabilities (PWDs). The barriers to accessing health care for PWDs are numerous and include incidents of physicians lacking understanding and comfort.1–7 Training to overcome discomfort in communicating with, positioning, and examining PWDs, as well as developing relevant and appropriate management plans for caring for them, begins in medical school, and several institutions are introducing curricular components that educate students in providing care for PWDs.8–12

In 2006 the University of South Carolina School of Medicine (USC SOM) Department of Family and Preventive Medicine introduced a disability curriculum based around didactic sessions in the second and third years of medical school. These sessions covered information about disability prevalence, disability culture, and special considerations for patients with mobility, sensory, and intellectual disability (ID). The lectures, given by a faculty member who has an adult child with ID and a staff rehabilitation engineer who has a spinal cord injury (SCI), stressed that physicians must be able to quickly assess the relevance of a person's disability as it relates to the chief medical issue while understanding that special issues related to each disability must be integrated into the encounter.

Objective evaluation of a student's ability to apply these skills to future clinical encounters with PWDs is essential. Objective structured clinical exams (OSCEs) are effective ways to assess a student's medical knowledge, patient care skills, ability to communicate and develop strong rapport, and understanding of systems-based care.13–16 Whereas standardized patients (SPs) used in OSCEs typically portray patients with common medical problems or psychosocial issues, PWDs are often underused as SPs in these scenarios and are more often used in training workshops or as part of a curriculum.17–24 Several overseas medical programs, as well as some dental programs, have developed comprehensive educational strategies to teach caring for PWDs and have included using PWDs as SPs in final examinations.23,24 In cases when PWDs are used as SPs in final OSCEs, the scenarios often focus on the disability and its complications.

The use of OSCEs to evaluate medical student proficiency is a long-standing tradition at USC SOM, and all third-year clinical clerkships are required to include an OSCE as part of a student's summative assessment for the clerkship. SPs without disabilities are primarily volunteers who are recruited from the community and trained by a faculty member and her staff who have been operating the SP program for over a decade. Initial training consists of a half-day session designed to teach SPs how to portray their scenarios, to observe and record student performance, and to provide feedback to the student. Annual required follow-up training focuses on any scenario modifications and any problems that are identified by the external scorers and clerkship directors.

The preparation of SPs with disabilities is a similar process to the one for SPs without disabilities, but it also involves targeted one-on-one orientation and additional training including mock sessions with resident physicians and videotaping. Feedback and debriefing continues for all the SPs with a disability throughout each year.

Our use of SPs with disabilities in OSCEs is different in several significant ways from traditional scenarios involving SPs with disabilities. We chose to use OSCE scenarios that portrayed a PWD visiting the doctor's office with a common medical complaint or disease. In this way, we were assessing our students' ability to care for PWDs outside the context of their disability. Instead of a physician preceptor, our SPs give direct feedback to our students, and we use their assessment and feedback in calculating final grades. Because we use scenarios that are not disability-specific, the OSCEs are also performed by SPs without disabilities; therefore, we have been able to compare students' performances in the same clinical situation, with the main variable being whether the SP had a disability or did not have a disability.

The purpose of our study was to evaluate medical students' performance during OSCEs using common outpatient scenarios and three groups of SPs—one group with SCI, a second group with ID, and a third group without a disability. We hypothesized that students would achieve lower scores on the OSCE for scenarios using SPs with disabilities.

Back to Top | Article Outline

Method

This research was classified as exempt by the University of South Carolina Institutional Review Board because it was deemed a standard teaching evaluation. The design of the study was quasi-experimental with comparisons between three groups of medical students during OSCE experiences involving (1) an SP with no disability, (2) an SP with SCI (SP-SCI), and (3) a pair of SPs comprising an SP with ID (SP-ID) and an SP portraying a caregiver.

All third-year medical students on the family medicine (FM) clerkship (146) during the study period participated. Sixty-six students participated in an OSCE scenario involving an SP without a disability who was diagnosed with diabetes, while 40 students participated in a similar scenario involving an SP-SCI. Seventy-three students were assigned to a scenario involving an SP without a disability who had hypertension, and 53 students were assigned to a similar scenario involving an SP-ID and an SP portraying a caregiver. The OSCEs are described in detail below. We examined the students' scores for each of these OSCEs at the end of each six-week FM clerkship during the academic years 2007–2008 and 2008–2009.

Back to Top | Article Outline
The OSCEs

The first OSCE scenario involved an SP presenting with fatigue and polyuria who ultimately was diagnosed with diabetes. This scenario was presented either by (1) an SP with no disability (diabetes) or (2) an SP-SCI (diabetes–SCI). All SP-SCIs were greater than 10 years postinjury, had an SCI between the C5 and T9 vertebral levels, used manual wheelchairs, and were verbal. The students were required to obtain a focused history including past medical history, family history, medications, social history, and a review of systems concentrating on cardiopulmonary, neurological, and genitourinary symptoms. The student must also have performed a focused cardiovascular, lung, extremity, and neurological exam. If the student asked for a finger stick blood glucose and/or urinalysis, the results were provided by the SP. Because this scenario focused on making a diagnosis for an undifferentiated problem, the primary critical action for the student was to order a hemoglobin A1c. Students also were expected to order a fasting lipid panel and recommend an ophthalmology referral to assess for additional risk factors and sequelae of the likely diagnosis of diabetes in this patient.

The second OSCE scenario involved an SP who presented to the office for evaluation of his or her elevated blood pressure and was performed either by (1) an SP with no disability (hypertension), or (2) a pair of SPs—an SP-ID and an SP acting as a caregiver (hypertension–ID). The SP-IDs were at least 18 years old, ambulatory, and verbal. The ID was moderate, and a number of the SPs had Down syndrome. The SPs portraying the caregivers included a mother of an SP-ID, an SP-ID's sister-in-law, and a staff member from a group residence where an SP-ID resided. Again, the students were required to obtain a focused history including past medical history, family history, medications, social history, and a review of systems concentrating on cardiovascular, pulmonary, and neurological systems. Students should have performed physical exam components such as a fundoscopic, cardiovascular, lung, and neurological exam. In addition to assessing for additional risk factors, students were expected to order a urinalysis or basic metabolic panel and to counsel the patient on starting medications and lifestyle modifications.

For the two scenarios, the students were allowed 15 minutes for each “visit.” At the conclusion of the role-play, the SP completed an itemized checklist of the required components. The SP-SCIs completed the checklist themselves, whereas the SP caregivers completed the checklists for the SP-ID pairs. A five-minute review of the checklist with the student provided an opportunity for the SP and the student to discuss strengths and weaknesses as well as any discrepancies in the assigned scores. In those OSCEs in which SPs with disabilities participated, the scenarios and checklists were slightly modified to incorporate additional disability-specific checklist items related to an SCI or an ID (List 1).

List 1
List 1
Image Tools

Three OSCE final exam scenarios were administered at the end of each clerkship during the study period and could include standard FM scenarios as well as the scenarios described in this study (Chart 1). Only one scenario per clerkship used an SP with a disability, and no clerkship's final three OSCEs included the same scenario with an SP with a disability and an SP without a disability (i.e., hypertension–ID and hypertension were not scheduled during the same clerkship). OSCEs were scheduled to distribute all the scenarios proportionately throughout the year.

Chart 1
Chart 1
Image Tools
Back to Top | Article Outline
Statistical analysis

We developed a composite score for the two OSCE scenarios which included category subscores for history, physical examination, lab tests, counseling, and interpersonal skills. For each category of the OSCE score sheet, we calculated the percentage correct (the number of the responses with a correct answer over the total number of questions). We used a t test to compare the difference in each category's percentage correct between students who saw an SP without a disability and students who saw an SP with a disability.

We compared the responses for medical students who saw an SP without a disability versus those who saw an SP with a disability using logistic regression modeling or Fisher exact test to determine whether the SP's disability status predicted successful management of the scenarios exhibited by completion of designated critical actions. For the diabetes scenario, the dependent variable was ordering a hemoglobin A1c. For the hypertension scenario, the dependent variables were two laboratory tests (a urinalysis and a basic metabolic panel) and four counseling areas (smoking cessation, exercise, salt intake, and medication use). Each of these was considered as a dichotomous dependent variable. The independent variable was always the SP disability status. We used logistic regression to obtain the odds ratio (OR) and 95% confidence interval (CI) to assess the magnitude and statistical significance of differences between the disability and nondisability groups. For the medication counseling response we used Fisher exact test for the prediction, because logistic regression should not be used when a cell size is less than five, and there was only one student who had a wrong answer when the SP had a disability.

All statistical analyses were performed using SAS 9.1 (Cary, North Carolina).

Back to Top | Article Outline

Results

The age range of the medical students involved in our study was 21 to 41 years, with a mean of 23 years. There were 82 (56%) males, 131 (90%) white students, 3 (2%) African American students, and 12 (8%) students of another race/ethnicity. In addition, 92 (63%) were public college instate graduates, 20 (14%) were private college instate graduates, 20 (14%) were public out-of-state graduates, and 14 (9%) were private out-of-state college graduates.

Table 1 shows the results of the OSCEs administered over academic years 2007–2008 and 2008–2009. In the diabetes scenario, 66 students saw an SP without a disability and 40 students saw an SP-SCI. There were statistically significant differences in all four categories of the encounter. When comparing students who saw an SP without a disability versus students who saw an SP-SCI, the latter group received lower scores in history taking (an average of 80% versus 70%), physical examination skills (93% versus 78%), ordering laboratory tests (92% versus 77%), and interpersonal skills (99% versus 97%).

Table 1
Table 1
Image Tools

When we compared the scores of the 73 students who saw an SP without a disability and the 53 students who saw an SP-ID and their SP caregiver in the hypertension scenario, there were statistically significantly lower scores for the students who saw an SP-ID in history taking (an average of 88% versus 79%), physical examination skills (84% versus 78%), and ordering laboratory tests (88% versus 74%). There was no statistically significant difference between student scores with an SP without a disability and with an SP-ID in the areas of counseling (87% versus 83%, respectively) or interpersonal skills (97% versus 92%, respectively).

In the diabetes scenario, the students who saw an SP without a disability were four times more likely to order a hemoglobin A1c (OR 4.16, 95% CI 1.78–9.71, P = .001) compared with the students who had an SP-SCI (Table 2). In the hypertension scenario, absence of a disability in the SP statistically significantly increased the likelihood of the student ordering a urinalysis (OR 3.08, 95% CI 1.34–7.08, P = .006). Disability status was not a significant predictor for the student's ordering a basic metabolic panel or providing any of the four counseling recommendations by themselves. However, students who saw SPs without a disability were more likely to address all four of the counseling recommendations (OR 2.15, 95% CI 1.04–4.44, P = .038).

Table 2
Table 2
Image Tools

In additional analyses, we looked at the medical student responses to five disability-specific checklist items for diabetes–SCI and three disability-specific checklist items for hypertension–ID (Table 3). Students did well on almost all of these items, and in diabetes–SCI we found scores ranging from 85% for “asking patient how they urinate” to 100% for “avoided talking down.” Students also scored well on items in hypertension–ID except for the item that required the student to ask “the caregiver to leave briefly to assess relationship violence” (21%).

Table 3
Table 3
Image Tools
Back to Top | Article Outline

Discussion

To our knowledge, this study is the first report of using SPs with disabilities in graded OSCEs. The purpose of our study was to use OSCEs to determine whether students evaluate PWDs in common outpatient scenarios differently from the way they evaluate patients without disabilities. We hypothesized that students would achieve lower scores on the OSCE for scenarios involving SPs with disabilities. However, an encouraging aspect of this study is the finding that the interpersonal skills during the exam (although statistically significantly lower for the SP-SCI group) were still quite high for both disability groups.

Our approach is an innovation in medical education because SPs with disabilities were not used to demonstrate disability-specific complaints but instead were used to depict typical primary care problems. Because of this structure, we could compare students' performance in interactions with SPs with and without a disability. Students were expected to perform histories and physical exams as well as initiate diagnostic exploration and make treatment recommendations that were the same regardless of whether the SP had a disability. Those aspects of the diagnosis, management, and visit dynamics that were different because of the patient's disability were added to the OSCE checklist.

Students who saw an SP-SCI scored statistically significantly lower on all four components on the encounter than did students who saw an SP who did not have a disability. More time spent on the medical history, which includes past surgeries, medical issues, and medications as well as questions regarding skin integrity, may have limited the time for some other aspects of the exam. During the physical exam there were items that were not completed as often in scenarios with SP-SCIs as in those with SPs without disabilities, such as palpating the abdomen and checking the lower extremities for a pulse. This observed difference may have been due to a fear of touching the patient with an SCI. Students may have felt that examining the extremities was less relevant than other parts of the encounter because the “patient” could not feel those areas. The students were more than four times more likely to order a hemoglobin A1c when the SP did not have a disability compared with when the patient was an SP-SCI. Students may have assumed the patient's symptoms were directly related to his or her disability and failed to pursue other diagnoses.

In the hypertension scenario, students were more than twice as likely to provide all four components of counseling (smoking cessation, exercise, salt intake reduction, and medication instruction) when the SP did not have a disability. This statistic tells us that although no one counseling topic was preferentially discussed less in OSCEs that used SPs with disabilities, the expectation that the student address all four areas did occur during the hypertension scenario but not during hypertension–ID. The students may have felt that they were less likely to influence lifestyle in the person with ID or that they would offend the caregiver by insinuating a lack of care. Scores were also lower in the interpersonal component of this OSCE which may be due to the challenges of relating to two individuals in the room simultaneously. Additionally, the students were more than three times more likely to order a urinalysis when the SP did not have a disability. Although the reasons for this observation are not obvious, there may simply have been time constraints and/or distraction by the disability itself.

Overall, students did not perform as well in OSCEs that used SPs with disabilities. Several additional reasons may have contributed to these findings. Students may not have had enough time to pursue these components because of accommodations they made in communication or physical exam styles for the SPs with disabilities. They also may have been more focused on the patient's disability despite the scenarios not emphasizing these medical diagnoses; therefore, students may not have explored additional separate diagnoses, appropriate chronic disease maintenance, or methods of therapy. Regardless, these findings highlight the need for further education of medical students about the importance of incorporating a person's disability into their general visit protocol without losing sight of the chief complaint.

Because the study was an evaluation of instructional content that evolved across a two-year period, we explored the differences between clerkships to determine whether there were improvements in OSCE performance related to the one improvement we made in year two in the disability curriculum. The curriculum was modified to include practicing transfers of patients with SCI in the classroom setting, with all students participating by lifting their classmates. We found there were no statistically significant differences over time in the OSCE categories; however, the number of students assisting with transfers to the exam table for an SP-SCI increased from the first year to the second year. The same two instructors taught these classes; thus, aside from the change in the way transfers were taught, there was consistency in the lectures.

The study does have some specific limitations. First, only two specific disabilities (SCI and ID) were tested. We cannot generalize our findings to other disability groups. In addition, although we used a quasi-experimental design, the OSCE scoring was not independently evaluated, and there could have been differential scoring by the SPs with disabilities compared with the SPs without disabilities. This concern is at least in part ameliorated by the discussion at the end of the OSCE when the student could challenge the grading of the checklist with the SP present. Additionally, because our study was conducted at one institution and over only a two-year period, our findings may have limited generalizability. Finally, we do not know if the lessons learned by the students will carry over into practice once they move on to other clerkships and into their residencies. Further research is needed to evaluate the effectiveness of SPs with and without disabilities in a variety of settings and with a range of curricular interventions.

Back to Top | Article Outline

Conclusions

Our study's aim was to evaluate medical students' performance during OSCEs using common outpatient scenarios in which PWDs served as SPs. Our analysis illustrated that students scored higher on OSCEs that used an SP without a disability compared with OSCEs that used an SP with a disability. Although the reasons for this discrepancy are multifactorial, our results suggest that medical student education should include instruction on appropriate care of PWDs.

Back to Top | Article Outline

Funding/Support:

Funding for this research was provided in part to the University of South Carolina through Cooperative Agreement Number 1U59DD000268-01 from the Centers for Disease Control and Prevention. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of the Centers for Disease Control and Prevention.

Back to Top | Article Outline

Other disclosures:

None.

Back to Top | Article Outline

Ethical approval:

This study was approved by the Institutional Review Board of the University of South Carolina internal review board.

Back to Top | Article Outline

Disclaimer:

The opinions expressed in this article are those of the authors alone and do not reflect the views of the University of South Carolina School of Medicine and the Department of Family and Preventive Medicine.

Back to Top | Article Outline

References

1. Iezzoni LI, O'Day BL. More Than Ramps: A Guide to Improving Health Care Quality and Access for People With Disabilities. Oxford, UK: Oxford University Press; 2006.

2. Institute of Medicine. The Future of Disability in America. Washington, DC: The National Academies Press; 2007.

3. Melville C. Discrimination and health inequalities experienced by disabled people. Med Educ. 2005;39:124–125.

4. Iezzoni LI. What should I say? Communication around disability. Ann Intern Med. 1998;129:661–665.

5. Tervo RC, Azuma S, Palmer G, Redinius R. Medical students' attitudes toward persons with disabilities: A comparative study. Arch Phys Med Rehabil. 2002;83:1537–1542.

6. Byron M, Cockshott Z, Brownett H, Ramkalawan T. What does “disability” mean for medical students? An exploration of the words medical students associate with the term “disability.” Med Educ. 2005;39:176–183.

7. Iezzoni LI. Going beyond disease to address disability. N Engl J Med. 2006;355:976–979.

8. Kahtan S, Inman C, Haines A, Holland P. Teaching disability and rehabilitation to medical students. Med Educ. 1994;28:386–393.

9. Crotty M, Finucane P, Ahern M. Teaching medical students about disability and rehabilitation: Methods and student feedback. Med Educ. 2000;34:659–664.

10. Eddey GE, Robey KL. Considering the culture of disability in cultural competence education. Acad Med. 2005;80:706–712.

11. Conill A. Living with disability: A proposal for medical education. JAMA. 1998;279:83.

12. Jain S. Care of patients with disabilities: An important and often ignored aspect of family medicine teaching. Fam Med. 2006;38:13–15.

13. Green AR, Miller E, Krupat E, et al. Designing and implementing a cultural competence OSCE: Lessons learned from interviews with medical students. Ethn Dis. 2007;17:344–350.

14. Corbett EC, Payne NJ, Bradley EB, et al. Enhancing clinical skills education: University of Virginia School of Medicine's clerkship clinical skills workshop program. Acad Med. 2007;82:690–695.

15. Hodges B, McNaughton N, Regehr G, et al. The challenge of creating new OSCE measures to capture the characteristics of expertise. Med Educ. 2002;36:742–748.

16. Iramaneerat C, Yudkowsky R. Rater errors in a clinical skills assessment of medical students. Eval Health Prof. 2007;30:266–283.

17. Well TPE, Byron MA, McMullen SHP, Birchall MA. Disability teaching for medical students: Disabled people contribute to curriculum development. Med Educ. 2002;36:788–790.

18. Eddey GE, Robey KL, McConnell J. Increasing medical student's self-perceived skill and comfort in examining persons with severe developmental disabilities: The use of standardized patients who are nonverbal due to cerebral palsy. Acad Med. 1998;73(10 suppl):S106–S108.

19. Minihan PM, Bradshaw YS, Long LM, Altman W, Perduta-Fulginiti S. Teaching about disability: Involving patients with disabilities as medical educators. Disabil Stud Q. Fall 2004;24. Available at: http://www.dsq-sds.org/article/view/883/1058. Accessed August 22, 2010.

20. Siebens H, Cairns K, Schlalick WO, Fondulis D, et al. PoWER program: People with disabilities educating residents. Am J Phys Med Rehabil. 2004;83:203–209.

21. Sabharwal S. Objective assessment and structured teaching of disability etiquette. Acad Med. 2001;76:509.

22. Sabharwal S, Sebastian JL, Lanouette M. An educational intervention to teach medical students about examining disabled patients. JAMA. 2000;284:1080–1081.

23. Thacker A, Perez W, Crabbe N, et al. The contribution of actors with intellectual disabilities to the training of medical students. Available at: http://www.intellectualdisability.info/how-to../the-contribution-of-actors-with-intellectual-disabilities-to-the-training-of-medical-students.

24. Dehaitem MJ. Dental hygiene education about patients with special needs: A survey of U.S. programs. J Dent Educ. 2008;72:1010–1019.

Cited By:

This article has been cited 1 time(s).

Intellectual and Developmental Disabilities
Reflective Impressions of a Precepted Clinical Experience Caring for People With Disabilities
Karl, R; McGuigan, D; Withiam-Leitch, ML; Akl, EA; Symons, AB
Intellectual and Developmental Disabilities, 51(4): 237-245.
10.1352/1934-9556-51.4.237
CrossRef
Back to Top | Article Outline

© 2010 Association of American Medical Colleges

Login

Article Tools

Images

Share