Secondary Logo

Journal Logo

Addressing the Diversity-Validity Dilemma Using Situational Judgment Tests

Juster, Fern R. MD; Baum, Robin Camhi MS; Zou, Christopher PhD; Risucci, Don PhD; Ly, Anhphan; Reiter, Harold MD; Miller, D. Douglas MD; Dore, Kelly L. PhD

doi: 10.1097/ACM.0000000000002769
Research Report: PDF Only

Purpose To examine the magnitudes of score differences across different demographic groups for three academic (grade point average [GPA], old Medical College Admission Test [MCAT], and MCAT 2015) and one nonacademic (situational judgement test [SJT]) screening measures and one nonacademic (multiple mini-interview [MMI]) interview measure (analysis 1), and the demographic implications of including an SJT in the screening stage for the pool of applicants who are invited to interview (analysis 2).

Method The authors ran the analyses using data from New York Medical College School of Medicine applicants from the 2015–2016 admissions cycle. For analysis 1, effect sizes (Cohen’s d) were calculated for GPA, the old MCAT, MCAT 2015, CASPer (an online SJT), and MMI. Comparisons were made across gender, race, ethnicity (African American, Hispanic/Latino), and socioeconomic status (SES). For analysis 2, a series of simulations were conducted to estimate the number of underrepresented in medicine (UIM) applicants who would have been invited to interview with different weightings of GPA, MCAT, and CASPer scores.

Results A total of 9,096 applicants were included in analysis 1. Group differences were significantly smaller or reversed for CASPer and MMI compared to the academic assessments (MCAT, GPA) across nearly all demographic variables/indicators. The simulations suggested a higher weighting of CASPer may help increase gender, racial, and ethnic diversity in the interview pool; results for low-SES applicants were mixed.

Conclusions The inclusion of an SJT in the admissions process has the potential to widen access to medical education for a number of UIM groups.

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

F.R. Juster was senior associate dean and associate professor of clinical pediatrics, New York Medical College School of Medicine, Valhalla, New York, at the time of this study. She is currently senior associate dean emeritus, New York Medical College School of Medicine, Valhalla, New York, and a graduate student in the Health Sciences Education Master’s program, David Braley Health Science Centre, McMaster University, Hamilton, Ontario, Canada.

R.C. Baum is assistant dean of admissions, New York Medical College School of Medicine, Valhalla, New York.

C. Zou is research scientist, Altus Assessments, Toronto, Ontario, Canada.

D. Risucci is assistant dean for assessment and evaluation and professor of surgery, New York Medical College School of Medicine, Valhalla, New York.

A. Ly is former director of analytics, Academic Administration, New York Medical College School of Medicine, Valhalla, New York.

H. Reiter is professor, Department of Oncology, McMaster University, Hamilton Ontario, Canada.

D.D. Miller is former dean and professor of medicine, New York Medical College School of Medicine, Valhalla, New York.

K.L. Dore is associate professor of medicine, McMaster University, Hamilton, Ontario, Canada.

Acknowledgments: The authors wish to thank the following individuals: Mr. James Demaio for his invaluable data management and commitment to this project, and Dr. Winfred Arthur Jr., who demonstrated the remarkable ability of transmitting multiple piercing insights into a one-hour conversation.

Funding/Support: The authors gratefully acknowledge the gracious support of the Stemmler Fund of the National Board of Medical Examiners (Grant # 1011—039, entitled “Ensuring diversity and test security: An examination of the reliability, validity, feasibility and acceptability of the Computer-based Assessment for Sampling Personal Characteristics (CASPer) in diverse populations of medical school applicants”).

Other disclosures: Two of the authors (H.R., K.L.D.) are founders, board members, and shareholders in a company (Altus Assessments), which creates and runs an online situational judgment test, CASPer. Neither of them receives a salary from the company. A third author (C.Z.) works at Altus Assessments, with salary paid partially by the company and partly by the Ontario Centres of Excellence researcher funds from the Province of Ontario.

Ethical approval: Ethical approval was received from the New York Medical College School of Medicine Institutional Review Board (NYMC-SOM IRB L-12,015).

Previous presentations: Some of the results presented here were also presented at the 2016 Canadian Conference on Medical Education (CCME), April 17, 2016, Montreal, Quebec, Canada, and as a poster at the Association of American Medical Colleges 2016 Annual Meeting, November 11–15, 2016, Seattle, Washington.

Correspondence should be addressed to Fern R. Juster, Master of Health Sciences Education Program, David Braley Health Science Centre, McMaster University, 100 Main St., Room 5003, Hamilton ON, Canada, L8P 1H6; telephone: (914) 441-9485; e-mail:

© 2019 by the Association of American Medical Colleges