Institutional members access full text with Ovid®

A Quantitative Experimental Paradigm to Optimize Construction of Rank Order Lists in the National Resident Matching Program: The ROSS-MOORE Approach

Ross, David A. MD, PhD; Moore, Edward Z. PhD

doi: 10.1097/ACM.0b013e31829ed3ae
Research Reports

Purpose: As part of the National Resident Matching Program, programs must submit a rank order list of desired applicants. Despite the importance of this process and the numerous manifest limitations with traditional approaches, minimal research has been conducted to examine the accuracy of different ranking strategies.

Method: The authors developed the Moore Optimized Ordinal Rank Estimator (MOORE), a novel algorithm for ranking applicants that is based on college sports ranking systems. Because it is not possible to study the Match in vivo, the authors then designed the Recruitment Outcomes Simulation System (ROSS). This program was used to simulate a series of interview seasons and to compare MOORE and traditional approaches under different conditions.

Results: The accuracy of traditional ranking and the MOORE approach are equally and adversely affected with higher levels of intrarater variability. However, compared with traditional ranking methods, MOORE produces a more accurate rank order list as interrater variability increases.

Conclusions: The present data demonstrate three key findings. First, they provide proof of concept that it is possible to scientifically test the accuracy of different rank methods used in the Match. Second, they show that small amounts of variability can have a significant adverse impact on the accuracy of rank order lists. Finally, they demonstrate that an ordinal approach may lead to a more accurate rank order list in the presence of interviewer bias. The ROSS-MOORE approach offers programs a novel way to optimize the recruitment process and, potentially, to construct a more accurate rank order list.

Dr. Ross is assistant professor, Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut.

Dr. Moore is assistant professor, Department of Engineering, Central Connecticut State University, New Britain, Connecticut.

Editor’s Note: A commentary by K. Baker appears on pages 1206–1208.

Funding/Support: None.

Other disclosures: None.

Ethical approval: Not applicable.

Previous presentations: A portion of these data were presented as a poster at the annual meeting of the American Association of Directors of Psychiatric Residency Training in March 2012.

Supplemental digital content is available for this article at

Correspondence should be addressed to Dr. Ross, Yale Department of Psychiatry, 300 George St., Suite 901, New Haven, CT 06511; telephone: (203) 785-6218; e-mail:

© 2013 by the Association of American Medical Colleges