The stated mission of Alpha Omega Alpha (AΩA), the national medical honor society, is to improve care by recognizing high educational achievement, honor gifted teaching, encourage the development of leaders in academia and the community, support the ideals of humanism, and promote service to others.1 While hard to find fault in this mission, the true value of AΩA in the 21st century is debatable, as many in medicine view AΩA membership as little more than an honorific. To complicate matters, recent data suggest that students from groups that have been underrepresented in medicine (UiM) are disproportionately excluded from admission into AΩA.2–4 These data come to light as researchers have used tools, both old and new, to show that bias is widespread in assessment in medical education.5–8 In light of these issues, some medical schools are reexamining their AΩA selection process while others have decided to discontinue student inductions.9
At the University of Chicago, Pritzker School of Medicine, both the national conversation and institutional experience led students and faculty to express concerns about our own AΩA chapter.4 A review of our data showed that UiM students were underrepresented among our AΩA inductees. We set out to address these concerns by revising our AΩA selection with the goal of creating a process to equitably choose students who fulfill our school’s mission to “inspire diverse students of exceptional promise to become leaders and innovators in science and medicine for the betterment of humanity.”10 We believe it is informative to share the details of our school’s efforts.
Historically, the selection of students to AΩA at Pritzker has adhered to the general guidelines as written in the AΩA constitution.7 At our medical school, the top quarter of the class, previously defined almost entirely by clinical clerkship grades, were designated as “AΩA eligible.” From this top quartile, an AΩA selection committee—constituted primarily by clerkship directors—selected one-sixth of the graduating class to the honor society. This committee chose students “not only for their high academic standing, but as well for leadership among their peers, professionalism and a firm sense of ethics, promise of future success in medicine, and a commitment to service in the school and community.”11
As we began reconsidering our AΩA selection process, our primary concern was that our student assessments had become outdated. While the means we use to select students for admission to the medical school, and in fact what we value most in future doctors, had changed, the way we assess them in the clerkships had not. Pritzker, like many medical schools, assesses its applicants in a holistic review. In addition to standard metrics (Medical College Admission Test score, grade point average, exposure to advanced sciences, research activities), the admissions committee considers experiences (leadership roles, clinical exposure, community service, “distance traveled”) and personal attributes (interpersonal effectiveness, motivation for medicine, resilience). Our admissions decisions recognize emotional quotient as much as intelligence quotient.12 Over the years, we have succeeded in developing a student body with diverse backgrounds, skill sets, and career goals without sacrificing promise and proven success. However, our assessment system in the clinical clerkships continued to assess only a small number of standard metrics. These metrics were those most easily measured and those that made up the sole admission criteria a generation or more ago.
The Task Force
To begin our process, the dean for Medical Education assembled a task force, made up of faculty and residents who were recent graduates of the medical school. The task force was charged “with exploring the criteria for selection into AΩA and making recommendations to ensure that this process is both fair and inclusive.” The task force set out to identify disparities in AΩA selection, discover the source of this disparity, analyze the effect AΩA had on residency selection, and recommend a new selection process.
Analysis of AΩA selection data from the most recent 5-year period showed that UiM students at Pritzker were less frequently selected to AΩA. The disparity was present at the first stage of the process, when students are determined to be AΩA eligible. There was no detectable disparity in selection of AΩA students from this eligible group. The magnitude of the disparity in AΩA eligibility was that 2 fewer UiM students were AΩA eligible each year compared with what would be expected in a class of 88 students with 15%–22% UiM students.
After the task force identified the magnitude of the disparity, attention turned to identifying its origin. This process began with an analysis of clerkship grades, since these grades dictated AΩA eligibility. A disparity in clerkship grades was present across all clinical clerkships; there was neither a well-performing clerkship that we could use as a model nor a poorly performing one that we could remediate. When evaluating the components of the clerkship grades, the task force found that disparity existed in performance on the NBME subject exams. However, because the results of these exams have only a small impact on final clerkship grades, this disparity had little effect on grade distribution. The largest source of the disparity in clerkship grades was written assessments of students by supervisors (e.g., faculty, residents, fellows) in clinical settings. Disparity was not detected on standardized assessments of clinical activities that used trained evaluators (e.g., objective structured clinical exams, standardized assessments of patient encounters) or in peer evaluations of competencies such as professionalism and humanism.
Extensive, open, reciprocal, and collaborative communication with the student community was maintained during the process. This included multiple town hall meetings and a survey to obtain feedback on a draft version of the task force’s recommendations.
The task force considered a range of recommendations that included discontinuing our AΩA chapter, changing our AΩA selection process, and changing the clinical assessment process, among others. In the end, the task force made 3 broad recommendations (see List 1). First, the task force acknowledged bias in our clinical assessment as evidenced by the disparities in clinical grading and recommended that the clinical evaluation system be changed. It recommended increasing the proportion of clinical clerkship grades determined by trained evaluators using standardized tools and recommended a furthering of efforts already underway to diversify the faculty. Implicit bias training was also recommended for all faculty members who would be evaluating students.
Second, the task force acknowledged that students admitted for strength and promise across varied domains were subsequently being evaluated only on narrow performance criteria. The task force, therefore, suggested broadening the criteria for identifying AΩA eligible students. In addition to clerkship grades, scholarly excellence (as judged by a student’s research as well as volunteer and institutional activities) and peer nomination should be considered. Peer nomination criteria would include mastery of medical knowledge, leadership capabilities, and professionalism.
Lastly, the task force recommended that the AΩA selection committee be reconstituted. They proposed that the committee be populated by members with deep knowledge of the factors that dictate eligibility—clinical performance, peer evaluation, and scholarly excellence—but not include clerkship directors who have already evaluated the students.
The task force acknowledged potential shortcomings of these recommendations, including challenges with accurately predicting excellence in clinical practice amongst trainees. This is especially true when assessing students relatively early in their clinical training and recognizing that the field for which we are training them is constantly in flux. Furthermore, a focus on more standardized measures will mean that important aspects of clinical performance that cannot be evaluated by standardized tools will go unevaluated. On balance, however, the task force felt that the recommended process would reduce bias in identifying students who possess the greatest promise for excelling in the complex task that is the practice of medicine.
Adoption of the task force’s recommendations is underway. The selection committee has been wholly reconstituted, and AΩA eligibility is now determined only after a thorough holistic review of each member of the class. A number of clerkships are piloting new, standardized assessments. Efforts to diversify our clinical faculty continue. While the numbers are too small to make any real judgment about the effects of our intervention, anecdotally, we seem to be recognizing a group of students with greater diversity of skills and accomplishments.
The adoption of the task force’s recommendations has not been without controversy. While some members of the community, both students and faculty, are unsatisfied with the extent and pace of change, others have complained that the selection committee is not recognizing students who, in the past, would have almost certainly been inducted into the honor society. There are also those who feel that having an AΩA chapter, at a school with a competency-based evaluation system where the majority of the curriculum uses grading on a pass/fail system, does more harm than good.
Over the last generation, our medical students have changed as much as the medicine we practice. We have made efforts to change our admissions process to select students who will excel in today’s field, delivering both outstanding clinical care and extraordinary scientific discovery. The AΩA honor society was founded to improve care by recognizing students who possess the greatest promise. If AΩA is to stay relevant, it is necessary that the processes by which we evaluate and recognize our students also evolve. We hope our institution’s efforts to this end will be instructive to other schools.
Summary of Pritzker School of Medicine AΩA Task Force Recommendations
- Change the clinical assessment process:
- a. Diversify the faculty.
- b. Increase the use of trained evaluators using standardized tools for student assessment in the clerkships.
- c. Require these evaluators to undergo bias training.
- In addition to clinical performance as judged by faculty, scholarly excellence and peer evaluation should be considered in selecting AΩA eligible students.
- Reconstitute the AΩA selection committee with members who have expertise in the measures that dictate eligibility.
Abbreviation: AΩA, Alpha Omega Alpha national medical honor society.
1. Alpha Omega Alpha. About Alpha Omega Alpha. http://alphaomegaalpha.org/about.html
. Accessed September 21, 2018
2. Boatright D, Ross D, O’Connor P, Moore E, Nunez-Smith M. Racial disparities in medical student membership in the Alpha Omega Alpha Honor Society. JAMA Intern Med. 2017; 177:659–665
3. Wijesekera TP, Kim M, Moore EZ, Sorenson O, Ross DA. All other things being equal: Exploring racial and gender disparities in medical school honor society induction. Acad Med. 2019; 94:562–569
4. Teherani A, Hauer KE, Fernandez A, King TE Jr, Lucey C. How small differences in assessed clinical performance amplify to large differences in grades and awards: A cascade with serious consequences for students underrepresented in medicine. Acad Med. 2018; 93:1286–1292
5. Heath JK, Weissman GE, Clancy CB, Shou H, Farrar JT, Dine CJ. Assessment of gender-based linguistic differences in physician trainee evaluations of medical faculty using automated text mining. JAMA Netw Open. 2019; 2:e193520
6. Rojek AE, Khanna R, Yim JWL, et al. Differences in narrative language in evaluations of medical students by gender and under-represented minority status. J Gen Intern Med. 2019; 34:684–691
7. Low D, Pollack SW, Liao ZC, et al. Racial/ethnic disparities in clinical grading in medical school. Teach Learn Med. 2019; 31:487–496
8. Dayal A, O’Connor DM, Qadri U, Arora VM. Comparison of male vs female resident milestone evaluations by faculty during emergency medicine residency training. JAMA Intern Med. 2017; 177:651–657
9. Ichan School of Medicine at Mount Sinai. Leadership letters: AOA, equity, and wellness. http://changenow.icahn.mssm.edu/leadership-letters-aoa
. Accessed February 10, 2020
10. The University of Chicago Pritzker School of Medicine. Objectives. https://pritzker.uchicago.edu/academics/objectives
. Accessed February 10, 2020
11. Alpha Omega Alpha. Constitution. http://alphaomegaalpha.org/constitution.html
. Accessed February 10, 2020
12. Emanuel EJ, Gudbranson E. Does medicine overemphasize IQ? JAMA. 2018; 319:651–652