Secondary Logo

Journal Logo

How Admission Committees Decide

Influence of Committee Members' Experience and Applicants' Academic Characteristics

ELAM, CAROL L.; STRATTON, TERRY D.; WILSON, JOHN F.; SCOTT, KIMBERLY L.

Section Editor(s): Tekian, Ara PhD

PAPERS: PERSPECTIVES ON STUDENT SELECTION
Free

Correspondence: Carol L. Elam, EdD, University of Kentucky College of Medicine, 800 Rose Street, MN102, Lexington, KY 40536-0298; e-mail: 〈clelam1@uky.edu〉.

Selection decisions made by medical school admission committees involve consideration of academic and personal information. In previous studies of voting behaviors, Sheperd1 and Daugherty, Eckenfels, and Schmidt2 reported that admission committees learn to distinguish among desirable and undesirable academic qualifications and personal characteristics of applicants. A standard emerges, and most committee members are able to agree who should be accepted and who should be rejected. Seeking unanimity of opinion, however, may not always be desirable. Daugherty and his colleagues reported that approximately 50% of their committee decisions to accept were unanimous. However, when committee voting patterns were linked to students' later medical school performances, they found individual dissent to be an important predictor of students who fell behind their classmates academically. The authors argued that admission committee members should be encouraged to voice doubts about applicants, even when the majority holds a different opinion.2

Several studies have sought to determine which variables are weighted most heavily in selection decisions. Mitchell, in a national survey of admission directors, found total undergraduate grade-point average (GPA), science GPA, quality of undergraduate institution, letters of evaluation, medical school interviews, involvement in non—health-related extracurricular activities, Medical College Admission Test (MCAT) scores, health-related work experience, and state of legal residence were selection variables of high importance.3 In a comparable study, Arnold and colleagues identified college GPA, medical school interviews, and MCAT scores as the three most important factors considered in selecting a candidate for admission.4

Various researchers have suggested that the composition of the admission committee and the characteristics of committee members may influence the weighting of selection variables and, consequently, which applicants are admitted to medical school.4,5,6 Given that admission committees are typically composed of individuals from a variety of backgrounds, it is reasonable to expect their weightings of academic and personal information—and their voting behaviors—to vary accordingly across applicants. Fostering discussions around applicant attributes may help committee members better identify issues of concern and develop a collective understanding of these variations. Elam and colleagues found that a deliberative process, wherein committee members discussed the applicants' files prior to casting their votes, corresponded with changes in approximately one of five votes cast.7

At our institution, admission committee members are expected to review the materials of all interviewed applicants prior to group consideration of the files. Members formulate an overall impression of each applicant's suitability for medicine and for admission to our institution, and are prepared to discuss each application at the committee meeting. Thus, they formulate two consecutive impressions —the first following preliminary review of the file, and the second after committee deliberation. This study focused on vote-changing behavior as one tangible outcome of a deliberative process; we did not seek to evaluate the content or efficacy of that process. Instead, we were curious whether committee members' background characteristics were associated with their propensities to change votes in a context that allowed for deliberation about applicants. Conversely, we also wondered whether academic characteristics of applicants were linked to members' vote-changing behaviors. We posited the following research questions: (1) What background characteristics distinguish committee members who are more (or less) likely to change votes? (2) Is unanimity of voting after preliminary file review related to the likelihood of vote changing after committee deliberation? (3) Are low interview scores, science or non-science GPAs, and MCAT scores associated with a greater likelihood of vote change following committee deliberation?

Back to Top | Article Outline

Method

In addition to academic factors such as undergraduate GPA and MCAT scores, applicants to this Southeastern medical school are screened for interview by the assistant dean for admissions according to personal information from letters of evaluation and the American Medical College Application Service (AMCAS) application, as well as exposure to medicine, leadership, and service. Because the school is a public medical school, preference is given to state residents, who constitute roughly 90% of the class each year.

The study period covered two consecutive admission cycles, 1999 and 2000. During this time, a total of 18 different individuals served on the 15-member committee, including ten College of Medicine faculty (six clinicians, four basic scientists), five medical students, and three medical center administrators or community members. Five members were female (three faculty, two students). Faculty, administrators, and community members serve three-year terms and can be reappointed; students serve two-year terms. Over the two admission cycles, the composition of the committee changed only marginally—usually due to the replacement of graduating students.

In a rolling admission process, files are reviewed and selection decisions are made once applicants are interviewed. Approximately 30 applications are discussed at each bimonthly meeting of the admission committee, and all committee members are expected to have reviewed the files of all applicants on the docket. At two separate time periods—after initial screening and again after group discussion—committee members were asked to record on the applicant's biographical sheet whether they intended to vote to accept, hold for comparison, or reject the applicant. Biographical sheets for each applicant were collected from all committee members at the close of each meeting, and file review data were transcribed, with both the preparatory (after screening) and the post-deliberation votes (after discussion) recorded in a database.

Application files for 608 applicants receiving interviews during the 1999 and 2000 application cycles were examined. Complete data were available for 597 applicants. The following data were examined for each applicant: undergraduate science and non-science GPAs, MCAT score (defined as lowest Verbal Reasoning, Biological Science, or Physical Science score on a 1–15 scale), and interview rating (defined as lowest of two interview scores on a 1–10 scale). Lowest MCAT and interview scores were used to better represent committee cues or flags. The committee members' backgrounds (faculty, student, administrator/community member) and years of committee service (<five years or >five years) were also recorded. Because of the relatively small number of female committee members, gender differences were not examined.

Simple univariate statistics were used to describe the committee characteristics, frequency, and nature of committee members' vote-changing behaviors. Multiple regression analysis was used to examine the relationship between committee members' characteristics and vote-changing behaviors, and logistic regression was used to examine the association between “low” academic indicators and the probability of vote change following deliberation. Analyses were performed on the two combined applicant cohorts, with year as a control variable.

Back to Top | Article Outline

Results

During the study period, the admission committee reviewed application files from 343 males and 254 females. Mean scores on the MCAT were Verbal Reasoning, 9.1 (SD = 1.7); Physical Sciences, 8.9 (SD = 1.9); and Biological Sciences 9.3 (SD = 1.6). The median score on the Writing Sample was P. Mean undergraduate science GPA was 3.48 (SD = 0.43), non-science GPA was 3.65 (SD = 0.38); and total GPA was 3.55 (SD = 0.39). A comparison of MCAT scores and undergraduate GPAs revealed no significant difference between the two applicant cohorts.

Lengths of service among the 18 various members constituting the committee (ten faculty, five students, three community members) ranged from one to more than ten years, with a median of five years. The ten faculty members were six full professors, three associates, and one assistant. Forty-three percent of committee members serving within the two-year period were classified as “high” in experience (>five years), with the remaining 57% characterized as “low” (<five years). Using multiple regression analysis, we examined the influences of length of committee service and background of committee member on vote-changing frequency. The two dependent variables for the regression analysis were: (1) percentage of times a committee member's vote changed in a negative direction (from accept to hold/reject, or from hold to reject); and (2) the percentage of times a committee member's vote changed in a positive direction (from reject to hold/accept or hold to accept).

Differences in vote-change percentages among faculty, student, and administrative/community members are shown in Figure 1. All mean values reported are least-squared means, adjusted for the presence of all other independent variables in the model. Negative vote changes occurred equally among the three categories (F(2,25) = .03, p = .97). However, students were more likely than faculty members and administrator/community members to change in a positive direction (p < .01). Similarly, faculty members were more likely than administrative/community members to change in a positive direction (F(2,25) = 20.08, p < .0001). More experienced committee members, with five or more years of service, were also significantly more likely to change votes in a positive direction (mean = 8.5%) than were less experienced committee members (mean = 5.3%) (F(1,25) = 11.22 p = .002), but equally likely to change votes in a negative direction (experience high mean = 10.5%, experience low mean = 9.4% (F(1,25) = 0.43, p = .52).

Figure 1

Figure 1

Second, we compared the unanimity of members' votes after preliminary review of the file with that following committee deliberation. We defined substantial vote change as more than one committee member's changing votes from after review to after deliberation. This occurred for 46% of the applicants (275/597). As can be seen in Figure 2, deliberation plays a much more important role when there is less consensus regarding applicants' overall suitability after the file review.

Figure 2

Figure 2

Finally, we used logistic regression to examine the likelihood of substantial vote change following deliberation when an applicant had one low interview score (<8, 49.3%), one low MCAT subscore (<8, 23.2%), or a low science or non-science GPA (<3.25, 24.0%). The likelihood of a substantial (>1) vote change was significantly more likely (Wald (3df) = 27.97, p < .0001) when a candidate had a low interview score (OR = 2.33, CI = 1.67–3.26). Low GPAs (OR = 1.19, CI = .81–1.75) and low MCAT scores (OR = 1.02, CI = 68–1.51) were not statistically significant predictors of substantial vote change.

Back to Top | Article Outline

Discussion

Deans of admission are responsible for selecting an admission committee, training the members, and overseeing the selection process. What is the appropriate demographic composition of committee members? What is an appropriate term of service? Should members serve consecutive terms? What is the appropriate balance between members with experience (continuity) and members with no or limited experience (infusion of new perspectives)? This study addressed a few of these questions by examining one element of committee function: members' voting behaviors.

The findings suggest that experienced committee members—as well as faculty and student members—are more likely to change their votes in a positive direction following deliberation. Committee factors were not associated with likelihood of negative vote change. We believe deliberation may provide an opportunity for the expression of individual committee members' concerns about applicants, ultimately leading to more informed decisions. The observation that experienced committee members (>five years) were more fluid in their votes than were their less experienced counter-parts (<five years) supports this contention. However, we also believe that vote change following discussion reflects a cultivated group atmosphere that values diverse interpretations of information from applicants' dossiers. In this sense, vote change following discussion could be considered one indication of a well-functioning group. Druskat and Pescosolido have suggested that effective groups develop shared mental models cultivated by the organizational structure. The shared learning of roles and process by the committee may result in more vote change by learning to rely on other committee members' input.8 The skewed distribution of vote changes when committee consensus is mixed following preparatory review suggests that this is an active and fluid process.

The relatively higher frequency of vote changes when one interview score is low perhaps points to this committee's need to interpret and gauge others' responses to subjective data such as the narratives that accompany interview scores. Hence, deliberation may be an important part of integrating less “objective” data (e.g., interviewer comments) into an overall assessment of an applicant.

Because this study was conducted at a single institution that subscribes to a rolling admission process, the findings may not be readily generalizable to all medical programs. Moreover, additional factors that may contribute to the selection process were not addressed (e.g., gender, race, and geographic origin of committee members and applicants, AMCAS personal statements, letters of evaluation, and re-applicant status). Future research should further identify and establish valid indicators of benefit or impact of vote-changing behaviors.

To ascertain the impact of deliberation on voting behavior, future studies should also analyze the content of committee discussion to explore what is said, by whom, and with what intensity. Limiting discussion when there is pre-meeting consensus may also be one way of streamlining the committee process. Further, the decision-making processes of experienced committee members should be explored to determine their relative valuations of various applicant characteristics and academic qualifications. Understanding the factors that contribute to a thorough review of applicant qualifications and suitability for admission, as well as those characteristics of committee members who contribute to an informed discussion, could prove useful in recruiting and training new members and, ultimately, improve the judgment process used in making selection decisions.

Back to Top | Article Outline

References

1. Shepard A. The selection of medical students. JAMA. 1987;257:2291–2.
2. Daugherty SR, Eckenfels EJ, Schmidt JL. Admission committee dissent as a predictor of problems in medical school. Acad Med. 1990;65(9 suppl):S1–S2.
3. Mitchell KJ. Traditional predictors of performance in medical school. Acad Med. 1990;65:149–58.
4. Arnold DM, Coe RM, Pepper M. Structure and function of medical school admissions committees. J Med Educ. 1984;59:131–2.
5. Kondo DG, Judd VE. Demographic characteristics of U.S. medical school admission committees. JAMA. 2000;284:1111–3.
6. Arnold L, Calkins V, Brumwell M. Influence of decision-makers' characteristics on outcome of a selection process in medical school. Psychol Reps. 1979;44:535–44.
7. Elam CL, Stratton TD, Scott KL, Wilson JF, Lieber A. Review, deliberation and voting: a study of selection decisions in a medical school admission committee. Teach Learn Med. 2002;14:98–103.
8. Druskat VU, Pescosolido AT. The concept of effective teamwork mental models in self-managing teams: ownership, learning and heedful interrelating. Human Relations. 2002;55:283–314.
© 2002 by the Association of American Medical Colleges