Secondary Logo

Resident Selection: A Key to the Future of Orthopaedics

Evarts, C, McCollister

Clinical Orthopaedics and Related Research: August 2006 - Volume 449 - Issue - p 39-43
doi: 10.1097/01.blo.0000224032.23850.55
SECTION I: SYMPOSIUM I: C. T. Brighton/ABJS Workshop on Orthopaedic Education
Free

Recognizing the challenges presented in the process of resident selection, in 1981 the American Orthopaedic Association formed a Steering Committee on Resident Selection. This Committee was charged with studying the processes involved in the selection of orthopaedic residents and developing guidelines and making suggestions to program directors. The activities of the Committee focused on five areas: (1) the mechanics of resident selection; (2) the assessment of cognitive skills; (3) the assessment of motor ability; (4) the assessment of noncognitive factors (the affective domain); (5) the assessment of “dropouts.” The Committee made the following recommendations to help program directors in the selection of residents: (1) use of a standardized application form; (2) full disclosure to applicants; (3) careful selection of candidates to be interviewed; (4) careful planning and implementation of the interview and visit; (5) broad faculty representation and discussion at time of selection; (6) due diligence when necessary. We still believe these criteria important in resident selection.

From the University of Rochester Medical Center, Rochester, NY.

Correspondence to: C. McCollister Evarts, MD, University of Rochester Medical Center, 601 Elmwood Avenue, Box 706, Rm. 1-4337, Rochester, NY 14642. Phone: 585-275-3407; Fax: 585-273-1059; E-mail: mac_evarts@urmc.rochester.edu.

With the emergence of orthopaedics in the 1980s as one of the most desired of all specialties, physician and residency programs became highly competitive and more individuals sought positions than existed. Outstanding graduates of medical schools were selecting orthopaedics as a career choice, representing a substantial change from past years.

The process of resident selection had been addressed in fragments by various committees or subcommittees of different national orthopaedic organizations. Until the 1980s, no concerted effort by representatives from the major orthopaedic organizations had been made. Therefore, the American Orthopaedics Association formed the Steering Committee on Resident Selection and charged it with studying the processes involved in the selection of orthopaedic residents, developing guidelines and making suggestions to program directors. Subsequently, other orthopaedic organizations participated and funding was received from each of the sponsoring organizations over a period of 3 years (1981-1984).

The activities of the Committee focused on five major areas regarding the processing of resident selection: (1) mechanics of resident selection (including the development of a “universal” application form, an informational packet to be sent to prospective residents, sample forms and letters to be used by program directors, and examples of techniques used during the interview process); (2) assessment of cognitive skills; (3) assessment of motor abilities; (4) assessment of noncognitive factors (the affective domain); and (5) assessment of “dropouts.”

The report contained data gathered from questionnaires regarding selection and performance of residents in orthopaedics, a series of questionnaires for the program chairs regarding the affective domain, the results of a pilot motor abilities examination, a survey of residents regarding the affective domain, and a survey of dropouts. It is important to note the results of the survey of 144 program directors. The key issues included the numbers of applicants interviewed, the time and effort spent in the selection process, the importance of affective domain factors, and apparent errors made in the current process of selection.

Back to Top | Article Outline

Mechanics of Resident Selection

The report included guidelines and recommendations for program directors and faculty members. The mechanics of resident selection included the development of a universal application form, guidelines for an informational packet, sample letters and forms; it also suggested techniques for interviewing. The report contained specific information about the assessment of cognitive skills, motor abilities, and noncognitive factors and a survey of dropouts. It was thought by the Steering Committee on Resident Selection that the information and references within the report would be helpful for program directors as they embarked upon the critical process of resident selection. The Committee suggested a universal application form be developed that could be used by all orthopaedic residency programs. An attempt was made to develop just such an application form. The suggested application form was extensive and requested a fair amount of data from each applicant, including the clear reflection of cognitive skills, research experience, presentations, career plans, and a brief personal statement. The form contained some of the information necessary to begin the process of resident selection. A standard form was distributed to the program chairs in 1982 and was subsequently used by several programs.

The guidelines suggested an informational packet accompany the universal application form based on the responses from the questionnaire, which indicated orthopaedic residency candidates did not receive adequate information about residency programs. It was noted the information packet should include detailed statistics about hospital rotations, electives, stipends, community, and so forth. Interview schedules should be arranged. The Internet plays a critical role in the transmission of such information.

The Committee also recommended including guidelines for forms and letters in the information packet. Examples of certain forms and letters could be helpful to the individual orthopaedic program directors in expediting the process of selection and rejection. Maintaining open communication with the resident applicant was considered critical.

The interview was also integral to the process of resident selection. At an interview, the resident candidate can be partly evaluated regarding cognitive skills, motor abilities, and noncognitive factors. Before being interviewed, most prospective residents have prepared an application form containing much information about their past cognitive performances and a personal letter indicating their career goals in orthopaedics. Therefore, at the outset of an interview, much is already known about the candidate.

The Steering Committee on Resident Selection examined a wide variety of interviewing techniques and processes from several institutions to gain an overview of diverse types of procedures. The interviewer should not be seduced by a flawless academic record. Rather, during the interview the surface must be scratched to expose distinct personality traits and candidates with broad interests. Ample time should be set aside for the interviews and the chief should plan an active role in the process.

Back to Top | Article Outline

Assessment of Cognitive Skills

A universal application called for National Board of Medical Examiners scores, college performance, election to honorary societies, preclinical course honors, clinical course honors, clerkship honors, selection to the Alpha Omega Alpha Honor Medical Society, published research, and the like. that the Committee believed the achievement of medical school honors and election to Alpha Omega Alpha correlated with successful performance on the part of residents. However, it was believed that to gather sufficient information about cognitive skills, it was necessary to examine the following areas: (1) Undergraduate education: academic records; election to special honor societies and general national honor societies, such as Phi Beta Kappa; scholarships, including merit scholarships; and status of undergraduate college, including the academic graduate school. Did the applicant attend graduate school? In what did he or she major? Did the applicant earn a degree? What was his or her score on the Graduate Record Examination? What was his or her performance in graduate school? Was the thesis successfully defended? Was the thesis published and, if so, in what publication? (2) Medical school: medical school admission test scores, class rank, number of honors, and election to Alpha Omega Alpha in the third or the senior year. Were any other honors achieved? What do the letters of recommendation say? Did the applicant take the National Board Examination? What were the scores in terms of percentile ranking in Part I? (3) Postgraduate year 1: If the applicant had a postgraduate year 1, what were the ratings of his or her knowledge base? What was the opinion of residents about the applicant? (4) Interview: An attempt should be made to document the cognitive skills during the interview process. The interviewer should cover questions about the work the applicant has done (for example, research on the thesis or in the laboratory).

Back to Top | Article Outline

Assessment of Motor Abilities

Surgeons have long struggled with ways to identify perceptual motor abilities. Little success has been achieved in developing standardized tests with predictive validity. Many anecdotes are offered about techniques for measuring motor abilities, including the actual sawing of a piece of wood into two pieces and creating a wooden cross in the presence of an interviewer. The assessment of such abilities is obviously important to the selection of a resident, and yet no satisfactory means have been identified to elucidate such abilities.

Dental school educators recognized the importance of technical skills and made a major attempt at assessing manual dexterity by a dental aptitude test. The first measure of manual dexterity was the chalk carving test, but this was found to have only a modest relationship with performance in preclinical technical courses. Since 1972, a perceptual motor ability test (a paper and pencil test) has replaced the chalk carving test.3 However, this test has been criticized for its inability to measure true manual dexterity.3 Other tests of technical ability have been proposed and administered by some dental schools. These include the wax carving test and the California Performance Test, which consists of clay modeling and wax and plaster carving ability tests.1 Most recently, a prototype dental dexterity test was developed; the pilot testings were promising for predicting a student's ability to successfully complete relevant preclinical and clinical courses in dentistry.4 However, no such tests have been developed for orthopaedics.

In an attempt to evaluate perceptual motor abilities, a pilot study was conducted by the Steering Committee on Resident Selection using the perceptual motor ability pen and pencil test as used by the dentists.1 Two overriding problems emerged from the study. The first is that the instructions, although directed to the program director, were turned over to other individuals without also sharing the explanations and rationale of the study. The second aspect of the study that proved to be a major stumbling block was the inability of most evaluators to separate overall resident performance from surgical skills. Some earlier experiences in this area led the examiner to believe that this would be a problem no matter how painstaking the explanations. The fact that such a situation existed was reinforced by follow-up letters to program directors asking for explanations of why someone had scored poorly on the test yet had been rated rather highly in terms of overall surgical skills.1 One chairperson stated, “I really don't know why he didn't do better; he always scores in the 90th percentile on the Orthopaedics In-Training Examination and is one of our best residents.”

However, even in those programs in which the distinctions were clearly understood, the value of the test remained questionable. The results on the poor performance side were examined, and an attempt was made to correlate the average test score with the average surgical rating for each program. The results of those whose scores fell below the mean in either one or both areas were then examined. This analysis revealed that 45 residents scored poorly (below their program mean) on the examination and also scored below the mean for their surgical skills. However, 37 scored below the mean on the examination but were rated above the mean for their surgical skills and 48 scored above the mean on the examination but were rated below the mean for their surgical skills. Obviously, the data were not very helpful in the screening process. The only correlation that could be found that might be of some value involved the segment of the residents who scored below 50% on the examination. Eighteen were in that category, and of those, 12 were rated below the mean for their surgical skills (a 2:1 confidence level). Looking at the reverse, for those who scored below 3 (which is an average score for surgical skills rating), the chances were approximately 50:50 that such a resident would score below the mean in the examination.

Further investigations into the identification of perceptual motor abilities are needed.

Back to Top | Article Outline

Assessment of Noncognitive Factors (Affective Domain)

The survey of program directors indicated that affective domain factors were very important in resident selection and performance. Most program directors thought information could be gathered about cognitive skills but that much help was needed in evaluating the noncognitive factors of potential residents. It was emphasized that overall physician clinical competence was based on cognitive skills and noncognitive factors, with motor abilities playing an important role in the procedurally oriented specialties. Obviously, these factors cannot be reported when considering human behavior, yet it was believed necessary to examine each in view of the process of resident selection. All were familiar with examples of resident candidates who had been selected because of outstanding cognitive skills yet whose performance was unsatisfactory because of poor attitudes: a failure of noncognitive factors. Attitudinal traits were not well defined and were less measurable or predictable than were intellectual or motor abilities.

The Steering Committee on Resident Selection viewed as critical a study to help define the attitudinal traits that might be correlated with superior performance as an orthopaedic resident. To accomplish this, a study determining the personal characteristics predictive of successful performance during orthopaedic residency was undertaken.1 In addition it was thought necessary to examine the process through which applicants are selected for a residency training program. The project involved a stratified representative sample of 33 orthopaedic residency programs in the United States, selected based on region and size, involving both large and small programs and both medical school-affiliated and nonaffiliated programs. A cohort of newly admitted orthopaedic residents was studied for 2 successive years, with plans to follow their progress throughout a 3-year program of orthopaedic education. The final evaluation was conducted at the end of the third year.

Identification of personal traits that were deemed to be desirable in residents led to a prospective study of residents entering 33 orthopaedic training programs. The relationship of their self-reported personal traits at admission to the residency program to subsequent performance as evaluated by their chiefs was examined. Data from the study on the role of affective domain indicated that the residents who were identified as poor performers were less likely to consider themselves to be well-adjusted, competent, and thorough. In addition, the study revealed that it was possible to classify good and poor performances.

Henry Wechsler, PhD (Director of Research, The Medical Foundation, Inc., Boston, Massachusetts) prepared two reports: “The Selection and Performance of Residents in Orthopaedics: A Survey of Chiefs of Training Programs” (Appendix 1) and “The Role of Affective Domain Factors in Performance during Orthopaedic Residency.”1 The latter report was extensive and summarized the studies regarding the evaluation of the affective domain. Some orthopaedic residency programs used the tests as described in Dr. Wechsler's 1983 report to aid in their selection of residents. The applicant's responses that were at variance were noted. No applicant was rejected on the basis of these tests alone, but further information regarding the applicant's affective domain was then obtained.

Back to Top | Article Outline

Assessment of Dropouts

Because of an apparent increase in dropouts of selected candidates before their arrival at an orthopaedic residency program and during a program, the Steering Committee on Resident Selection conducted a study of dropouts. The study included both applicants who accepted positions for residency then changed their minds before beginning the residency and orthopaedic residents who decided to resign after starting a residency program. A questionnaire was submitted to program chiefs in an attempt to gain further information about dropouts.

The data collected were summarized as follows: Twenty-eight accepted candidates withdrew from the program before beginning; (1) no indication had been provided by 27 of the 28 candidates that they would drop out, and one response was unclear; (2) of the 28, nine had been accepted to a 3-year residency, 16 to a 4-year residency, and three to a 5-year residency; (3) of the 28, 14 had been accepted to a combined surgery orthopaedic program and 13 to a straight orthopaedics program, and one person did not respond to the question; (4) of the 28, 11 went into other orthopaedic residencies, 10 went into other surgical residencies, and two went into a nonsurgical field; five respondents did not know where the candidates had gone.

Thirty-four more residents resigned after starting their orthopaedic residency: (1) of those who resigned, three resigned from a 3-year orthopaedic residency, 17 from a 4-year orthopaedic residency, and 14 from a 5-year residency; (2) of those who resigned, 15 resigned during their first year of orthopaedic residency, 11 during their second year, six during in their third year, none during their fourth year, and two during their fifth year; (3) thirteen of those who resigned were in a combined surgery orthopaedic program, and 20 were in a straight orthopaedic program (one unknown); (4) those who resigned were asked for their reasons for resignation. In four cases, no reasons were given, and in nine cases, multiple reasons were given. The reasons given included 11 personal reasons, 19 changes of career interest, two dislikes of the work, eight assessments of too much work, and one dislike of the program; (5) of the 34 who resigned, five did not resume residency elsewhere, four entered other orthopaedic programs, six entered radiology programs, three entered emergency medicine programs, five entered residency in another surgical field, four entered residency in a nonsurgical field (not radiology), four did “other things.” Two respondents did not know, and one did not answer.

Back to Top | Article Outline

Recommendations

Recommendations were made by the Committee to help program directors in the selection of residents.

  1. Collect as much preliminary information as possible from the applicant, including a clear statement of his or her educational goals and career plans. The standardized application form was designed to provide this information.
  2. In addition, specific information should be requested from those who are listed as references by the applicant. Even if they have written letters of recommendation, the letters often are so general that they are useless. Once the program has decided what it wants to know about the applicant, the references should be requested to provide such information on those points, perhaps with the use of a checklist.
  3. Provide as much information to the applicant as possible, including a statement of the educational goals of the program and the methods of achieving them. To this end, each program should provide an informational packet that covers all the points listed in the summary.
  4. If the information provided under recommendations 1 and 2 is complete and clear, each party should be able to decide whether the other is an appropriate choice.
  5. For screening of applications, the program director (or faculty member in charge of the resident selection process), after review of all applications and letters of recommendations, should select those applicants who seem most suited to the program for further evaluation by interviews.
  6. For the interview process, prerequisites include setting aside adequate time and meaningful involvement by the program director. Components of the interview process include the following:
    1. Description of the program. If multiple candidates are being evaluated on the same day, this can be accomplished with a presentation to the entire group, preferably at the beginning of the interview process. Thus, during the individual interviews, time does not have to be spent describing the program to each applicant. Instead, that time should be used for specific questions and answers.
    2. Opportunity to see the service in action. For example, a conference, a seminar, and rounds could be presented.
    3. Tour of the facilities. The tour should include the emergency department, operating rooms, inpatient floors, clinics, and other hospital features.
    4. Meeting with residents. A meeting between the applicants and the residents can be accomplished informally during lunch or when the applicants are taken on the tour. It might be advantageous to additionally provide a formal opportunity for a resident to sit down with small groups of applicants and discuss the program with them.
    5. Individual interview by the faculty. At least two 30-minute interviews should be conducted, during which an effort is made to uncover the unique personal characteristics of the applicant that might affect his or her performance in the program. This is perhaps the most critical part of the interview process. Qualities that are sought should be determined in advance, along with their relative weight, and questions or comments that will reveal these traits should be developed. Particularly useful questions should be standardized and used by all interviewers.
    6. Group discussion. If possible, a group discussion or interview in which a faculty member leads a discussion on broad socioeconomic educational issues with the entire group of applicants is held. This format is useful not only to uncover distinct personality traits but also to assess leadership qualities.
    7. “Debriefing” by the program director. Information is imparted about the internship, how to apply to the program, etc.
    Optional parts of the interview process might include interviews with the applicants' spouses, tours of the campus, and so on.
  7. Each faculty member, using all available information, ranks all the residents interviewed, from which a preliminary average ranking of candidates is compiled. Because individual rankings may vary widely, the rank list is reviewed, candidate by candidate, at a subsequent meeting of the entire faculty. This gives faculty members who rated candidates in a widely disparate fashion the opportunity to discuss the reason for their ratings and to change the ranking if appropriate.
  8. The orthopaedic chairperson, or faculty member in charge of the selection process, should call the program chairman at the school of any candidates about whom there is some uncertainty. The final rank order list may be changed if such seems indicated from these conversations.
TABLE 1

TABLE 1

Back to Top | Article Outline

DISCUSSION

The processes involved in the selection of residents were examined and discussed by the Steering Committee on Resident Selection, and certain guidelines and recommendations for program directors were included in the report. The mechanics of resident selection, including the development of a universal application form, guidelines for an informational packet, sample letters and forms, and suggested techniques for interviewing were presented in detail. The report contained specific information about the assessment of cognitive skills, motor abilities, and non-cognitive factors and a dropout survey. It was thought by the Steering Committee on Resident Selection that the information and references within this report would be helpful for program directors as they embark upon the critical process of resident selection.

During my Presidential Address to the Association of Orthopaedic Chairmen on November 6, 1983, I made the following statement: “The right people, the correct numbers, placed into appropriate environments will produce the best and the brightest, therefore beginning a golden era for orthopaedics.”2 This leads to the conclusion, today just as it was in the 1980s, that resident recruitment and retention are of the umost importance to the future of orthopaedics. Resident selection is a linchpin!

Back to Top | Article Outline

Reference

1. Evarts CM, Kelly PJ, Smith RJ, Thompson RC, Cooper RR, Wilson FC, Jr, Kopta JA, Hartman JT. Report by the Steering Committee on Resident Selection. Chicago, Alpha Omega Alpha Honor Medical Society. 1984.
2. Evarts CM. Vantage Points: Association of Orthopaedic Chairmen, Presidential Address. Orthopaedics. 1983;7:226-229.
3. Graham JW. Substitution of perceptual-motor ability test for chalk carving in dental admission testing program. J. Dental Education. 1972;9-14.
4. Weinstein P, Kiyak HA. Assessing manual dexterity. Pilot study of a new instrument. J. Dental Education. 1981;45:71-72.
© 2006 Lippincott Williams & Wilkins, Inc.