The selection of medical students for specialty training of any kind is fraught with many difficulties, not the least of which is distinguishing one medical student from another. The formal academic transcript of grades, awards, and any individual research or special clinical activity is easily recorded though this may or may not reflect the type of individual behind those achievements. The literature describing how to select medical students for specialty training is characterized by a paucity of published data in all areas of the process, especially extensive multi-school collaborations to look at the contributions of individual student evaluation tools and the method that the student uses to select a specialty. Survey results published by the Association of American Medical Colleges (AAMC) and a collective wisdom or mythology, depending on one's view, of Associate Deans of Medical Education and Student Affairs, often form the basis for discussion of this endeavor.
In addition, a functional disconnect may exist between student expectations and program needs. The students often have an incomplete view of the specialty in which they may be a member for several decades, since their exposure to a field is often limited to the purview of one or two practitioners mentoring them in that particular branch of medicine. In addition, much of their knowledge may have been gleaned from residents who have not practiced in the “real world” and have not painted an accurate portrait of the life of an orthopaedist or other specialized physician. For the program's part, students often are unknown quantities who have achieved academic success throughout their lives but seem remarkably like any other medical student applicant. Performance on United States Medical Licensing Examination (USMLE) Step 1 lamentably is used as an indicator of success, and clinical year clerkship performance is distinguishable primarily through the number of superlatives lavished on any particular individual.
Rather than throw up one's hands and use the transparent metrics of tests and grades as determinants of student success, let us examine three areas more opaque areas of student evaluation in the residency application process. These areas are the Interview conducted by the residency itself, the Dean's letter (now officially called the Medical Student Performance Evaluation, or MSPE), and a variety of so-called Affective Issues that cover a variety of behavioral and psychiatric questions. Two areas that will not be examined are the recommendation letters and the senior audition elective. Recommendation letters should be universally glowing given the average faculty member's desire to help further the career of a fledgling physician with problems much more likely to be noted in the MSPE. As in all behavioral realms, patterns of behavior observed in multiple settings are much more powerful than isolated incidents of praiseworthy or deplorable actions. Along the same line of reasoning, the MSPE should note the full range of student performance, not only that associated with a senior audition elective, the explicit purpose of which is to show one's best behavior.
If one looks at these three pieces of the application process as pools of data, then each can be mined for useful information. The difficulty is the vague and often hidden nature of the data itself, and the time and energy required for discovery. First, consider the factors that impede straightforward ascertainment of student performance. Much student clinical work goes unobserved, students themselves do not report difficulties, with perfection seeming to be an unstated if futile goal, medical schools tout their own students as proof of school excellence, and students and medical schools struggle with the conception of “fairness” in the evaluation process. Before looking at the three areas of inquiry, consider the baseline; namely, what is the nature of the pool that generated the medical students themselves. I will briefly examine the interview process of medical student applicants to a residency, the Dean's Letter (now called the Medical Student Performance Evaluation [MSPE]), and a variety of affective issues. However, one must start with a consideration of the pool that generated the medical students themselves
What are These Students Like?
Although matriculants to medical schools in 2004 show men still in the majority, the current pool of applicants to medical school shows a majority of women.5 While this is not true currently for the residency applicant pool for orthopaedics, the trend will almost certainly affect this specialty, especially with the attempts to limit duty hours during residency. During the next 1 to 2 years, it is anticipated that the majority of entering medical students will in fact be women, with minority groups continuing to be underrepresented in both applicant and matriculant pools.4 Despite a commonly held assumption that medical schools are accepting large numbers of non-science or other nonstandard applicants, the data show that those matriculating to medical school in 2004 still included 75% with an undergraduate science major, with the majority of those being in the biologic sciences.3 Although many of the incoming students have taken 1 or 2 years to do work, travel, or continue academic studies before medical school, the vast majority were still in their early 20s when they began medical school and hence are in their late 20s during application to residency. Older students have generated a great deal of publicity, but medical school remains the domain of 20-somethings.
Consider that the overwhelming majority of students have always lived with the threat of AIDS; they have never suffered economic hard times; they love television; they love computers; they absorb vast amounts of information in small bits and chunks. Their view of the world is deprecating, sarcastic, and satiric (eg, The Simpsons and Family Guy as entertainment, with recent polls showing that The Daily Show, a satire, is often referred to as a primary source of world news).12 Most defining, perhaps, is that they have been overscheduled since birth because of the dreams and demands of their parents. Many of these students have been involved in multiple activities since grade school. Sports, music, world travel, and school performance have dominated their lives for years. For many, getting into medical school was not as difficult as getting into the undergraduate college of their choice, with Swarthmore College, as an example, admitting only ⅓ of the high school valedictorian/salutatorian applicants that it considers.11 Medical students often seem to know large amounts of material but have few coherent views of how that information should be valued.
Based on this background, it seems the interview should focus on three areas: familiarity with the specialty of orthopaedics, motivation for that specialty, and a series of personal characteristics currently subsumed within a somewhat nebulous category called professionalism. Familiarity with the specialty seems like an odd choice for an interview topic, but the average medical school student may or may not have sufficiently explored his or her specialty choice before deciding on a career. Based on experience at the University of Maryland, some students may be initially attracted to orthopaedics due to their own contact with the specialty through sports injury and rehabilitation. Further, because orthopaedics is not a required clerkship at many medical schools, the breadth and depth of exposure to this field is highly variable, especially if 4th year elective experiences are in highly specialized clinics. We lack data to know what proportion of residents are disappointed with their specialty choice and rather choose to soldier on and change specialties only a last resort.
Motivation for medicine in general, as well as any particular specialty, is difficult to measure and may be confounded by expectations of financial security, but should be explored in an interview. Given that orthopaedics is perceived as a highly remunerative endeavor, the applicant should be asked about his or her views of money, practice opportunities, and the role that finances played in residency selection. The 2004 Association of American Medical Colleges Graduation Questionnaire indicated that more than 80% of medical school graduates agree with the statement, “medicine will not be as financially rewarding in the future as in the past.”2 Fewer than 50% endorsed the statement that “physicians who work hard will always be able to build a successful practice.” One third of the graduates stated that their level of educational debt influenced, at least to a minor extent, their specialty choice. Although 30% stated that they wanted to join full-time university faculty, only 11.5% stated that they wanted substantial involvement in research.2
In addition, medical school administrators have long been familiar with students who go to medical school intent more on becoming a particular specialist than a physician who specializes. Medical students may become distraught when they find out after a year of solid but not spectacular grades that orthopaedics is out of reach and they really do not want to consider another aspect of medicine. Therefore, motivation should be examined during the interview for medicine in general and not just orthopaedics in particular. Those students who embrace a wide view of the profession of medicine may well be more able to adapt to changes within the field during their careers. All of the surgical subspecialities are grappling with changes in technology and the very definition of their fields. Students who espouse a narrow view of their future are more likely to be disappointed. Familiarity with and motivation for orthopaedics is best assessed in the interview with a series of open-ended questions and hypothetical questions about the future. “Describe your life, not just your work, five years after you finish training.” “What changes would you foresee if a new medication significantly reduced the number of hip fractures.” “In what sort of clinical settings do you see yourself practicing?” The key to all of these lines of inquiry is the ability to wait and let the applicant give a second or third answer after the first glib response has been offered. “What else?” is an effective phrase to extend superficial answers.
The interview should provide some hint of what the applicant is like as a person. This is as close as one can get in an interview setting to answering questions of professionalism. Professionalism in this case becomes an extension of personal qualities such as self sacrifice, leadership, commitment to excellence, and altruism. Because all medical students have shown the ability to jump through innumerable hoops in the premedical and medical student world, attention to questions of personality, self-image, and interpersonal success may seem redundant; however, the challenge of specialty training is one of functioning in a team setting with progressive acceptance of leadership and progressive responsibility. A survey conducted in 2000 showed multiple choice questions and general faculty ratings were still the most frequently used information sources for decisions regarding the progress of medical students.8 These sources, however, may tell little of the ability to function within a team. The push to include standardized patient evaluations throughout medical school and licensing testing is a result of the dissatisfaction with previous assessment tools.
Assuming a couple of minutes of introduction, a couple minutes of closure and questions, and the social niceties that accompany any interpersonal interaction, a minimum of 30 minutes is needed to conduct an interview that will do anything more than scratch the surface of the applicant as a person. Any less leaves no time for follow up questions, “active listening” on the part of the interviewer, or hypothetical questions about such areas as bioethics or the effect of changes in health care organization. Can the applicant talk about times when he or she had a leadership role? What were the challenges? What are the applicant's strengths and weaknesses, meaning actual personal/interpersonal weaknesses, not something that they would like to do better? How would a friend describe them? Answers that emphasize service to others and the ability to forestall gratification are more comforting than self-centered answers emphasizing achievement. Just because a person can talk doesn't mean a person can interview another individual. Faculty development focusing on interviewing skills may be an excellent investment for a department to consider.
The interview of at least 30 minutes' duration has the potential to evaluate many skills, including communication, ability to listen, flexibility, and self-reflection. The amount of literature available that actually tracks interview ratings with subsequent success is disappointingly small.
The Dean's Letter
The MSPE, which is still sometimes referred to as the Dean's Letter, is an attempt to capture the overall performance of a medical student during his or her undergraduate career in medical school. The Group on Student Affairs-the AAMC's body consisting of associate Deans of Student Affairs throughout the country-has attempted to make the MSPE uniform, shorter, and more standardized. Despite the hundreds of hours of work at each medical school devoted to these letters, many residency training directors admit they read only the final paragraph including the last sentence, which is coded to rank the student in his or her class. Although the changes in MSPE have resulted in greater standardization, several factors limit their usefulness. First, the MSPE is supposed to include quotes, free of redaction, from the third year clerkships. However, schools may include informal comments for internal evaluation purposes that are not judged to be important enough for inclusion in the MSPE but may nonetheless show a pattern of problematic behavior. Second, the Liaison Committee on Medical Education rules state that a student has the right to review his or her MSPE before it is sent out from the student's school.7 These rules tend to blunt any criticism or render information vague to the point of loss of meaning. Last, the MSPE introductory paragraphs often distinguish the student based on the level of enthusiasm in tone of the letter itself. Only a careful evaluation of the letter can detect these nuances.
The MSPE is required to include comments about professionalism and any problems in the continuity of education. Although these goals are laudable, they have yet to be fully implemented because of the difficulty in the measurement of a variety of components of the category of behavior known as professionalism. Considering the aforementioned forces leading to uncritical acceptance of student performance, any problems noted in the MSPE are notable, at least to the level of further investigation. An increasing amount of literature shows that professionalism problems of a medical student likely predict more professionalism problems in the future. Therefore, any hint of problematic behavior should be followed up within the applicant interview or through telephone contact with the referring institution. Anecdotal reports among Associate Deans reveal the MSPE often does contain major concerns that are ignored by residency directors. Most startling is to see members of a profession pore over radiographs, laboratory data, and physical findings looking for minute subtleties, but not apply the same rigor to the review of an MSPE. Although not all interviewers within a residency program have the time for thorough review, each program should identify two individuals who will go over each MSPE with a fine toothed comb looking for changes in tone and enthusiasm that may signal larger difficulties. Clerkships where a student, otherwise described as superior, “completes” all requirements should raise an eyebrow. A description that appears apologetic about some aspect of student performance should, likewise, signal potential trouble. Uneven performance may indicate that a student has problems with motivation. More troubling is two or more clerkships reporting even subtle difficulties. Students are entitled to some benefit of doubt on a rotation given the wide range of residents and faculty who will evaluate them. However, a pattern of such things as lateness, brusqueness, or defensiveness no matter how understated in the MSPE is cause for concern.
Affective issues, a euphemism for psychological and psychiatric problems that contribute to problematic behavior, is a subcategory of the broad field of professionalism that occurs over a wide spectrum: they range from the innocuous, though annoying, habits of some medical students to irresponsible and/or dangerous behavior promulgated by pathologic personality abnormality or mental disorder. The convenience of combining these traits into a category is offset by the real difference in causes and outcomes in which these issues may result. In evaluating this area, one must look at the general behavioral norms of the group itself, mainly medical students in their mid to late 20s. For example, the demographic shift to more women in medical school reduces the likelihood of alcohol abuse, a predominantly male behavior; however, the increasing number of women means depression and eating disorders are now statistically more common characteristics of a class of medical students.6
Considering the stringent requirements for acceptance to medical school, the general assumption that severe student psychopathology is absent seems, in general, true though the actual number is protected by confidentiality concerns. However, no psychometric screening is conducted during admission to medical school. Anecdotal and unpublished accounts (oral communication Russell Monroe, MD, 1990) reveal the use of psychological screening tools, such as the Minnesota Multiphasic Personality Inventory™ does, in fact, identify future problematic students. However, these psychometric tools are not used in the admission process and are not likely to be used for ethical and legal reasons. Nonetheless, the repetitive nature of human behavior has received deserved attention; although the phrase, “A leopard can't change its spots” may not be fully accurate, it is rarely a surprise when a student is found to continue a pattern of shortcuts and behavior of convenience first witnessed in medical school. Currently, the medical schools in the AAMC are attempting to formulate policy for criminal background checks for all incoming students. Certainly, increased attention has been given to undergraduate behavior violations, ranging from underage alcohol use to property destruction and occasional felonies. This perceived need for background checks illustrates the partial information being provided by colleges and universities to medical school admission committees. In addition to criminal checks, medical schools may include on the admission application any disciplinary measures taken against the student while an undergraduate. However, the medical school has little independent ability to verify these statements without cooperation from the undergraduate institution. There are, as yet, no estimates on a nationwide basis of the extent of these problems.
One recently published study evaluated a wide variety of markers in the admission and medical student performance areas.10 Nothing in the admission process indicated future professionalism problems. However, indicators of conscientiousness (compliance with receiving vaccination, completing course evaluation forms) did predict professional behavior. Likewise, students who overrated themselves during standardized patient exercises on internal medicine rotations also had less favorable professionalism ratings on those rotations.10 At the University of Maryland, we have found students who overestimate the quality of their patient communication, as assessed with standardized patient evaluations, fall into a Myers-Briggs type characterized as extrovert, intuitive, and feeling (ENF) (personal oral communication, Sandra Dolan, PhD, 2003). No link has been made to professionalism problems as of yet, although the sense of self-satisfaction as opposed to the attempt to conscientiously meet external standards may be a harbinger of problems. One attempt to characterize doctors' professional problems, based on data collected by insurers in California and Ohio, was published in the Hartford Courant in 2003.13 Subsequent regression analysis of those data suggests larger class size and lower Medical College Admission Test scores may predict increased disciplinary problems. The University of Maryland has followed graduates 1 year postgraduation and has found no predictors of future bad behavior on the part of graduates (personal oral communication, Gary Plotnick, MD, Assistant Dean for Student Affairs, 2004). Data from the University of California, San Francisco has noted connections between unprofessional behavior in medical schools and problems once in practice.9
Another wide area for consideration in the evaluation of medical student performance can be found under the rubric “different learning styles.” Included in this category are various levels of attention deficit disorder, assorted reading difficulties, and subtle learning and cognitive discrepancies that have not interfered with behavior or academic achievement until medical school. As an example, during the last several years, the USMLE has been notoriously stringent in allowing students extra time to complete either Step 1, 2, or 3 of the examination (personal oral communication, Sandra Dolan, PhD, 2005). Students now come to medical school complete with psychological testing showing subtle changes in reading and comprehension ability and often demand testing accommodation. These changes in learning style may not reflect an overall change in intellectual ability. Students who have had to overcome specific learning difficulties may in fact be more intellectually capable in other ways to match the performance of their peers. In addition, the life of a practicing physician bears little resemblance to sitting for a 3-hour multiple choice question examination, which is what we often use as a yardstick of success. Again, national figures are not available for the extent and range of these problems.
Attention Deficit Hyperactivity Disorders may present subtle changes that have gone unrecognized. The assumption for many years was that hyperactivity of childhood resolves as the individual moves into adulthood. The currently held view of the field of psychiatry is that some attentional problems that continue into adulthood may respond to psychostimulants. The effect of this disorder and treatment in a medical professional has not been well reviewed, and one can imagine that a doctor admitting use of psychostimulants could prove problematic.
Depression presents a tremendous challenge in evaluating medical student performance. Although it is tempting to say depression should be treated like any other illness with subsequent time off and allowance for treatment, the stigma of mental illness continues to place depression (and more severe psychiatric disorders) in a special category. Up to 10% of women experience a major depressive episode at some point during their lives, and between 50% and 85% will have at least one recurrence, often within the first several years.1 In addition, depression often begins to manifest itself during the time spent in medical school and residency training.6 Work performance may be affected to a subtle or major degree and, paradoxically, the subtle forms of depression often are most problematic. This is because they are not recognized as disease, per se, but rather as changes in attitude, motivation, or ability. Depression remains among the most treatable of psychiatric disorders and, if properly managed, should not provide a barrier for further training and successful practice. Of particular concern with increasing numbers of women is the phenomenon of postpartum depression. Busy women physicians who are in competition to show the ability to compete with men often shorten the time of work stoppage postpartum. This is a time when they may be particularly vulnerable to postpartum depression, and the increased stress of resuming work is likely not to be helpful.1
Of greatest concern because of its intractable nature is personality pathology. Self-centeredness, entitlement, inappropriate anger, deflecting of responsibility, and inflexibility are all anathema to the practice of medicine. These attributes may be signs of immaturity but may also portend more problematic characteristic attributes of the person's personality; behaviors that are out of normal bounds, whether noted in the MSPE or in student evaluations, bear serious attention. Although any one event may seem trivial, it has taken place in an environment in which behavior is highly self-regulated, in which medical students rapidly adhere to norms, and in which all students are fully aware that their behavior is subject to scrutiny. This environment makes it difficult to shrug off behavior disturbances that are clearly not normal behavior expected of college or graduate students. The attention to professionalism in medical schools during the past 10 years is testament to the worries that behavioral norms have been relaxed or overlooked.
While other forms of psychopathology exist in some members of the medical profession such as overt psychosis and severe alcohol abuse, the affective entities selected are those that may present as initially minor or subtle changes in behavior or performance. Any break in education deserves exploration as to the cause and corrective action, if any.
Medical students and medical schools do not overtly collude to hide information about medical student performance that is less than satisfactory. Perhaps it would be more accurate to say that the student is given every benefit of doubt to explain away problems or difficulties as momentary lapses. As a result, the system of reporting we currently utilize is problematic and tends to gloss over potential behavior problems. Careful attention to the interview, the MSPE, and careful evaluation of a student's record can all be helpful in the evaluation of a potential resident, providing information and insight beyond that gained from grades and USMLE test scores. Unfortunately, there are few studies that give guidance to this process and a system to collect data that protects both the student and the reputation of the school is problematic. It is unlikely that more “truthful” statements (helpful judgmental comments) will be added to the evaluation process anytime in the near future. Within the specialty of orthopaedics the competition for limited residency positions will guarantee applicants with outstanding academic qualifications. Faculty time and effort should then be directed to the other sources of information about these individuals. Anything less will result in problematic residents, leading to the fulfillment of the old axiom, “One bad apple can spoil the bushel.”
The author wishes to thank Donna Parker, MD, Associate Dean for Student Affairs, University of Maryland School of Medicine, for consultation and helpful comments.