Educational Deficiencies in Musculoskeletal Medicine

Freedman, Kevin B. MD, MSCE; Bernstein, Joseph MD, MS

Journal of Bone & Joint Surgery - American Volume:
Scientific Article
Abstract

Background: We previously reported the results of a study in which a basic competency examination in musculoskeletal medicine was administered to a group of recent medical school graduates. This examination was validated by 124 orthopaedic program directors, and a passing grade of 73.1% was established. According to that criterion, 82% of the examinees failed to demonstrate basic competency in musculoskeletal medicine. It was suggested that perhaps a different passing grade would have been set by program directors of internal medicine departments. To test that hypothesis, and to determine whether the importance of the individual questions would be rated similarly, the validation process was repeated with program directors of internal medicine residency departments as subjects.

Methods: Our basic competency examination was sent to all 417 program directors of internal medicine departments in the United States. Each recipient was mailed a letter of introduction explaining the purpose of the study, a copy of the examination, and our answer key and scoring guide. There was no mention of the results of the first study. The subjects were requested to rate the importance of each question on the same visual analog scale, ranging from not important to very important, as had been used by the orthopaedic program directors. These ratings were converted into numerical scores. The program directors were also asked to suggest a passing score for the examination, and this score was used to assess the examinees' performance on the examination. The results on the basis of the internal medicine program directors' responses and those according to the orthopaedic program directors' reponses were compared.

Results: Two hundred and forty (58%) of the 417 program directors of internal medicine residency departments responded. They suggested a mean passing score (and standard deviation) of 70.0% ±; 9.9%. As reported previously, the mean test score of the eighty-five examinees was 59.6%. Sixty-six (78%) of them failed to demonstrate basic competency on the examination according to the criterion set by the internal medicine program directors. The internal medicine program directors assigned a mean importance score of 7.4 (of 10) to the questions on the examination compared with a mean score of 7.0 assigned by the orthopaedic program directors. The internal medicine program directors gave twenty-four of the twenty-five questions an importance score of at least 5 and seventeen of the twenty-five questions an importance score of at least 6.6.

Conclusions: According to the standard suggested by the program directors of internal medicine residency departments, a large majority of the examinees once again failed to demonstrate basic competency in musculoskeletal medicine on the examination. It is therefore reasonable to conclude that medical school preparation in musculoskeletal medicine is inadequate.

Author Information

Kevin B. Freedman, MD, MSCE; 1725 West Harrison Street, Suite 1063, Chicago, IL 60612

Joseph Bernstein, MD, MS; Department of Orthopaedic Surgery and Leonard Davis Institute of Health Economics, 424 Stemmler Hall, University of Pennsylvania, Philadelphia, PA 19104-6081. E-mail address: orthodoc@uphs.upenn.edu

Article Outline

Musculoskeletal care is provided by a variety of practitioners, including internists, family practitioners, rheumatologists, emergency physicians, pediatricians, and orthopaedic surgeons. Mastery of the basics of musculoskeletal medicine is therefore essential for many, if not all, medical students. Ideally, a solid knowledge base would be acquired in medical school and refined during postgraduate training.

We previously evaluated the quality of musculoskeletal knowledge among a cohort of recent medical school graduates1. In that study, we administered a basic competency examination in musculoskeletal medicine to eighty-five residents on their first day of residency at our institution. We validated this examination by surveying orthopaedic program directors. These experts were asked to review the examination, rate the importance of each question, and establish a minimum passing score. According to their criterion, 82% of our test group of recent medical school graduates failed to demonstrate basic competency in musculoskeletal medicine.

On the basis of these data, we suggested that medical school training in musculoskeletal medicine is inadequate. Still, we recognize that some may believe that orthopaedic surgeons are not the best group to validate the examination. It could be argued that orthopaedists provide only a minority portion of musculoskeletal medical care and thus may not appreciate what points are important to the average practitioner. Also, it is possible that surgeons overvalue topics with surgical relevance.

To respond to these critics, we surveyed a cohort of program directors who were not surgeons. We wished to first determine whether standards are similar and then to use this second, nonsurgical standard to ratify (or negate) our conclusion that many examinees failed the basic competency examination. Our second purpose, no less important, was to identify topics that both surgeons and nonsurgeons rate as important. Our hope was that the set of topics with broadly acknowledged importance could then be used to generate a consensus curriculum for medical students.

Back to Top | Article Outline

Methods

As previously reported1, a sample of topics in musculoskeletal medicine with which all physicians should be familiar was selected and then elaborated into twenty-five short-answer questions ( Table I ). Needless to say, not all important topics could be included or represented by the number of items commensurate with their importance. An open-response format was selected to eliminate the possibility of the examinees scoring points on the basis of random guessing. A formal answer key and scoring system were developed prior to administration of the examination. To validate our examination, all 157 directors of orthopaedic residency programs in the United States were asked to review it. The program directors were asked to rate the importance of each question on a visual analog scale ranging from not important to very important. Their responses were converted to an importance score for each question, ranging from 0 (not important) to 10 (very important). After reviewing the examination, the orthopaedic program directors were asked to suggest a passing score (as a percentage) for all medical school graduates to demonstrate basic competency in musculoskeletal medicine on this examination.

The examination was administered to all eighty-five medical and surgical residents who were in their first postgraduate year at our institution. The residents completed the examination on their first day of residency. There was no time limit for completion of the examination. Demographic information was obtained by questionnaire; these data included each resident's medical school of origin and his or her participation in required or elective courses in orthopaedic surgery, neurology, rheumatology, and rehabilitation medicine in medical school. The examination was scored anonymously according to the answer key, with a 1-point maximum for each of the twenty-five questions. Partial credit was given for some questions. This raw score was then multiplied by four in order to obtain a percentage score for the examination.

Prior to publication of the first study1, this basic competency examination was sent to all 417 program directors of internal medicine departments in the United States2. Each recipient was mailed a letter of introduction explaining the purpose of the study, a copy of the examination, and our answer key and scoring guide. There was no mention of the results of the first study. The internal medicine program directors were requested to rate the importance of each question on the same visual analog scale, ranging from not important to very important, as had been used by the orthopaedic program directors. With use of instructions identical to those given to the orthopaedic program directors, the internal medicine program directors were asked to suggest a passing score for the examination.

The responses of the internal medicine program directors and those of the orthopaedic program directors were compared and serve as the basis of this report.

Back to Top | Article Outline

Results

Two hundred and forty (58%) of the 417 program directors of internal medicine residency departments completed and returned the survey. The passing score (that is, the score above which an examinee demonstrate[s] basic competency in musculoskeletal medicine) proposed by the internal medicine program directors was 70.0% ±; 9.9% (mean and standard deviation). This score was very similar to the previously reported score of 73.1% proposed by the 124 orthopaedic program directors1. As reported previously1, the subjects scored a mean of 59.6% ±; 12% (range, 35% to 86%) on the examination. Sixty-six (78%) of the eighty-five subjects scored below 70.0% and thus failed to demonstrate basic competency on the examination according to the criterion set by the internal medicine program directors.

The internal medicine program directors assigned a mean importance score of 7.4 of 10 to the questions on the examination; the mean score assigned by the orthopaedic program directors was 7.0. Twenty-four (96%) of the twenty-five questions were given an importance score of at least 5 of 10æa score that previously was deemed to indicate that the topic was at least important—by both sets of respondents. The internal medicine program directors gave seventeen (68%) of the twenty-five questions an importance score of at least 6.6.

Comparison of the ratings of the questions on the 10-point visual analog scale between the orthopaedic and internal medicine program directors revealed that the scores assigned by the two groups were within one point of each other for fifteen of the twenty-five questions and were within 2 points of each other for twenty-two of the twenty-five questions.

Back to Top | Article Outline

Discussion

According to the standard suggested by the program directors of internal medicine residency departments, a large majority of the examinees once again failed to demonstrate basic competency in musculoskeletal medicine. It is reasonable, therefore, to conclude that medical school preparation in musculoskeletal medicine is inadequate. The use of a second criterion to evaluate the musculoskeletal competency examination further validates the use of the examination and confirms the conclusions from our previous study1.

As we noted previously1, the average amount of time spent in courses or rotations dedicated to orthopaedics was only 2.1 weeks for all examinees, and 33% of them graduated from medical school with no such exposure. One or two weeks, representing <2% of the entire typical curriculum, is probably insufficient, and even if all students were to receive two weeks of instruction, it would be important that the emphasis of that exposure be on topics of acknowledged importance. (The standard rotation in orthopaedic surgery probably emphasizes too many particulars of surgical practice.) The ideal course in musculoskeletal medicine should concentrate on common outpatient orthopaedic problems, orthopaedic emergencies, and the musculoskeletal physical examination. As seen from the responses of program directors of internal medicine departments, topics such as fractures, back pain, arthritis, and infections are recognized as important. Indeed, for sixteen of twenty-five questions, the importance score given by the internal medicine program directors was higher than that given by the orthopaedic program directors.

It should be noted that the mean absolute difference in importance scores was <1 for fifteen of the twenty-five questions and <2 for twenty-two. Still, some of the areas of disagreement are instructive. One of the three questions for which the difference in the importance scores was >2 (question 1) asked about perinatal hip dislocation; this question addresses a topic that is no doubt important but perhaps not to a group of program directors of internal medicine departments. It is fair to say that the patients seen in their departments do not present with this condition. Likewise, the other two questions (18 and 21), which asked about differences between rheumatoid arthritis and osteoarthrosis and between osteoporosis and osteomalacia, can be thought of as more typical issues for internists. Accordingly, the internal medicine group ranked the importance of these questions higher than the orthopaedic group did. Thus, if only one group of doctors contributes to curricular design, it is likely that some important topics will be omitted. These data suggest that a concerted effort will be most productive.

Medical school curricula must place a greater emphasis on musculoskeletal medicine. Because of the aging of the population, the prevalence of bone and joint diseases in the United States—already the primary reason that people seek medical care—is sure to rise. Thus, the demands will soon be even greater. Students must master the topic of musculoskeletal medicine. The results of these studies suggest that they have not.

Investigation performed at the University of Pennsylvania School of Medicine, Philadelphia, Pennsylvania

In support of their research or preparation of this manuscript, one or more of the authors received grants or outside funding from the Office of the Dean for Health Services Research, University of Pennsylvania. None of the authors received payments or other benefits or a commitment or agreement to provide such benefits from a commercial entity. No commercial entity paid or directed, or agreed to pay or direct, any benefits to any research fund, foundation, educational institution, or other charitable or nonprofit organization with which the authors are affiliated or associated.

1. Freedman KB, Bernstein J. The adequacy of medical school education in musculoskeletal medicine. J Bone Joint Surg Am, 1998;80: 1421-7 .
2. Accreditation Council for Graduate Medical Education. Directory of graduate medical education programs. Chicago: American Medical Association; 1997.
Copyright 2002 by The Journal of Bone and Joint Surgery, Incorporated