Garr, David R. MD; Lackland, Daniel T. DrPH; Wilson, Diane B. EdD, RD
Significant changes in the health care system and expectations of consumers have begun to influence medical education in the United States. Medical schools are being encouraged to undertake thorough assessments of the contents of their predoctoral and postgraduate curricula to better prepare physicians for the future.1–3 Educators are recognizing that physicians will need to be able to provide population-based preventive health care as well as high-quality medical care to individuals. While this shift has been perceived by some as dramatic, others see the linkage between medicine and population/public health as essential if health care providers are to provide the best possible health care to the U.S. population. This latter view is reflected in the recommendations endorsed by the Association of American Medical Colleges (AAMC) in its report on population health.2,3
Recognition of the importance of teaching prevention is not new. In 1945 the AAMC recommended that each medical school establish a department of preventive medicine.4 More recently, an expert panel convened by the Association of Teachers of Preventive Medicine proposed curricular requirements in prevention using the Guide to Clinical Preventive Services “to achieve the goal of making preventive medicine an integral part of the education, training, and practice of physicians.”5 Despite these initiatives and the amount of evidence that exists supporting the value of preventive care, the integration of prevention into medical education has been neither rapid nor easily achieved.4–10
The Association of Teachers of Preventive Medicine (ATPM) wished to address the need for more education about prevention in medical schools by establishing the Prevention Curriculum Assistance Program (PCAP). This initiative, jointly funded by the ATPM and the Health Resources and Services Administration (HRSA), began in the fall of 1994. The PCAP was created to achieve two primary goals: (1) to help U.S. allopathic and osteopathic medical schools examine the extents to which they are evaluating medical students' learning about disease prevention/health promotion principles and their applications; and (2) to develop a network whereby medical schools seeking to improve their prevention education and evaluation methods can obtain assistance from institutions willing to be a resource for others.
HISTORY OF THE PCAP
The concept for the PCAP arose with recognition that more needed to be done to help medical schools improve the quality of their teaching of and the evaluation of students' learning about prevention. The first step was to design, test, and distribute an instrument for assessing needs in prevention education. The purpose was to acquire information from medical educators that would guide the development of a program of greatest utility for educating medical students about prevention. The Survey Research Facility at the Medical University of South Carolina undertook the process of designing and testing this needs assessment questionnaire.
The final version of the questionnaire was distributed by telephone facsimile (fax) to the curriculum deans at 125 U.S. medical schools. Ninety-one (91) responses (73%) were received. More than 80% of the respondents acknowledged there was a need for more prevention education in their medical schools. More than 70% expressed interest in using a prevention self-assessment inventory to compare their prevention curricula with a recommended national standard. Based on the results of this needs assessment, the Prevention Self-Assessment Analysis (PSAA) was created. It incorporated many of the recommended curricular-content areas contained in the Inventory of Knowledge and Skills Relating to Disease Prevention and Health Promotion.6
The Inventory was revised for this project in 1995 to incorporate current content objectives in prevention education thought to be essential for all medical students prior to graduation. In July 1995, a national conference entitled “Prevention in Medical Education for the Year 2000” was held. The conference chairperson, William Wiese, MD, presented the following challenge to attendees: “Leaders in medical care services and medical education are now called upon to examine the teaching of prevention in order to assure that graduating physicians have the basic knowledge, skills, and attitudes with respect to how prevention will be practiced in medical care at the start of the new century.” This conference validated the importance of proceeding with the PCAP initiative and, specifically, with the distribution of the PSAA.
The PSAA instrument was developed by the investigators in consultation with an advisory panel of medical educators. The PSAA consisted of 37 questions divided among the Inventory's four sections (clinical preventive services, quantitative methods, community dimensions of medical practice, health services organization and delivery) (Table 1). In October 1996, the PSAA was pilot-tested by the prevention leaders at six additional medical schools for clarity, ease of use, and appearance. Two coauthors of this article (DG, DL) met on-site with faculty who completed the PSAA to obtain their feedback. Based on recommendation, the final version of the instrument was prepared for distribution.
The responses to the survey instrument were used to calculate indicators of institutions' expectations of their students' competence. An indicator was determined for each of the four categories: (1) clinical prevention, (2) quantitative methods, (3) community dimensions, and (4) health services organization. The category indicators were based on the responses to the individual components delineated in Table 1 and were calculated on an overall mean of the institutional responses. Respondents also reported measurement strategies for those categories where competency was expected.
One of the key concerns about distributing the PSAA was to make sure that it reached the appropriate people at each medical school. The investigators used databases from the AAMC and the ATPM in an attempt to identify the curriculum and prevention leaders on each campus. In 1997, the PSAA was mailed to all 125 allopathic and 19 osteopathic medical schools. This article presents the results from the PSAA and a discussion about where attention might need to be focused in the future to better address the prevention education needs in U.S. medical schools.
Of the 144 schools, 98 returned completed surveys between May 1997 and April 1999. The 88 completed surveys from the allopathic medical schools represent a 70% response rate; ten (53%) of the osteopathic medical schools returned surveys. Information derived from the completed surveys provided useful insight into the status of prevention education and evaluation. The following are some of the high-lights from the completed surveys.
Clinical Preventive Services
* Ninety-six percent of the respondents expected medical students to be able to identify the age/sex-specific recommendations for screening tests, prevention counseling, immunizations, and chemoprophylaxis. Ninety-one percent expected students to have the skills to provide these four categories of preventive services to patients. Ninety-four percent indicated that they measured the students' competence in these areas. Written tests and unstructured observation were the two most commonly used methods for evaluation; less than half of the respondents used standardized performance-based assessment methods such as objective structured clinical evaluations (OSCEs) as part of their evaluation programs.
* Only 31% indicated they were satisfied with the quality of the evaluations of outcomes they were employing to assess students' abilities in clinical preventive services. (“Satisfied” was the category to which respondents were assigned when they recorded either a 4 or 5 on a five-point Likert scale, with 5 indicating “very satisfied.”)
* Forty-one percent expressed a desire to receive assistance with designing their curricula and/or evaluation methods relating to clinical preventive services.
* Twenty-six percent indicated that they or their faculty colleagues would be willing to help those at other schools design curricula and/or measurement strategies within the area of clinical preventive services.
* More than 90% of the respondents indicated that they expected their students to achieve competence in seven of the ten skill areas in the quantitative methods section of the survey; depending on the specific question, between 68% and 88% of all respondents indicated that they measured students' competence in the particular area.
* The two lowest response rates, with fewer than 55%, were for expecting students to be able to (1) describe the principles and values of a cost-effectiveness analysis and (2) identify and interpret the basic elements of a decision tree.
* Depending on the skill area being evaluated, written tests were employed 80–97% of the time to assess students' competence in managing information relevant to quantitative methods. Unstructured observation was used 20–30% of the time.
* Forty-seven percent were satisfied with their outcomes evaluations in the quantitative methods area.
* Thirty-seven percent were interested in receiving assistance with designing their quantitative methods curricula and/or evaluation systems.
* Thirty-four percent expressed willingness to assist other schools in teaching quantitative methods and evaluating students' knowledge in this area.
Community Dimensions of Medical Practice
* Depending on the specific content area, between 40% and 86% of respondents expected students to achieve the objectives contained in the community dimensions section of the survey. The highest percentages of positive responses were to the outcomes relating to two areas: an appreciation of the effects of language, culture, and style on the provision of health care services, and the ability to identify community resources to complement individualized care. The fewest positive responses were to the questions relating to risk factors both for individuals traveling abroad and for immigrant populations.
* Of special note, fewer than 60% expected students to be able to describe how to “implement community-responsive, population-based health care.”
* Seventy-six percent indicated that they measured students' competence in the community dimensions content areas. Unstructured observation and written tests were the evaluation methods used most frequently.
* Only 21% were satisfied with the quality of their evaluation methods.
* Forty-six percent wanted assistance with methods to teach and/or evaluate learning within the area of community dimensions.
* Seventeen percent said they would be willing to assist other medical schools with teaching and evaluation methods in the content area of community dimensions.
Health Services Organization and Delivery
* Between 56% and 80% of the respondents expected students to achieve the objectives included in this section. The highest rate of positive responses was to the question relating to students' being able to describe the basic principles of governance and regulations of the medical profession. Only 56% of the respondents expected their students be able to describe methods used to assess the quality of health care services to individuals and populations.
* Of those that did expect the students to learn the content within the area of health services organization, only 72% actually evaluated the learning of the students.
* Depending on the specific question, written tests were employed between 66% and 85% of the time to evaluate students' knowledge about health services. Unstructured observation was cited an average of 33% of the time as the second most commonly used evaluation method.
* Only 12% were satisfied with the outcomes evaluations they employed; no respondent was “very satisfied.”
* Forty-nine percent wanted to receive assistance with either curriculum design or evaluation methods in this area.
* Only 12% indicated willingness to assist other schools with teaching health services organization and delivery.
Comparisons across Content Areas
Figure 1 shows that the majority of schools responded that they expected students to achieve measurable outcomes in each of the four content areas. However, substantially more schools expected their students to acquire expertise in clinical prevention and quantitative methods than in either community dimensions or health services. Similarly, more schools reported that they were currently measuring outcomes in the former two content areas than in the latter two (Figure 2).
When asked to identify specific methods used to measure competence, “written tests” was the response recorded by the majority of respondents in each of the four areas. “Unstructured observation” followed, with more than 50% reporting use of this technique when evaluating students' learning of both clinical preventive services and community dimensions of medical practice. Standardized performance-based assessment was used by 25% of the respondents for measuring outcomes in clinical prevention and by fewer than 12% of the respondents to measure student competence in the three other content areas of the Inventory.
Satisfaction with current measurement techniques varied across the four content areas and was not associated with either currently measuring competence or requesting assistance with designing measurement strategies (Figure 3). Only 25% of the respondents indicated they were satisfied with outcome-evaluation methods in at least two of the four content areas. All of the respondents who indicated they were not satisfied with the measurement of outcomes in two or more areas indicated that they would like assistance in at least one of those areas.
The results from the PSAA provided valuable information pertaining to the education of medical students about prevention and the evaluation of their learning. This discussion sequentially addresses the four sections of the PSAA and the issues that arise when analyzing the data.
Clinical Preventive Services
The responses to the clinical preventive services section of the survey clearly verified that faculty expected medical students to achieve these preventive competencies. The vast majority of the respondents agreed that the knowledge and skills relating to clinical preventive services are important.
Of concern, however, is that written tests and unstructured observation were the methods used most often to evaluate students' learning. Yet, of the four sections of the PSAA, the area of clinical preventive services is most amenable to objective evaluation using methods such as OSCEs. Simply knowing what needs to be done to counsel patients about preventive care is only part of the challenge. The other vitally important aspect is to prepare students who possess the skills to effectively use clinical preventive approaches in the care of their patients. Helping people develop and maintain healthy behaviors is one of the great challenges for physicians. Medical schools need to assess students' competence at incorporating prevention into clinical care. In addition, medical schools need to establish the infrastructure that will enable them to incorporate more extensive performance-based evaluations into their curricula. It would be desirable for medical schools that have created effective standardized performance-based assessments to share their materials and approaches with institutions that seek to improve the quality of their evaluation methods.
The need for the investment of resources and expertise in the evaluation of students' abilities to provide clinical preventive services is verified by the fact that only 31% of all respondents were satisfied with the quality of their outcome evaluations. Clinical preventive services are and will remain an important part of the work of physicians. The results of this survey indicated that more needs to be done to ensure that future physicians are adequately equipped with the skills required to help people take optimal care of themselves.
Most respondents indicated that they expected their students to achieve competence in most of the areas listed in the quantitative methods section of the PSAA. Fewer medical schools actually evaluated the competence of students in this area when compared with the questions relating to the evaluation of knowledge and skills in clinical preventive services. It is interesting that barely half of the respondents expected students to understand the principles of a cost-effectiveness analysis and to be able to utilize a decision tree. In light of the changes occurring in health care, vis-à-vis cost and evidence-based principles, it is concerning that these two areas are receiving less attention than they probably deserve.
The respondents to the quantitative methods section recorded the highest degree of satisfaction with their outcome-evaluation methods. This section also had the largest number of respondents indicating willingness to provide assistance to other medical schools. It would be a worthy goal for medical schools to attain similarly high levels of satisfaction with the other three content areas included in the PSAA.
Community Dimensions of Medical Practice
Relatively few respondents expected students to achieve the objectives in community dimensions of medical practice. Of particular concern was that fewer than 60% of the respondents expected their students to be able to describe how to deliver population-based health care. In this era of expanding managed care and increasing attention to the needs of populations, medical schools need to prepare physicians to practice the type of medicine that will enable them to think beyond the individual patient. Adequate preparation for population-based care will require more than just talking about the subject in the classroom. Community-based experiences need to be provided if we expect our medical school graduates to both understand and be able to practice community-responsive care. This involves substantive changes in where students' learning will occur and in the types of experiences they will require to prepare them for the future.
A smaller number of respondents indicated that they actually assessed students' competence in the area of community dimensions. The need for assistance in assessing students' abilities is confirmed by the fact that 46% of the respondents desired assistance with methods to teach and/or evaluate students' learning in the area of community dimensions. The fact that this is an area that warrants additional attention was also confirmed by the lower percentage of people who were satisfied with their evaluation methods and the small number willing to be of assistance to other medical schools.
Health Services Organization and Delivery
The educational outcomes described in this section of the PSAA generated the smallest number of respondents who felt that these competencies were ones that medical students should attain prior to graduation. Similarly, this section had the lowest percentage of responses of any of the four sections of the PSAA relating to the evaluation of students' learning. Fortunately, there appeared to be substantial interest in receiving assistance with either designing curricula or improving the evaluation methods in health services. It was particularly ironic that the outcomes reflected in this section were of lesser interest to medical educators, given that issues relating to health services organization and delivery are of particular relevance in today's society. It would be unfortunate for medical schools to graduate physicians who are not adequately prepared to understand the system of health care in which they will be practicing.
It was encouraging to find nearly unanimous acknowledgment by the respondents of the importance of including clinical preventive services and quantitative methods in medical student education. Community dimensions and health services are areas more recently recognized as important for future physicians, so the results showing less emphasis in these two areas may not be surprising. New models, such as the combined medicine and public health curriculum developed at Tufts University, may help integrate population health into medical education in the future.9 Others have identified the need for physicians to be knowledgeable about the business and organization of the health care system, thus validating the need for more emphasis on teaching about health services organization and delivery.6
An observation that emerged during the conduct of this survey related to the issue of leadership for prevention education in medical schools. When follow-up contact was made with curriculum leaders at non-responding medical schools, many indicated it would be difficult for them to determine who should complete the PSAA at their institutions. This response is concerning, given that the need for defined faculty leaders for prevention has been identified as an important issue.6 Certainly most medical schools have clearly designated leaders who coordinate the curriculum for such content areas as physical diagnosis and history taking. It seems equally important that one or more individuals be given the responsibility and authority in each medical school for coordinating the prevention curriculum.
One of the other insights gained from this study was that it appeared that there was quite a range of opinion across medical schools as to what should be taught within the prevention curriculum. It would be desirable to have a process on each campus to enable the faculty to explicitly determine what medical students should learn about prevention, using evidence-based recommendations from national bodies. Then, a coordinated approach could be used to ensure that the areas deemed important were adequately addressed in the medical school curriculum.
There were some potential limitations of this study. They included: (1) The quality of the information from each medical school was dependent on the familiarity of the respondent(s) with the details of the prevention curriculum. Specific recommendations accompanied the survey suggesting how the information might be gathered. Optimally, the person completing the survey on each campus would have obtained direct input from the members of the faculty responsible for teaching the content of the prevention curriculum. If all requisite information was not available to those completing the survey, the resulting responses might not be an accurate reflection of the status of the prevention curriculum on each campus. (2) It is possible that some of the survey questions were not optimally clear and thus accurate information was not provided. (3) Responses to the survey were received between May 1997 and April 1999. The status of prevention education on the campuses of the responding medical schools might have changed between the time they returned their completed surveys and the publication of the results of the survey.
The results of the Prevention Self-Assessment Analysis provide a perspective on the status of prevention education and evaluation in U.S. allopathic and osteopathic medical schools. It is encouraging that a high percentage of the respondents concurred that it is important to teach and evaluate students' abilities in the areas of clinical preventive services and quantitative methods. It is also clear, however, that considerable attention needs to be paid to helping medical schools that seek to improve their prevention education and evaluation methods. Certainly, designated leaders to coordinate prevention education on each campus will be critical. In addition, resources such as the Internet can be used to facilitate networking between medical schools, leading to improved educational programs and evaluation methods in prevention.